Insignificant parameters in MNL

[Linlin]

Dear Prof Bliemer,

I have several questions related to efficient designs.

First, I have 7 parameters estimated in the pilot survey. However, only 2 of them are statistically significant, which can be used as priors. Why many of them are insignificant?

Second, what values should I assign to the insignificant parameters in the D-error efficient design? Can I use the estimations from the pilot survey even though they are insignificant?

Many thanks,
Lin

[Linlin]

Sorry Prof Bliemer, I forgot to give a quick introduction to the pilot study.

I collected a sample of 185 respondents, but I have designed the pilot survey with priors from different literatures which is totally a huge mistake. But when I realized that it's not a proper way to do so, I've dilivered the questionnaire.

the estimation results of MNL and MXL are demonstrated below:
MNL in preference-space

var. coef. st.err. p-value
Remote control of electricity use morning 0.2266** 0.0946 0.0167
Remote control of electricity use afternoon -0.1085 0.1081 0.3154
Remote control of electricity use evening -1.1265*** 0.1611 0.0000
Maximum number of controlled switch-offs each year -0.0042 0.0036 0.2421
Maximum duration each controlled switch-off 0.0005 0.0014 0.7060
Compensation 0.0034 0.0098 0.7324
Compensation due -0.0040 0.0033 0.2205

Model diagnostics:
LL at convergence: -1107.9510
LL at constant(s) only: -1067.8275
McFadden's pseudo-R²: -0.0376
Ben-Akiva-Lerman's pseudo-R²: 0.3842

MXL in preference-space
Means Standard Deviations
dis coef. st.err. p-value coef. st.err. p-value
Remote control of electricity use morning n 0.2466** 0.0988 0.0125 0.1417 0.1068 0.1844
Remote control of electricity use afternoon n -0.1472 0.1196 0.2184 0.2591* 0.1358 0.0564
Remote control of electricity use evening n -4.6073*** 0.7415 0.0000 4.7724*** 0.7364 0.0000
Maximum number of controlled switch-offs each year n -0.0058 0.0039 0.1340 0.0173*** 0.0061 0.0045
Maximum duration each controlled switch-off n 0.0004 0.0016 0.7781 0.0071*** 0.0024 0.0031
Compensation n 0.0061 0.0107 0.5679 0.0368*** 0.0111 0.0010
Compensation due n -0.0040 0.0035 0.2568 0.0120*** 0.0033 0.0003

Model diagnostics:
LL at convergence: -936.8781
LL at constant(s) only: -1067.8275
McFadden's pseudo-R²: 0.1226
Ben-Akiva-Lerman's pseudo-R²: 0.4432




3 Is it better to use the results from MXL for final survey?

4 And is the parameters from MXL avaliable for priors in MNL in Ngene? Or the parameters from MXL just avaliavle for Bayesian design in Ngene?

5 The model has an attribute ("Maximum duration each controlled switch-off ") that has four levels: 15, 30, 60,120 for three alternatives). In Ngene, will it make a defference when we write these level use [15,30,60,120] or [1.5,3,6,12]? Will it affect the fitting of the discrete choice model later?

[Michiel Bliemer]

My answers to your questions:

1. Parameters will often not be statistically significant in the pilot study because of the small sample size.

2. As long as the signs of the parameters and the preference order of the dummy coded variables make sense, then you can use the priors from the pilot study. Priors are merely a "best guess" and the best guess you have is the value of the parameter estimate. The standard error indicates the unreliability of the prior and you can specify these as Bayesian priors as (n,mean,stdev), where mean is the parameter estimate and stdev is the standard error of the parameter estimate. If the parameter has the wrong sign, you can choose to use a very small positive/negative value, for example (n,-0.00001,stdev), where stdev still refers to the standard error. If the standard errors are very large, then you may choose to use ;eff = (mnl,d,median) instead of ;eff = (mnl,d,mean), to avoid extreme outliers in the draws from the Bayesian prior distribution.

3. I recommend using the MNL model parameter estimates and optimise for the MNL model in your design. It is not computationally feasible to optimise for a MXL model in your design. A design that is optimised for the MNL model will generally also be reasonably efficient for estimating a MXL model.

4. Bayesian priors come from the MNL model as follows: for remote control of electricity use morning you use b1[(n,0.2266,0.0946)] to define the Bayesian prior.

5. Your attribute levels need to be consistent with the priors. If you use levels 15, 30, 60 and 120 as levels, then your prior will be 10x as small as when using 1.5, 3, 6 and 12 as levels. You need to use the same attribute levels in Ngene that you will also use in model estimation to make sure that your priors are properly scaled. So it does not matter which attribute levels you use (as long as you are consistent), the behaviour model will be the same, but your priors will scale up or down based on the levels you use such that the contribution to utility remains the same.

Michiel

[Linlin]

Thank you Prof Bliemer, I run the Bayesian MNL using priors from the pilot study, but I got high S estimate.
How many samples shall I collect? There is impossible for me to obtain such a huge number of samples.

Bayesian
S estimate 11310.036(Fixed) 123078107.57(Mean) Std dev.(1736508989) Median(6984.76) 182.497(Mininmum) 24558222009(Maximum)

Here is the code:

Code: Select all

Design
;alts = alt1*, alt2*,alt3*
;rows = 12
;block = 2
;eff = (mnl, d, mean)
;model:
U(alt1) = ele.dummy[(n,-0.1085,0.1081)|(n,-1.1265,0.1611)]*A[1,2,0]+OffNo[(n,-0.0042,0.0036)]*B[2,12,30]+Offdu[(u,-0.1,0)]*C[15,30,60,120]+Com[(n,0.0034,0.0098)]*D[2,4,6,10]/
U(alt2) = ele.dummy*A                        +OffNo*B                 +Offdu*C                    +Com*D             /
U(alt3) = 0
$

[Michiel Bliemer]

ele(d0) --> your prior value indicates that this level does not contribute much to utility (0.1) and is unreliable (relatively large standard error), therefore the sample size estimate is very large

ele(d1) --> you should have no issue in estimating this parameter

offno --> your prior value indicates that attribute B does not contribute much to utility (at maximum -0.0042 * 30) and is unreliable (relatively large standard error), therefore the sample size estimate is very large. Would you expect that this attribute is not important to respondents? is the range too narrow? Also check if you used the same levels (2, 12, 30) in model estimation.

offdu ~5 --> you should have no issue in estimating this parameter

com --> your prior value indicates that attribute D does not contribute much to utility (at maximum 0.0034 * 10) and is unreliable (relatively large standard error), therefore the sample size estimate is very large. Would you expect that this attribute is not important to respondents? is the range too narrow? Also check if you used the same levels (2,4,6,10) in model estimation.

Michiel

[Linlin]

Many thanks, Prof Bliemer

I believe that design with the priors from different literature has led to so many troubles to the pilot results. So I did the pilot study again, and the code is provided with the sign of parameters:

Code: Select all

Design
;alts = alt1*, alt2*,alt3*
;rows = 12
;block = 2
;eff = (mnl, d)
;model:
U(alt1) = ele.dummy[-0.00001|-0.00003]*A[1,2,0]+OffNo[-0.00001]*B[2,12,30]+Offdu[-0.00001]*C[15,30,60,120]+Com[0.00001]*D[2,4,6,10]/
U(alt2) = ele.dummy*A                        +OffNo*B                 +Offdu*C                    +Com*D             /
U(alt3) = 0
$

I obtain the sample with 100 respondents, and the estimation result is below:

MNL in preference-space

var. coef. st.err. p-value
Remote control of electricity use afternoon[ele(d0)] 0.5945*** 0.1276 0.0000
Remote control of electricity use evening [ele(d1)] 0.7241*** 0.1292 0.0000
Maximum number of controlled switch-offs each year*[OffNo] -0.0004 0.0040 0.9248
Maximum duration each controlled switch-off[Offdu] -0.0467*** 0.0176 0.0081
Compensation[Com] 0.0378** 0.0152 0.0130

Model diagnostics:
LL at convergence: -541.9503
LL at constant(s) only: -561.4660
McFadden's pseudo-R²: 0.0348


I got two questions:
1 OffNo--> I believe the range is too narrow, so instead of [2,12,30], I would like to use [10,30,50]. In Ngene, shall I use -0.0004 as prior for final prior?
2 the signs of the parameters and the preference order of the dummy coded variables don't make sense. Now ele(d0,d1)=0.5945, 0.7241, it should be -0.5945,-0.7241.
is it because I set the code ele.dummy[-0.00001|-0.00003]*A[1,2,0] (in the Ngene) instead of ele.dummy[-0.00001|-0.00003]*A[2,3,1] (in the estimation model)? How to set the prior for dummy in final design?

[Michiel Bliemer]

1. Your prior for OffNo remains -0.0004 independent of the range of the attributes, so yes you would use -0.0004. Only when you use different units (e.g. minutes instead of hours) you need to scale the priors accordingly.

2. How did you implement ele.dummy[-0.00001|-0.00003]*A[2,3,1] in the estimation model?

In Ngene, ele.dummy[-0.00001|-0.00003]*A[2,3,1] is replaced by:
ele(d0) * A1 + ele(d1) * A2,
where A1 = 0 and A2 = 0 when A = 1
where A1 = 1 and A2 = 0 when A = 2
where A1 = 0 and A2 = 1 when A = 3
Please make sure that you correctly apply dummy coding in estimation. The values 1,2,3 are just symbols, just like 0,1,2, both are replaced with zeros and ones in dummy coded variables.