Ngene hangs+ huge S estimate

[Sara]

Dear Ngeners,

I have been facing with problem with Ngene since yesterday. It keep hanging almost 2 out of every 5 times I click to see output window.

Second problem is with huge S estimate, however, I put the priors (significant and correct one) from pretest and ((u,-1,0) for insignificant or variables with incorrect signs) , I keep getting either very huge S-estimate or all the MNL efficiency measures are unidentified! why it is happening to me? I am using Pivot design, had 60 respondents for short trip pretest and 60 for long!

Thanks
Sara

[Michiel Bliemer]

Dear Sara,

It is difficult to say what is the problem without having the Ngene syntax. Could you perhaps share the syntax so we could replicate the problem?

Thanks,
Michiel

[Sara]

Hi Michiel,
Many thanks. Design is as follow. I put the priors (significant and correct one) from pretest (obtained from MNL model), (n, mean, standard error) for less insignificant but correct signs, and for those variables which are very insignificant or having wrong sign I put (u,-1, 0) in the main design.

Those decimals in levels are eventually percentage change, but when I use integers I get error.

I also tried to separately design Long and Short segments using Bayesian efficient design. Or substituting all actual levels with 1,2,3. I'm still getting huge S-estimates and also attributes level balance is another issue. With 2 blocks in design some of the levels are repeating 3 times in one block but only once in another block.


Design
;alts(short) = car, train,bus
;alts(long) = car, train,bus
;rows = 12
;block=2
;eff = F1(mnl, d,mean)
;fisher(F1) = des1(short[0.45]) + des2(long[0.55])

;model(short):

U(car) = b20[(u,-1,0)] + b1[(u,-1,0)]*HC[0.25,0.50,0.75] + b3[-0.22] * TVC[0.25,0.50,0.75]+ b4[(u,-1,0)]
*toll[0.33,0.66,1]+ b5[-0.29] *CP[3,5,7]+b6[(u,-1,0)] *CF[2.5,3.5,4.5]/

U(Train) = b22[(n,2.76,2.11)] + b7[-0.025] * H[-0.75,-0.50,-0.25] + b8 [-1.20] *Ac[1,2,3]+ b9[(n,-0.11,0.11)] * FT[5,7.5,10]+
b10[-0.19]* TVT[-0.40,-0.30,-0.20] +b11[(n,-0.28,0.19)] *CT[1.5,2,2.5]+ B16[(u,-1,0)]*TCOMF[1,2,3] /

U(Bus) = b12[(n,-0.012,0.010)] * HB[-0.50,-0.25,-0.125] +b13[-1.63]*acb[1,2,3] + b16[(u,-1,0)]*Fb[5,7.5,10] + b14[-0.26] * TVB [-0.30,-0.20,-0.10]
+b15[(u,-1,0)] *CB[0.5,1,1.5]+B16*TCOMF

;model(long):

U(car) = b20[-4.29] + b1[0.02]*HC[0.25,0.50,0.75] + b3[-0.087] * TVC[0.25,0.50,0.75]+ b4[(n,-0.20,0.13)] *toll[0.33,0.66,1]+
b5[-0.16] *CP[3,5,7]+b6[(u,-1,0)] *CF[2.5,3.5,4.5]/

U(Train) = b22[(n,0,1)] + b7[(u,-1,0)] * H[-0.75,-0.50,-0.25] + b8 [-1.50] *Ac[1,2,3]+ b9[(n,-0.19,0.13)] * FT[5,7.5,10]+
b10[-0.05]* TVT[-0.40,-0.30,-0.20] +b11[(n,-0.36,0.27)] *CT[1.5,2,2.5]+ B16[(u,-1,0)]*TCOMF[1,2,3] /

U(Bus) = b12[(u,-1,0)] * HB[-0.50,-0.25,-0.125] +b13[-1.46]*acb[1,2,3] + b16[(u,-1,0)] *Fb[5,7.5,10] +
b14[-0.083] * TVB [-0.30,-0.20,-0.10]+b15[(u,-1,0)] CB[0.5,1,1.5]+B16*TCOMF$



God I feel relieved now knowing I 'm going to get some help. I’m really thankful.

Sara

[Michiel Bliemer]

Just having a quick look at your syntax, I observe a few things.

1. There needs to be a multiplication * between b15 and the CB attribute in the long model.
2. You have a lot of Bayesian priors, please consider using larger numbers of Bayesian draws than leaving it to the default, e.g. ;bdraws = sobol(2000)
3. You have combined priors that you know quite accurately (e.g., b3[-0.22]) with priors for which you do not know much (e.g., b6[(u,-1,0)]). Note that you are multiplying b6 with relatively large numbers, ranging from 2.5 to 4.5, while b3 is multiplied with relatively small values ranging from 0.25 to 0.75. This means that attribute CF will become more dominant when you do a draw of b6 close to -1. Please note that setting priors should be done with extreme care. if you only know the sign of a variable but nothing much else, you could for example put in b6[(u,-0.1,0)], which makes the prior much closer to zero and hence make it has less impact.
4. You will get very large sampe sizes if your priors are very close to zero. For example, b12 is very close to 0, so this is likely a difficult to estimate parameter. So make sure your priors are realistic and make sense, only under this assumption the sample sizes make sense.

I have not run your syntax yet as I have no computer currently available running Ngene, I will try that tomorrow.

[Michiel Bliemer]

I also noted that b16 appears twice in the utillty function as b16 and B16.

Regarding blocking, note note blocking for non-orthogonal arrays does not really exist. So for efficient designs, blocking is only approximated by looking for a way of blocking the design that makes each block AS MUCH AS POSSIBLE attribute level balance. But it will not be able to exactly do this for each attribute. The only way to do this is to use an orthogonal design, but usually the blocking of an efficient design is not too bad.

I am trying to narrow down the problem why the S-errors are so bad. I am just focussing on the first model, which seems already problematic. I will look further into it.

Code: Select all

Design
;alts = car, train,bus
;rows = 12
;block = 2
;bdraws = sobol(1000)
;eff = (mnl,d,mean)
;model:
U(car) = b20[(u,-0.1,0)] +
         b1[(u,-0.1,0)] *HC[0.25,0.50,0.75] +
         b3[-0.22] * TVC[0.25,0.50,0.75]+
         b4[(u,-0.1,0)] * toll[0.33,0.66,1]+
         b5[-0.29] * CP[3,5,7]+
         b6[(u,-0.1,0)] * CF[2.5,3.5,4.5] /

U(train) = b22[(n,2.76,2.11)] +
           b7[-0.025] * H[-0.75,-0.50,-0.25] +
           b8 [-1.20] * Ac[1,2,3]+
           b9[(n,-0.11,0.11)] * FT[5,7.5,10] +
           b10[-0.19] * TVT[-0.40,-0.30,-0.20] +
           b11[(n,-0.28,0.19)] * CT[1.5,2,2.5] + 
           b16[(u,-0.1,0)]*TCOMF[1,2,3] /

U(bus) = b12[(n,-0.012,0.010)] * HB[-0.50,-0.25,-0.125] +
         b13[-1.63] * acb[1,2,3] +
         b17[(u,-0.1,0)] * Fb[5,7.5,10] +
         b14[-0.26] * TVB[-0.30,-0.20,-0.10] +
         b15[(u,-0.1,0)] * CB[0.5,1,1.5] +
         b16*TCOMF
$


[Sara]

Thank you so much Michiel, I am trying it as well. Unfortunately I keep getting errors. The results of short model from pretest wasn't that good. One more thing, in our pretest, TVC (car travel time) is pivoted around current reported travel time, and TVT and TVB pivoted around TVC. All attributes also are highly correlated.

Thank you again

Sara

[Michiel Bliemer]

Ok we have looked into it, and we narrowed it down to two problems.
First, some of your Bayesian priors with normal distributions have quite large standard deviations, and as such it may draw from a quite extremely value of the distribution, making the attribute dominant and hence no model can be estimated.
Secondly, some of your priors are quite large (such as b5), as it multiplies a large value such as 7. This makes this attribute quite dominant, and again the model becomes more difficult to estimate.

So the main issue is in your selection of the priors, they do not really seem to make sense. When priors are not realistic, then the resulting S-estimates do not really have any meaning. Your design may still work well, but it is hard to determine how efficient it is as the priors are not very good.

Are you able to to a pilot study and obtain useful priors? Alternatively, I would be quite conservative in setting the priors, in which you set them all close to zero (and ignore the S-estimates, as they have not much meaning when setting them close to zero).

[Sara]

Thank you Michiel,

You cant believe how data collection for choice experiment is difficult here. Response rate is almost only 10% and the time I was doing pretest I wasn't very sure whether respondents really trade off. This pretest took almost 3 months for us. But since I myself collected the data and not survey company or etc, I am pretty sure even though I spend another 3 months for second pretest the results wont be much different. But thank you so much, we will try to come up with some solution.

Very bests
Sara

[Michiel Bliemer]

I typically bug my colleagues or students and let them fill out heaps of choice tasks to obtain some initial data. It at least provides some values, but still I would be conservative with the values I get from such a rough pilot study (for example, take half of the values that come out of estimation).

[Sara]

Thanks, that 10% response rate is also correct for colleagues, what to do? . However, since most of them pay same toll and zero parking fee, we were afraid all the answers would be very similar. I am thinking of having few designs and some of attributes only included in one design and not rest. Doing so I think respondents might face less attributes and it would be easy to them to trade off. We tried vouchers and gifts but it didn't work because most of people do not allow us even to approach them or open our mouth then reject us. When I searched the reason seems its because of conservative nature of people here. Any way please wish me luck. I really need it. I will happily invite your all team to come here and experience it on your own, you might come up with some genuine idea which could be applicable for rest of developing countries.

Sara