Interpreting efficiency measures

[Andrea]

Dear all,

I am creating an efficient MNL model based on the priors of a pilot study (orthogonal MNL):

Code: Select all

;alts = scen1*, scen2*, none    ? scen1* => * means alternative is generic!
;rows = 16 ?32
;block = 2 ?4
;eff = (mnl,wtp(wtp1))   
;wtp = wtp1(*/b_cos)
;model:
U(scen1) =
b0[0.5] +
b_for[-0.00639] * FOREST[0,20,40,60] +
b_set.dummy[-0.780|0.149|-0.978] * SETTLE[1,3,4,2] +
b_flo.dummy[0.0817|0.108|0.442] * FLOOD[1,3,4,2] +
b_cos[-0.124] * COST[-2,0,2,4] /
         
U(scen2) =
b0 + b_for * FOREST + b_set * SETTLE + b_flo * FLOOD  + b_cos * COST
$


It is my first efficient design, so I am not sure about the interpretation of the efficiency measures. Several questions arised:

1. Which D-errors and A-errors indicate a good / tolerable / bad design?
2. Is there a way to set a "maximal utility" value, e.g. B-estimate <= 90?
3. The Ngene Manual states that the number of necessary choice tasks in efficient designs is much lower than in orthogonal designs. I created two design versions, the only difference being the number of choice tasks and blocks: 32 CT in 4 blocks (S-estimate = 299) vs. 16 CT in 2 blocks (S-estimate = 613). Is this large difference caused by the two dummy attributes I use?
4. How do I interpret the terms WTP estimate and WTP n? The values are very similar to the S-estimate. Do they also specify the minimum necessary sample size?
5. Adding a no-choice alternative changes the choice probabilities. Is there any way to minimize the probability of selecting the no-choice alternative?

Best regards,
Andrea

[Michiel Bliemer]

Dear Andrea.

To answer your questions:

1. The A and D-error measures are case specific, so in one model a D-error of 0.1 is very good, in another model it is very bad. The lower the better, that is all we can sau. Typically the S-error provides a good indication of the efficiency. Again, some models require a sample size of 100 to obtain statistically significant parameter estimates, and other models require perhaps a sample size of 5.
2. I am not sure what you mean with "maximum utility" value. I am not sure why you would want to do this. Perhaps you are afraid of alternatives in which the difference between utilities is too high, so an alternative becomes dominant? An efficient design should be able to find good trade-offs in the utilities by default.
3. The S-error (sample size) is over a complete design replication. So in your case you need 613 replications of 16 choice tasks, so 2x613 respondents if you block the design in 2. Alternatively, you will need 299 replications of 32 choice tasks, so 4x299 if you block the design in 4. Both give about the same sample size. This is a very high sample size by the way, probably because your b_for parameter is very small, therefore hard to estimate.
4. The S-estimate for the WTP is indeed the sample size required to get the WTP statistically significant (from zero) at the 95 per cent level. Note that the S-estimates should be seen as lower bounds, you may need a larger sample size.
5. Minimizing the probability of the no-choice can only be done by making the other alternatives more preferred. You can do this by changing the attribute levels. I am not sure you want to do this. Something that we often so to avoid that too many people choose the no-choice alternative, is to ask two questions. First, ask which alternative is preferred including the no-choice. In case they choose the no-choice, ask a second question, namely which alternative they prefer if they have to choose one of the alternatives (and not the no-choice). This ensures that you get sufficient data out of each choice task. If most people choose the no-choice, then little information if captured and you may not be able to estimate all your parameters. So by asking the second question, you overcome this problem. You can use the answers to both questions in your estimation (combined or separate).

Michiel

[Andrea]

Dear Michiel,

Thanks a lot, your answers were very helpful.I have some following questions:

2. Actually I meant "maximum utility balance", sorry. As far as I know, a utility balance of ca. 70-90% would be ideal. The design I generated has a value of >95, so I was concerned that this would make it difficult for people to choose.

3. It is indeed possible that people don't have strong preferences for the FOREST attribute. I tried
a) dummy-coding the attribute and genereating an efficient design based on the new priors and
b) estimating a model without FOREST and generating an efficient design based on those priors.
However, a) increased the sample size very much (just what I expected) and b) is not able to generate a design ("A random design could not be generated after 2000000 attempts"). Do you have any suggestions how to handle this problem?

5. That's a great tip, I will do that.

Best,
Andrea

[Michiel Bliemer]

The utility balance is a result of the optimisation of the D-efficiency. Typically, a D-efficient design with two alternatives will indeed result in a utility balance of 70-90 per cent. However, you have three alternatives, so the optimal utility balance may not translate to three alternatives. I looked at the efficient designs generated with your syntax, and while resulting in a utility balance of 92 per cent, the choice probabilities of each choice task have a nice spread over different alternatives, they are definitely not 33-33-33%, so I do not suspect any problem there. The designs generated are efficient (of course relying on your priors).

The flood dummies are responsible for the high required sample sizes, not the forest attribute, so I am not sure why you want to change the forest attribute. You mention you changed the priors and then Ngene was not able to generate a design anymore. This is likely due to "bad priors", did you inflate the forest parameters? Or make any other significant changes? Bad priors may yield dominant choice tasks due to a dominant attribute, such that Ngene cannot find any good design. So I suspect it is a problem with your priors, they have to make sense. You can check how much each attribute contributes to utility by computing beta * attribute(average) for each attribute and see if one attribute dominates the utility. That means the priors for this attribute are too high (and unrealistic).

Michiel

[Andrea]

Thanks - I found one or two 32.x-33.x-33.x% values and wasn't sure about the interpretation, but most of the values are completely ok.

You are right about the forest attribute, I noticed my mistake the next day (working late is not always the best idea). The original forest priors are fine. The flood priors are indeed poor, probably because of a poor attribute level description in the pretest and also because the choice experiment itself is very complex or because opinions may be spread over a wide range. I cannot repeat the pretest, so I will have to deal with the poor priors.

I changed the b_flo(d0) and the b_flo(d1) to Bayesian instead of fixed priors, using first a normal, then a uniform distribution and a std dev. of 0.3. The normally distributed priors didn't enhance the design, but the uniformly distributed version (see code) seems to work.

Code: Select all

 b_flo.dummy[(u,0.0817,0.3)|(u,0.108,0.3)|0.442] * FLOOD[1,3,4,2]

After running the above syntax for a few hours the Sp estimates are for b_flo(d0) = 58.46 and for b_flo(d1) = 67.95, the overall S estimate = 78/120/58 (fixed, mean, std dev.) which is very nice compared to the original numbers of b_flo(d0) = 299 and b_flo(d1) = 171. Do you think that this is a valid approach or am I spoiling my design with "worsening" the priors?

Best,
Andrea

[Michiel Bliemer]

Apologies for the late reply.

You say that you get nice results for the S-estimates by changing your priors. Note that this does not necessarily improve your design, it could even make your design worse if your priors are far off from the correct values. The most important thing is that the priors make sense, and are as accurate as possible (usually by means of a pilot study). So please be careful in selecting priors and drawing conclusions that you have found a better design, because this depends on the assumption of correct priors. Clearly, uncertainty is taken into account in a Bayesian design. A Bayesian design is more robust against prior misspecification than a locally optimised design, therefore I believe you have done the right thing. Just be careful drawing conclusions on which design is 'better'.

[Andrea]

Dear Michiel

In your first response to my thread (question 5) you recommended me to ask two questions to get sufficient data from the experiment. I did that and got great results by using a combined choice variable (Rho-square increased by a factor of 10 compared to the first-choice only model). Is there any literature about this method that I can refer to? I didn't find any publications - maybe I just used the wrong search terms.

Best,
Andrea

[johnr]

Hi Andrea

One thing you may wish to do is ask two questions per choice task, one with a forced choice and one without. The following paper reviews the literature on this topic and discusses the pros and cons of doing so as well as modelling issues.

Rose, J.M. and Hess, S. (2009) Dual Response Choices In Reference Alternative Related Stated Choice Experiments, Transportation Research Records, Paper #09-2432, Vol. 2135, 25-33.

From a design perspective, you can use the model averaging approach to optimize designs for this type of task where you have one model with the no choice and one model without.

John

[Andrea]

Hi John

Thanks for the quick reply, the paper is very helpful!

Andrea