Efficiency measure and utility balance

[jonashl]

I have recently experimented a great deal with different ways of creating designs in Ngene and I have a couple of questions and concerns.
The designs in question all involve to unlabelled alternatives with some knowledge as to the sign of effects, and possibly expectations regarding the relative size of effects but not the absolute size of parameters.

1) Efficiency measures and utility balance?
Since the purpose is to elicit WTP, I have used the WTP efficiency measure together with uniformly distributed Bayesian priors. The resulting designs all resulted in a very high B estimate (98-99). The utility balance seems too high for a good design? If I use a D-efficiency criterion instead, the B-estimates drop to around 80, which seem more reasonable. Can you possibly explain why the wtp efficiency measure induces high utility balance, and whether to prefer the D-efficiency criterion on the grounds of a more reasonable utility balance (even though the target outcome is WTP ratios)?

2) Dummy vs linear coding?
If we expect a parameter to be linear but cannot be certain, is it then preferred to use dummy-coding – or is there a great loss in efficiency if it is in fact linear?

3) Using priors in the pilot?
I always pilot before final design, but to what extent is it a good idea to use priors in the pilot when sign, but not absolute levels are known? And if priors are not used and there are unlabelled alternatives is there reason to prefer an ood design over using a d-efficiency (or wtp) criterion?

Looking very much forward to your reply!

[johnr]

1) Efficiency measures and utility balance?
Since the purpose is to elicit WTP, I have used the WTP efficiency measure together with uniformly distributed Bayesian priors. The resulting designs all resulted in a very high B estimate (98-99). The utility balance seems too high for a good design? If I use a D-efficiency criterion instead, the B-estimates drop to around 80, which seem more reasonable. Can you possibly explain why the wtp efficiency measure induces high utility balance, and whether to prefer the D-efficiency criterion on the grounds of a more reasonable utility balance (even though the target outcome is WTP ratios)?

Kanninen (2002) JMR explored the optimal probabilities for the MNL model assuming D-error. She found

a. That for binary choice tasks, the optimal probabilties were around 0.72/0.28, however in a later paper, she found that this varies depending on the number of choice tasks.
b. That the optimal choice probabilities change as the number of alternatives changes.

Michiel and I have looked at other efficiency measures and found similar but different optimal choice probabilities, precisely what you are describing above. The optimal choice probabilities differ by effiency measure. So to answer your second question, the optimal balance is efficiency measure specific and you are best optimising on what is most important to you.

2) Dummy vs linear coding?
If we expect a parameter to be linear but cannot be certain, is it then preferred to use dummy-coding – or is there a great loss in efficiency if it is in fact linear?

If you mispecify the utility function, you will loose efficiency. Provided you have enough degrees of freedom and the desired effects are present within the design, you should be able to estimate the model, you will just need a larger sample size. Whether you loose more efficiency assuming linear effects when in fact they should be dummy, or dummy codes when they should be linear will be design specific.

3) Using priors in the pilot?
I always pilot before final design, but to what extent is it a good idea to use priors in the pilot when sign, but not absolute levels are known? And if priors are not used and there are unlabelled alternatives is there reason to prefer an ood design over using a d-efficiency (or wtp) criterion?

Some use orthogonal designs for the pilot with zero priors. I typically use expert judgement as you may know for example that price is negative, sowhy ignore that in the pilot where you are using a small sample (efficient designs come into the element in small samples, so it makes no sense to not use them when you are collecting the smallest sample you are going to have during a project). When I do this, I try to use Bayesian priors where I balance the contribution to overall utility so that I don't assume any one attribute will dominate, however this may not always be the best strategy in every case, particularly if you think one attribute will have a stronger influence. Re OOD designs, the answer is no. These are optimised under very strict sets of assumptions, using particular coding that you are not likely to be using. I would suggest if you are going to use zero priors, use zero priors instead of OOD.

[jonashl]

Thankyou for the response!
I've done some more experimenting with the code and noticed that for the WTP efficiency measure, the B-estimate appears to become lower if I make the price prior more uncertain (widen the uniform interval), which I think makes sense.

I have a question regarding the coding (I will post it as a new post) that I hope you'll get a chance to look at too.
Thanks