General questions regarding choice experiment design - Ngene

[prefer]

Dear programers and users of Ngene

Fisrt of all thanks for allowing me to join this team of experts in designing stated choice experiment (CE) studies. Eventhough I have some experience in analyzing choice experiment data, I didn't try designing my own experiment but now I am required to design a CE, and for that I am using Ngene and the manual is actually a comprehensive guide, which I found it important to begin with. However, while designing my CE, I came accross with some questions, and I list them below and I really appreciate your responses. The codes are included for reference.

1. As I understand, keeping the rdraws and rep properties, an MNL model can be used in the eff property to produce an rppanel design. However, can this design based on MNL model be used to estimate an EC model? And what about other models, e.g. latent class models, heteroskedastic logit models, and even a probit model?

2. Is it only the issue of time that is making designing an rppanel design difficult, or does the model generally can't converge?

3. Should I focus on s-estimates because sometimes I can see unreasonabily high s-estimates like in ten thousands?

4. when specifying dummy or effects coded attributes, should always one of the levels be a base? I raised this question because I have an attribute for which I want to see the effects of all the levels.

5. I produced a design with a cost attribute continuously specified, but I didn't follow the procedure for designs with continuous levels rather I just specified it as continuous level in design (see the code).

6. I am thinking of using a fractional orthogonal design for a pilot study and then use the estimates as priors to produce an efficient design. Does this generally cause a loss in efficiency of model estimates later on?


Design
; alts = alt1, alt2, alt3
; rows = 24
; eff = (mnl,d)
; rep = 500
; block = 4
; rdraws = halton(300)
; Model :
U(alt1) = b1[-0.01]
+nutrion.effects[n,0.3,021|n,0.18,0.06]*nutritive[2,1,0]
+food.dummy[n,0.2,0.03]*saftey[1,0]
+stuff.effects[n,0.01,0.01|n,0.021,0.002|n,-0.02,0.002]*feed[3,2,1,0]
+pinfo.dummy[n,0.6,0.0111]*information[1,0]
+env.effects[n,0.001,0.0008|n,0.0005,0.0003]*emission[2,1,0]
+cost[-0.001]*price[65,95,115,165]/
U(alt2) = nutrion*nutritive+food*saftey+stuff*feed+pinfo*information+env*emission+cost*price$


Kind regards
Mohammed
University of Copenhagen, Denmark

[johnr]

1. As I understand, keeping the rdraws and rep properties, an MNL model can be used in the eff property to produce an rppanel design. However, can this design based on MNL model be used to estimate an EC model? And what about other models, e.g. latent class models, heteroskedastic logit models, and even a probit model?

Yes. To date we have only done MNL, MMNL and EC models. We are not likely to do the probit model anytime soon given the math. The LCM and heteroskedastic MNL we have the deriviatives but they are not available at the present.

2. Is it only the issue of time that is making designing an rppanel design difficult, or does the model generally can't converge?

It is typically a time issue. Convergence issues mostly arise with the RP cross sectional model due to identificiation issues, as a result of trying to seperate out random and error effects which would occur in empirically. Identification issues can also arise in the panel model, but these are typically analyst induced identification where you are attempting to estimate too many effects (the cross section model also suffers from these types of effects).

3. Should I focus on s-estimates because sometimes I can see unreasonabily high s-estimates like in ten thousands?

Given that s = var(beta)*t^2/beta^2, you will get larger s-estimates when the beta is close to zero. This makes sense given that you need larger sample sizes to detect differences from zero when the parameters are closer to zero. The s-estimates are therefore particularly sensitive to the priors you assume. If you use loose Bayesian priors that cross zero, then it will take draws close to zero (or zero), and given that the Bayesian error is averaged over draws, and averages get pulled to outliers, the s-error will explode. You need to therefore use expert judgment on this one. If the parameter really is close to zero then the s-error is what it is. if it is because you have used loose priors, then it is an artefact of that.

4. When specifying dummy or effects coded attributes, should always one of the levels be a base? I raised this question because I have an attribute for which I want to see the effects of all the levels.

One level has to be the base for identification purposes. You can work out the base effect which will be unique if you use effects coding - so you can obtain all effects - they are relative to the base so if you know the L-1 effects, you can work out the base effect. If you mean that you want absolute rather than relative estimates, then this is not possible. It is one of the critiscisms of DCEs offered by Louviere (see Louviere and Islam 2008 for example) which led to the development of best-worst scaling.

5. I produced a design with a cost attribute continuously specified, but I didn't follow the procedure for designs with continuous levels rather I just specified it as continuous level in design (see the code).

This is the most common approach to designing DCEs. The use of 'continuous levels', ala Kanninen (2002) is possible but not (yet) common in the literature.

6. I am thinking of using a fractional orthogonal design for a pilot study and then use the estimates as priors to produce an efficient design. Does this generally cause a loss in efficiency of model estimates later on?

Yes, see my previous post about this - efficient designs are specifically designed for small samples. It makes no sense to me to use a non-efficient design when you have the smallest possible sample. But if you have absolutely no idea about the priors, then you may be stuck with doing this.

[prefer]

Dear John,

Thanks for your answers, really nice.

I am a PhD student and my phd entirely depends on a design that I am going to produce, and I want to analyze my data (collected later on) using different models, may be including probit models. So, in this case, if I have an efficient design, then I won't have a flexbility of employing different models later on, so don't you think fractional orthogonal designs are more appropriate than efficient designs in such cases? I know that the decision is partly dependent on the sample size - but I am sure I will have enough sample (like 500 or more).

Kind regards
Mohammed

[johnr]

Hi Mohammed

An efficient design does not preclude the estimation of model types that you have not optimised on. Orthogonal designs may under certain situations be considered optimal for MNL models with zero priors, and when viewed in this way may be considered an efficient design themselves. You can still estimate a Probit model with a model generated for a logit model (as I said, an orthogonal design is nothing more than an efficient design itself in this context, so if you can estimate a Probit on orthogonal designs, it stands to reason, you can do so also for an efficient design - also, Probit models and logit models produce similar results depending on the error structure assumed). You can always take a MNL design and apply it to a MMNL logit, and in the same way, you can apply it to a Probit.

There are other reasons to prefer efficient designs over orthogonal designs in choice experiments beyond sample size as well that should be considered. Orthogonal designs by definition are concerned with the attribute level combination pairings and not the results of the intended model. It turns out that for MNL models, if you assume zero priors, an orthogonal design will (generally) be efficient, but that is because the model probabilities in such a scenario collapse to a constant 1/J where J is the number of alternatives and the model approximates a linear model. Putting that aside, the problems are

1. Unless one is deliberately setting up an experiment to fail, nobody would design an experiment in which they believe the parameters are truly going to be zero in reality - this is equivalent to stating that you have selected a set of attributes that do not affect the decision process. I get that people argue that they may not have priors, or even know their signs, but then why not use Bayesian priors that span both signs.

2. As I stated above, orthogonal designs by definition are concerned with the attribute level combination pairings and not the results of the intended model. This means that you can get lots of dominated alternatives in the design. This has implications for scale and possible model separation issues which need to be dealt with post data collection, which rarely if ever are actually ever dealt with. Depending on how you generate the efficient design, it should try to obtain questions that will force people to trade-off - that is maximising the amount of information that is obtained from each question. This may not always be possible due to user imposed constraints such as attribute level balance which may result in situations where you have one or two silly questions which you cannot improve without making one or more other questions worse off in terms of efficiency and hence affect the overall efficiency of the design, however this is caused by the user, not the theory.

John

[prefer]

Hi John,

Thank you very much indeed for spending your time explaining my questions, I feel completely comfortable with your explanations.

Kind regards
Mohammed