choice-metrics.com

Hello. Having conducted mostly marketing DCE's in the past, I am designing my first one for environmental valuation. Here is the design syntax I have developed to start out, based upon my reading of literature, Ngene documentation, and this forum:

Code: Select all

Design
;alts = alt1*, sq*
;rows = 24
;eff = (mnl,d)
;block = 6
;con
 
;model:
U(alt1) =     B1[-0.00001] * Price[10,20,40,60,80,100] +
                  B2[0.00001] * Buffer[40,60,80] +
                  B3[0.00001] * Quality[75,80,85,90] +
                  B4[0.00001] * Income[80,90,100,110] +
                  B5[0.00001] * Jobs[80,90,100,110] +
                  B6[0.00001] * Wildlife[85,90,95,100] +
                  B7[0.00001] * Infrastructure[85,90,95,100] /
 
U(sq) =       B0[-0.00001] +
                  B1 * Price2[0] +
                  B2 * Buffer2[25] +
                  B3 * Quality2[75] +
                  B4 * Income2[80] +
                  B5 * Jobs2[80] +
                  B6 * Wildlife2[85] +
                B7 * Infrastructure2[85] $

I plan to conduct a pilot of approximately (n=200) responses to estimate betas for use in a Bayesian efficient design. The ultimate plan is to collect 2,000 responses for the final SP survey. Now, I am specifying my model and creating a design for use in the pilot. In past marketing studies, I have used an OOD design for the pilot study. In this case, I have been unable to find an OOD design with Ngene, and based on the forum reading I gather this is not a surprise given the number of attributes and levels that I have. Bliemer's recommendation to others has been to go ahead and use an efficient design rather than an orthogonal design for the pilot study, and to specify low-information priors (i.e. 0.00001 or -0.00001) if possible. This also has the benefit of reducing the number of rows needed, if I read correctly.

A few more details about the attributes/levels:
The status quo (SQ) alternative has levels representing outcomes of groundwater services in the year 2050, with the groundwater policy management alternative presenting counterfactual outcome levels for 2050. Values for the levels are all percentage values (continuous) with larger percentages being more desirable.

I would like to present each respondent with 4 choice sets, preferably, and no more than 6. With 7 attributes, I know each question is relatively heavy on cognitive load. Can I get away with 24 rows (or possibly even fewer) and still be able to estimate parameters with significance? Am I correct to be pursuing an efficient design now, rather than an orthogonal design, for the pilot? I would certainly say that unbiased priors are important to me, but ultimately I can't say that unbiased estimates are MORE important to me than low standard errors (that is my understanding of the different advantages between orthogonal and efficient, respectively). I would like to be able to get "good" estimates with as-small-as-possible standard errors with a sample of 2,000, and I would hopefully like to be able to divide that 2,000 between 4 to 6 different treatments.

I would appreciate any help I can get in evaluating my model specification and any suggestions about what practices might be best to incorporate into my design.

Thank you,
Grant West
gwest@uark.edu

Hi Grant

Let S = number of tasks, J = number of alternatives and K = Number of parameters (including constants). To invert the Hessian, S => (J-1) * K. In your case, 24 => (2-1)*8, suggesting all else being equal, you have enough degrees of freedom to estimate the model.

With respect to your second point about biased estimates, I've heard this recently too. It is as far as I can tell a falsehood being spread by people with an agenda. Lets say that you have a problem where you need to generate a design with 1 billion possible combinations (a conservative number - most designs will have more than this). Lets assume that there are say 100 possible orthogonal designs that can be constructed out of the 1 billion potential designs. Are you suggesting that these 100 designs will produce unbiased estimates whilst the remaining 999,999,900 designs will all cause biased estimates? I am yet to see a formal proof of this, empirical or mathematical. We have conducted millions of simulations (when Michiel and I started working on design theory, the first thing we did was run 4 million simulations over the summer break, and we have run millions since). I am yet to see evidence of this phenomenon. The onus is on those making the claims to substantiate this formally.

These are non-linear models - look at the exponential in the probability formula. Properties of these models are not the same as linear models. Orthogonality is not the same - if you truly want an orthogonal design for these types of models, you would generate a design that is orthogonal in the differences (across alternatives, accounting for parameters also), not a design that is orthogonal in absolute levels. Consider what is happening with the model compared to linear models. Lets start with linear models.

We know that B = (X'X)^-1X'Y and VC = sigma^2(X'X)^-1.

We can look at the properties of the matrices to see the impact of the design on the estimates B, and VC matrix. If X is orthogonal, then the story goes, you get unbiased estimates, B, and a good VC matrix (small values on leading diagonal, and zero values on the off diagonals).

Now we don't have an equivalent OLS estimator for DCMs to get B. We use maximum likelihood to do this. We do have the equivalent for the VC matrix however. For the MNL model, let Z = (x_jk-sum_j(P_jk.x_jk)*sqrt(P_j). Now the VC = (Z'Z)^-1.

You can see the link with the linear model, where we replace X with Z basically. This also is also where the answer to your first question came from. We need to invert the Hessian which is the (negative) inverse of the VC matrix. That is, we compute Z'Z, and we need to invert this matrix (same for (X'X) in the linear model). Now at a minimum, the number of rows of X (or Z) need to be greater than or equal to the number of columns for the resulting inverted matrix to be non-singular (there are other conditions, but this is a good start). Now in the linear model, the number of rows are = the number of observations and the number of columns = the number of parameters, K.

Now if you look at Z, each row is an alternative in this matrix. If you have more alternatives, you have more rows and as per above, as long as the number of rows (alternatives * choice observations) => K, you have the start of an invertable matrix.

If you want (model) orthogonality, then you want the matrix (Z'Z) to be orthogonal, not (design) orthogonality in X.

So what can we say about the betas (getting back to your comment about biased estimates)? Not much. We know that there is no algebraic equivalent to OLS regression. We also know that if there were, it would be highly non-linear (like the VC matrix). The estimates would somehow be linked to the probabilities which is an exponantial (hence why we call these non-linear models - they are non-linear in the probabilities) of utility (the parameter estimates and the design). So to conclude, there is no proof that non-orthogonal designs induce biased estimates. Indeed, the opposite (given the non-linearity of the model) is most likely if the parameters are non-zero (I say most likely, as I don't have a formal proof).

John

John,

Thank you for debunking the rumor of orthogonality and unbiased betas and laying out the econometric theory behind it. Practically speaking, should I interpret this to mean that an efficient design is appropriate here? Based on my current knowledge, orthogonality is not important to me for any particular reason. What other guideposts can I use in helping me to determine the best design to pilot? It seems as though I would still have enough D.O.F. with a 12-row design. Should I be conducting sensitivity analyses with respect to betas (for testing sample size issues) or perhaps use uniformly distributed priors for the pilot design? Is this something that might help me to decide between designs with differing numbers of rows?

Regarding the ASC for SQ, I've seen repeated findings in the literature of negative utility associated with the status quo option based off of the levels representing a trajectory of decline. Is this directional assumption appropriate to make at this point or would I be safer with a zero prior for the ASC?

Also regarding ASC's, is there any value to conducting a labeled experiment when presenting only a single policy alternative and a status quo? I am considering treatments where the groundwater management policy mechanism is different for each treatment. Would it change things if I presented [Status Quo Alternative vs. Managed Aquifer Recharge Alternative] rather than [Status Quo Alternative vs. Generic Management Alternative]? Within the treatment, each management alternative presented would always be labeled the same way (i.e. always "Managed Aquifer Recharge Alternative"). I'm not sure this constitutes a case where labeling is a relevant and unobserved driver of choice, though I could be convinced that it is. If it is, would that make it appropriate to label it and include an ASC for "Managed Recharge"? In that case, I would only be able to compare ASC's in a pooled model, correct (from other treatments for "Irrigation Efficiency" and "Surface Water Infrastructure")? I may be way off here, but I haven't seen many analogues in the literature that would help me with my thinking on this. My guess is that probably means there's little added value to labeling when it is the only management alternative presented. is it then just a treatment effect in the modeling?

Thank you again, John, for your thorough answer about orthogonality and the misconception about resulting betas. I'm hoping you might also elaborate on what other steps and changes you might make in specifying this model/design if you were preparing this for a pilot study.

Thanks,
Grant

I can confirm John's point that both orthogonal and efficient designs both lead to unbiased parameter estimates. Orthogonal designs have no benefits in estimating choice models over efficient designs. Under certain conditions, an efficient design with zero priors is actually orthogonal or near-orthogonal.

Note that sample size estimates in Ngene are only useful when you have specified good priors (i.e. coming from a pilot study). Using uninformed priors close to zero will not result in any useful sample size estimates and can therefore be ignored. If you use priors close to zero then it is not so important to assume randomly distributed priors. Using Bayesian priors with such a random distribution is more important if you are using priors further away from zero.

A typical strategy is:
1) Create an orthogonal or efficient design with zero priors (or small positive/negative values in case you want Ngene to remove dominant alternatives) for a pilot study
2) Use data from the pilot study to estimate the parameters b and obtain standard errors s
3) Use normally distributed priors with mean b and standard deviation s to create a Bayesian efficient design for your main study

Unlabelled experiments are useful for valuation studies, while labelled experiments are useful for predicting market shares and elasticities, but can also be used for valuation studies. Unlabelled experiments are usually simpler, while labelled experiments allow estimating alternative specific parameters. So it depends on your research goal what is the best approach.

You may want to post these more general experimental design questions (that are unrelated to Ngene) in the other forum.

Michiel

Thank you. I will keep my questions venue-specific.