Number of attributes/levels and statistical power

[yuhan97]

Hello,

Since this is our first time designing a discrete choice experiment, there are some uncertainties arising during the design phase. Any comments or suggestions are welcome!

Q1
Our current design consists of three attributes, with 2, 3, and 3 levels respectively. However, the third attribute is only relevant to the second attribute. To mitigate the concern that the dependency between attributes will make the statistical analysis more complicated, we have decided to combine the second and third attributes, resulting in two attributes with 2 and 7 levels respectively (one level less due to restrictions on combinations). Is this a reasonable approach? Compared with the standard DCE design, ours is very small.

Q2
Since we have only 14 profiles in total and each time we present respondents with two alternatives, we have a total of 91 profile combinations. Furthermore, we expect 1000 respondents to participate in the survey. We have decided to include all possible combinations and use a random approach so that each respondent will receive 4 random combinations out of the pool. Is this a reasonable approach, or is it better to restrict the number of choice sets based on D-efficiency?

Q3
Furthermore, I want to ask whether the number of questions a respondent receives matters for the statistical power. Based on my reading, each alternative is considered as one data point in the analysis. That is, if I have 2000 respondents in the end, and each of them receives 2 questions, the statistical power will be theoretically comparable with 1000 respondents, each with 4 questions. Multinomial mixed logit will be relevant because we want to analyze the main effects of the attributes as well as interaction effects based on individual characteristics. The reason for asking this question is that our experiment is part of a big survey, and space is limited. Nevertheless, we do not want to compromise statistical power too much.

Thank you very much for your help and suggestions!

Best,
Yuhan Liu

[johnr]

Hi Yuhan

Below are my two cents, for what that is worth.

Q1: There are two ways that come to mind with respect to this question. One is to do it the way you suggest. The second is to use constraints to handle the issue. The way you suggest is probably the quickest and easiest approach, but it comes at the cost of not being able to put separate priors on attributes two and three, given that you have collapsed these into a single attribute. The theoretical intent of efficient design theory is to try and guess what your model will look like when you estimate it using data you have collected from the design you have generated. This is where it breaks down somewhat, as the utility specification cannot look like how it will in real life given how the attributes have been collapsed. Is this a bad thing? Probably not, particularly if you have a large sample size as you suggest in your next question. So, what you suggest is the quick and dirty way to solve the problem that at the end of the day, is likely to work (always pilot it to make sure).
The other way to do it is to use constraints. In your case, it might look something like

;cond:
If (AltA.B =[1,2], AltA.C= 0]),
If (AltA.B =[3], AltA.C= [1,2,3]),
If (AltB.B =[1,2], AltB.C= 0]),
If (AltB.B =[3], AltB.C= [1,2,3])

Here you have both attributes B and C separate (and so can provide different priors for each). The downside of this approach, is that often Ngene cannot find a design that matches the desired constraints and won’t give a design. This often occurs in relatively small designs such as what you describe as the number of possible attribute combinations is rather small and there may not be enough of them to generate a design. In such cases, you may be stuck with your first solution as the only way forward.

Q2: Blocking a design is always preferable to random assignment. If you randomly assign the treatments, you run the risk that the profile combinations present within your data appear an unequal number of times. This means that the properties of the design you generate don’t translate to the data (what you thought you were going to get, isn’t what you get). In large samples this doesn’t matter so much (not as much as people think), but it can introduce some biases depending on how it ends up. That said, many things can cause such biases, so from a practical perspective (assuming large enough samples), I wouldn’t be overly concerned, but from a purely theoretical perspective, the answer would be a no, don’t do it.

Q3: For the MMNL model, the model performs best when you have a larger sequence of choices per respondent. If you have only one or two choices per respondent, the model may struggle somewhat. You have to remember that utility has two components, observed and unobserved.

U_nsj = bx_nsj + e_nsj

If you make b random, then it needs to distinguish between random observed error (associated with the x’s) and random unobserved error. With only a small number of observations per person, this is difficult. So, econometrically, 2000 * 2 may not be the same as 1000 * 4, though on paper they may look the same. If you look at the log-likelihood function, there is a product within respondent, so in the 2000 * 2 case, the product multiplies two choice probabilities together, whereas in the later case it is multiplying four choice probabilities together. If it were a summation, I would agree with you, but it isn’t. I would recommend 1000 * 4 any day of the week, and twice on Tuesdays over 2000*2. But again, if practical reasons say you can only collect 2000 * 2, practical reasons trump all other considerations, and you do what you can do and work within those constraints.

John

[yuhan97]

Hi John,

Thank you so much for your quick and detailed response; it really helps a lot!

I have a follow-up question related to Q3: Before going into the field, we want to conduct a power analysis to see whether the design would provide sufficient statistical power for hypothesis testing. In other words, we would like to get a better understanding of how large the sample size should be given the current design. For this purpose, I would like to refer to one of your works, co-authored with Prof. Bliemer:

Rose, J. M., & Bliemer, M. C. (2013). Sample size requirements for stated choice experiments. Transportation, 40, 1021-1041.

In this paper, you discuss and compare different strategies to determine the minimum sample size. In the conclusion, you state that parameter estimates are necessary. To obtain these parameters, I reckon that conducting a prior study would be helpful. Do you have any practical recommendations for conducting a prior study (e.g., how large should the sample size for the prior test be)? Do you have other good references that we could use for a convincing power analysis?

Again thank you very much for your time. Any suggestions are highly appreciated!

Best regards from Germany,
Yuhan

[johnr]

Hi Yuhan

You might also wish to consult de Bekker-Grob et al. (2015) who extended our earlier work. With that out of the way, there are a few ways to proceed that have been tried and found to work in the past.

1. Use an orthogonal design for a pilot. This may or may not be feasible as there are limited numbers of orthogonal designs, and there may not been one available to match your exact problem. If there is, it turns out that under certain assumptions, an orthogonal design may be optimal. One such condition is if the parameters are all zero, so such designs are (again under a number of assumptions) equivalent to assuming the parameters are all simultaneously zero. Remember, that orthogonal designs have been used in choice analysis for over 40 years and have worked, and will continue to work. People assume that our promotion of efficient designs is a rally against orthogonal designs. This is not the case. I once used an orthogonal design for a pilot and based on 20 respondents, was able to estimate an MNL model with all the parameters significant and of the expected signs. I kept the design and didn't bother updating it.

2. You can assume very small prior values that capture the expected signs of the estimates. For example, you know that price should be negative - so why assume that it will be zero. You might assume it is -0.0001 for example. This may be enough to reduce choice tasks that are unlikely to capture you much information from.

3. Use your judgement as to what the likely parameters will be. Remember, if you knew them exactly in advance, then you wouldn't need to conduct the survey in the first place. You will never get them exactly right anyway. So you can use literature, or your own judgement.

4. Use a Bayesian design approach with uninformative priors. This is what I tend to do in practice. For some attributes, I have an expectation of the parameters (again, think price), but may not know the exact value it will take. In such cases, I may use a Bayesian Uniform prior U(-0.5,0) - where the exact values I use are scaled to account for the magnitude of attribute (i.e., I don't use U9-0.5,0) blindly). For parameters I don't know the sign of (e.g., categorical variables), I still use an uninformative prior but one that is not bounded at zero (e.g., U(-1,1). Again, the exact values I use depend on the magnitudes of the variables the prior is attached to. I typically do this rather than assume zero priors as I would argue that zero (fixed or local) priors are optimized for a particular value = zero). I hear people say use zero priors if you are unsure - I hate this argument - if you optimize for zero priors, you are basically saying you are sure of the value - it will be zero. Bayesian priors represent uncertainty by the analyst as to the exact value.

You might want to check out Bliemer and Collins (2016) who offer great advice on priors.

Now to get more to specifics of what I do (again - not saying I'm right, but the process I typically go through and which tends to work for me).

a) I typically have a pilot of between 20 and 50 respondents (although I have gone up to 100 but that maybe overkill). This is not just to get priors, but also to test logistics of the survey, allow them to make open ended text comments, etc.

b) Check the relative magnitudes of the average contributions of each attribute to overall utility...

Say I have U = b1[U(-0.5,0)]*x1[5,10,15] + b2[U(-0.8,0.2)]*x2[2,4,6]

If I look at the first attribute, the average parameter value will be -0.25, and the average attribute level is 10, hence the average contribution to utility is -2.5. For the second attribute, the average contribution to utility is -2. Hence, on average, I am assuming that the first attribute will have a larger impact on utility on average than the second. I have to ask myself, is this an assumption I wish to make?

c) Check the marginal rates of substitution I am assuming (WTPs for example) given the priors I have assumed. Do these make sense?

Following a), b) and c), I haven't had too many issues in the past.

John

EW de Bekker-Grob, B Donkers, MF Jonker, EA Stolk (2015) Sample size requirements for discrete-choice experiments in healthcare: a practical guide, The Patient-Patient-Centered Outcomes Research 8, 373-384.

MCJ Bliemer, AT Collins (2016) On determining priors for the generation of efficient stated choice experimental designs, Journal of Choice Modelling 21, 10-14

[yuhan97]

Dear John,

tausend thanks for the detailed suggestions!! They are very helpful!!

Best,
Yuhan