Size of choice experiment

[amela_95]

Hello,

I have a question regarding the design of my experiment.

So, my experiment has 2 unlabeled choices to choose from, both having 9 attributes with two levels (full fractional universe: 512). I have decided to use a fractional factorial design with orthogonal optimal in the differences (OOD) design, and I'm now debating on how to choose the appropriate size of the choice experiment. I want to include blocks, so that each respondent evaluates 4 choice tasks. I tried to run it with either 40 choice sets (=10 blocks), or 48 choice sets (=12 blocks).

Now my questions are the following:

- Is there a way to determine which of these two options is better? I aim to estimate main effects and two-way interactions without confounding. It appears feasible in both cases. Is that correct?
- How should I interpret the correlations with blocks? Is this a potential threat?
- If I increase the blocks to 5 choice tasks per respondent, it seems like I would need to raise the total number of choice tasks to 80. However, I encountered an issue in Ngene ("index is out of range") when looking at the interactions. Is it worthwhile to increase the number of choice tasks per respondent, or is 4 tasks per respondent sufficient? I'd prefer to keep the total number of choice tasks relatively small since I have to generate them manually due to using another survey software.

Thanks so much and kind regards,
Amela

[Michiel Bliemer]

If you use ;orth = ood then it creates an orthogonal design with minimal overlap, i.e. attribute levels will always be different across the two alternatives. Please make sure that there is no dominant attribute across the 9 attributes as otherwise you would not get any trade-offs on the other attributes. A design generated using ;orth = ood only considers main effects. If you want to make sure that you have no correlations between main effects and interaction effects, you can add ;foldover to your script, which doubles the number of choice tasks in such a way that it creates such zero correlations.

You could also consider explicitly adding the interaction effects and use ;eff = (mnl,d) using zero priors to generate the design, which guarantees that all main and interaction effects can be estimated. Note that you do not need zero correlations to estimate such effects, you merely want to avoid perfect correlations.

Because you have 9 attributes, the number of interaction effects is very large and the matrix with correlations across interaction effects is huge, too large to display and therefore Ngene does not generate it.

I would say that 4 choice tasks per respondent is sufficient.

Michiel

[amela_95]

Dear Michiel,

Thanks so much for your response.

I'm not sure if I understood it correctly. So, In my current design, if I use OOD and create 48 choice sets with 12 blocks, then the "interaction" table in Ngene shows me some correlations between 0.333 and - 0.333 for various interaction effects. But since there is no perfect correlation, I can still estimate main effects and interaction effects even though I am using an OOD design and do not include the foldover or ;eff command. Is that correct?

All my other correlations are zero, except correlations of "block" with all attributes (X1, X2, ..., X9), and the interaction block * X1 correlates for example with X2. The correlations are rather small and not perfect. I'm I correct to assume that this is also not a problem?

Thanks so much and all the best,
Amela

[Michiel Bliemer]

Yes your understanding is correct, if it shows something like 0.333 then there is no perfect correlation and you can include and estimate the interaction effect. But if there exists two interaction effects, such as A*B and C*D, that are perfectly correlated, then only one of those interaction effects can be included and estimated in the model.

Correlations with the blocking column are not really important. If they indicate zero then it means that blocking is orthogonal, which typically means perfect attribute level balance within each block, while if they are non-zero then it means non-orthogonal blocking, i.e., no perfect attribute level balance within a block. But attribute level balance within a block has no impact on model estimation, it is merely a desirable feature.

Michiel

[amela_95]

Perfect! Thanks so much for clarifying

All the best,
Amela