choice-metrics.com

Dear Community,

we are planning to conduct a discrete choice experiment where we want to manipulate the illustration of individual attributes. As a result, we need to create separate d-efficient designs for the different experimental groups.
The discrete choice experimet is about identifying causal effects. To identify a possible causal effect of different illustrations, of course, only the illustration should differ between the groups. However, since we additionally use different d-efficient designs for the experimental groups I am not sure whether a possible causal effect is really due to the different illustration, to the systmatically different choice sets (due to the different d-efficient designs), or to a combination of both.
In short, my question is whether I can attribute a possible causal effect to the different illustrations for the experimental groups despite the different d-efficient designs for the experimental groups?

Thanks in advance for your help.
Haverkamp

When the objective is to compare choice behaviour across experiments and/or across groups, it is beneficial to keep as much as possible the same across the experiments and groups to allow for more statistical power when testing differences across these choice behaviours. Indeed as you say, if the design is different and the sample is different, then it is more difficult to make such a comparison and a much higher sample size would be needed to have sufficient statistical power.

In your case I would use a within-subject experiment with a homogeneous design.

WITHIN-SUBJECT EXPERIMENT

In a within-subject experiment, you show the same individual multiple treatments. For example, you show 4 choice tasks with one set of illustrations and then later in the survey you show the same choice tasks where only the illustrates are different. This may not be possible in all circumstances because of memory-effects, so if you think that respondents will remember the choice tasks then you will need to show different choice tasks (and lose some statistical power when making comparisons). Your experimental design can of course be much larger than these 4 choice tasks, you could for example use a design of 24 choice tasks where the first respondent is given the first four, the next respondent is given the next four, etc.

HOMOGENEOUS DESIGN

You can use a homogeneous design for all sets of illustrations. In this case, you would use one of the following designs:

* Orthogonal design using design coding (0,1,2,...), where you replace the levels for illustrations with new illustrations and keep all other levels the same. Such a design cannot avoid dominant choice tasks and no constraints can be imposed.

* D-efficient design with (near) zero priors using design coding (0,1,2,...). When using zero priors, the actual attribute levels no longer matter and you can simply use design coding. Then you can do the same as above, you replace the levels for illustrations with new illustrations and keep all other levels the same. You will need to apply dummy coding for categorical attributes. When using near-zero priors, e.g. 0.00001, 0.00002, or -0.00001 to indicate the preference order of the attribute levels, you can also automatically avoid dominant alternatives in Ngene. Further, in a D-efficient design you can impose further constraints if desired.

While an orthogonal design and a D-efficient design with (near) zero priors is not as efficient as a D-efficient design with informative nonzero priors, in your case I would be happy to sacrifice some statistical efficiency to allow for a more powerful comparison across choice observations.

Michiel