design with changing levels for the baseline alternative

[mfa]

Dear Ngene team,

I'm currently working on an experimental design where the definition of the baseline alternative (mlr in my design) changes each time across split samples (while being constant within each split sample). In each split sample, the baseline alternative takes different levels for two of the attributes: either 2 or 50 for DIST and either 0, 1 or 2 for WEALTH (all combinations possible such that there are 6 possible baseline alternatives, one for each of the 6 possible split samples). The levels of the other attributes in the baseline alternative are always the same across all split samples (level 0 for BIOD, 0 for RECR and 0 for Tax).

I would like to prepare a unique orthogonal design (for my pilots) that works for all split samples, while avoiding dominant alternatives. Is there a way to achieve that in Ngene? I suspect that it involves the specification of the baseline alternative (mlr) - which I haven't defined beyond a constant for now - and perhaps model averaging (though not available for orthogonal designs) or pivoting?

Any guidance regarding how to proceed would be greatly appreciated!

Draft experimental design reported below:

Design
;alts=alt1,alt2,mlr
;rows=72
;block=9,minsum
;orth=sim
;foldover
;con
;model:

U(alt1)= b1.dummy[1|0.7] * BIOD[2,1,0]
+ b2 * DIST[1,2,5,10,25,50,75,100]
+ b3.dummy[1] * RECR[1,0]
+ b4.dummy[-1|-0.8] * WEALTH[2,1,0]
+ b5 * TAX[4,8,16,32,48,64,80,96]/

U(alt2)= b1.dummy[1|0.7] * BIOD[2,1,0]
+ b2 * DIST[1,2,5,10,25,50,75,100]
+ b3.dummy[1] * RECR[1,0]
+ b4.dummy[-1|-0.8] * WEALTH[2,1,0]
+ b5 * TAX[4,8,16,32,48,64,80,96]/

U(mlr)=asc[0]$

Many thanks in advance for any help you can provide with this.

Best wishes,

M.

[Michiel Bliemer]

Dominant alternatives cannot be avoided with orthogonal designs. Orthogonal designs require a very strict structure and therefore may contain dominant alternatives. You will need to manually check the choice tasks in that case.

Instead of adding a blocking column, you could add the levels in mlr and then group these such that each block has the same levels. Using the syntax below, you would get 6 groups of 12 choice tasks, where group 1 consists of DIST = 2 and WEALTH = 0, ..., group 6 consists of DIST = 50 and WEALTH = 2.

Code: Select all

Design
;alts=alt1,alt2,mlr
;rows=72
;orth=sim
;con
;model:

U(alt1)= b1.dummy[1|0.7]   * BIOD[2,1,0]
       + b2                * DIST[1,2,5,10,25,50,75,100]
       + b3.dummy[1]       * RECR[1,0]
       + b4.dummy[-1|-0.8] * WEALTH[2,1,0]
       + b5                * TAX[4,8,16,32,48,64,80,96]
       /
U(alt2)= b1                * BIOD
       + b2                * DIST
       + b3                * RECR
       + b4                * WEALTH
       + b5                * TAX
       /
U(mlr) = asc[0]
       + b2                * DISTmlr[2,50]
       + b4                * WEALTH
$

You have more flexibility if you opt for an efficient design, which can automatically avoid dominant alternatives in choice tasks. Note that you will need to provide a prior for DIST and TAX as otherwise Ngene does not know how to treat these attributes with respect to dominance. Assuming that DIST and TAX have a negative prior, the following syntax would avoid dominant alternatives. It may take some time for the modified Federov algorithm to find a design without dominant alternatives and also satisfying some attribute level balance constraints.

Code: Select all

Design
;alts = alt1*, alt2*, mlr*
;rows = 72
;eff = (mnl,d)
;alg = mfederov(candidates = 2000)
;con
;model:

U(alt1)= b1.dummy[1|0.7]   * BIOD[2,1,0]
       + b2[-0.01]         * DIST[1,2,5,10,25,50,75,100](6-13,6-13,6-13,6-13,6-13,6-13,6-13,6-13)
       + b3.dummy[1]       * RECR[1,0]
       + b4.dummy[-1|-0.8] * WEALTH[2,1,0]
       + b5[-0.01]         * TAX[4,8,16,32,48,64,80,96](6-13,6-13,6-13,6-13,6-13,6-13,6-13,6-13)
       /
U(alt2)= b1                * BIOD
       + b2                * DIST
       + b3                * RECR
       + b4                * WEALTH
       + b5                * TAX
       /
U(mlr) = asc[0]
       + b2                * DISTmlr[2,50]
       + b4                * WEALTH
$

Michiel

[mfa]

Dear Michiel,

many thanks for your help in response to my message and for taking the time to make some suggestions to improve the design - that's much appreciated.

I've run your last suggested design (which I agree is better) and I had some questions:

1) while I like the idea of specifying the different possible baseline levels for MLR, I'm also a bit concerned about using different choice card combinations for different split samples. The idea, as usual with split samples, will be to compare responses across the splits. However, unless I keep the experimental design fixed across split samples, I wouldn't know if differences are driven by treatment effects (what I'm interested in) or rather the type of choice cards displayed (or a mix of both). I thought that using the same design/choice cards for all split samples might be best to avoid this problem. Would this be possible within your suggested design and, if not how would you suggest to proceed?

2) When running your code below I get an uneven number of scenarios per split sample group (from 10 to 17 depending on the split sample, rather than exactly 12 choice cards for each splt sample). Is that correct? How could I make sure that every group gets the same number of choice sets, which I think would be best, as we know that different number of choice cards might be yet another driver of differences across split samples? Using blocking could avoid this issue, isn't it? Why were you suggesting not using blocking? Also, would we not get more information by having different blocks of cards in each split sample?

3) Since my last message, I've been playing with the code and found that I needed to add some conditions to avoid the MLR option dominating alt1 or alts:

;cond:
if(alt1.DIST>=2,alt1.BIOD>0 or alt1.RECR=1),
if(alt2.DIST>=2,alt2.BIOD>0 or alt2.RECR=1)

To avoid strict dominance of alt1 over alt2 or viceversa, I've added the * command, as you suggested.

Unfortunately, it seems that the ;cond command is however incompatible with the mfederov algorithm. How would you suggest to proceed to add these conditions in the design?

4) I understand that (6-13,6-13,6-13,6-13,6-13,6-13,6-13,6-13) after DIST and TAX aims to define a range of times each attribute level appears (to try to achieve attribute level balance). Why is this only sought for DIST and TAX? Under what circumstances do I use this command?

5) After my last message, I've tried to improve my draft design (also added interaction effects) and ended up with the following, which seems to work if I manually replace the levels of the MLR option depending on the split sample:

Design
;alts=alt1*,alt2*,mlr*
;rows=24
;block=3,minsum
;eff=(mnl,d)
;con

;cond:
if(alt1.DIST>=2,alt1.BIOD>0 or alt1.RECR=1),
if(alt2.DIST>=2,alt2.BIOD>0 or alt2.RECR=1)

;model:

U(alt1)= b1.dummy[1|0.7] * BIOD[2,1,0]
+ b2[-0.5] * DIST[1,2,5,10,25,50,75,100]
+ b3.dummy[1] * RECR[1,0]
+ b4.dummy[-1|-0.8] * WEALTH[2,1,0]
+ b5[-0.01] * TAX[2,4,8,16,32,48,64,96]
+ b6[0.2] * BIOD.dummy[1]*RECR.dummy[1]
+ b7[0.25] * BIOD.dummy[2]*RECR.dummy[1]
+ b8[-0.1] * RECR.dummy[1]*DIST
+ b9[-0.1] * RECR.dummy[1]*WEALTH.dummy[2]
+ b10[-0.05] * RECR.dummy[1]*WEALTH.dummy[1]/

U(alt2)= b1.dummy * BIOD[2,1,0]
+ b2 * DIST[1,2,5,10,25,50,75,100]
+ b3.dummy * RECR[1,0]
+ b4.dummy * WEALTH[2,1,0]
+ b5 * TAX[2,4,8,16,32,48,64,96]
+ b6 * BIOD.dummy[1]*RECR.dummy[1]
+ b7 * BIOD.dummy[2]*RECR.dummy[1]
+ b8 * RECR.dummy[1]*DIST
+ b9 * RECR.dummy[1]*WEALTH.dummy[2]
+ b10 * RECR.dummy[1]*WEALTH.dummy[1]/

U(mlr)=asc[0]$

Is there a way of combining the features of both your design and mine, by keeping the best of both and while avoiding the problems mentioned above (esp. points 1-3)?

Many thanks in advance for your help with that and looking forward to see what you think.

Best wishes,

M.

[Michiel Bliemer]

1) Yes, that is correct, you will lose some statistical power in checking for differences across the samples. So if your study is mainly to study the differences across samples, then I would indeed use a homogeneous design. This can be achieved by the following syntax, which creates a single design for each of the six samples, identified by D2_W0 ... D50_W2. It will no longer be possible to check for dominance with respect to mlr if you are using a single design, so only dominance between alt1 and alt2 will be checked. This syntax uses the property ;fisher, which is not compatible with the modified Federov algorithm, so I reverted back to the default swapping algorithm. The swapping algorithm is not able to find a design with 72 rows where no dominant alternatives exist, but it could find a design with 24 rows. In your case, 24 rows should be more than sufficient for estimating this model. The design will be attribute level balanced.

Code: Select all

Design
;alts(D2_W0) = alt1*, alt2*, mlr
;alts(D2_W1) = alt1*, alt2*, mlr
;alts(D2_W2) = alt1*, alt2*, mlr
;alts(D50_W0) = alt1*, alt2*, mlr
;alts(D50_W1) = alt1*, alt2*, mlr
;alts(D50_W2) = alt1*, alt2*, mlr
;rows = 24
;eff = fish(mnl,d)
;fisher(fish) = design1(D2_W0[0.167],D2_W1[0.167],D2_W2[0.166],D50_W0[0.167],D50_W1[0.167],D50_W2[0.166])
;con
;model(D2_W0):
U(alt1)= b1.dummy[1|0.7]   * BIOD[2,1,0]
       + b2[-0.01]         * DIST[1,2,5,10,25,50,75,100]
       + b3.dummy[1]       * RECR[1,0]
       + b4.dummy[-1|-0.8] * WEALTH[2,1,0]
       + b5[-0.01]         * TAX[4,8,16,32,48,64,80,96]
       /
U(alt2)= b1                * BIOD
       + b2                * DIST
       + b3                * RECR
       + b4                * WEALTH
       + b5                * TAX
       /
U(mlr) = b0[-0.02]         ? -0.01*2 +0
;model(D2_W1):
U(alt1)= b1.dummy[1|0.7]   * BIOD[2,1,0]
       + b2[-0.01]         * DIST[1,2,5,10,25,50,75,100]
       + b3.dummy[1]       * RECR[1,0]
       + b4.dummy[-1|-0.8] * WEALTH[2,1,0]
       + b5[-0.01]         * TAX[4,8,16,32,48,64,80,96]
       /
U(alt2)= b1                * BIOD
       + b2                * DIST
       + b3                * RECR
       + b4                * WEALTH
       + b5                * TAX
       /
U(mlr) = b0[-0.82]         ? -0.01*2 -0.8
;model(D2_W2):
U(alt1)= b1.dummy[1|0.7]   * BIOD[2,1,0]
       + b2[-0.01]         * DIST[1,2,5,10,25,50,75,100]
       + b3.dummy[1]       * RECR[1,0]
       + b4.dummy[-1|-0.8] * WEALTH[2,1,0]
       + b5[-0.01]         * TAX[4,8,16,32,48,64,80,96]
       /
U(alt2)= b1                * BIOD
       + b2                * DIST
       + b3                * RECR
       + b4                * WEALTH
       + b5                * TAX
       /
U(mlr) = b0[-1.02]         ? -0.01*2 -1
;model(D50_W0):
U(alt1)= b1.dummy[1|0.7]   * BIOD[2,1,0]
       + b2[-0.01]         * DIST[1,2,5,10,25,50,75,100]
       + b3.dummy[1]       * RECR[1,0]
       + b4.dummy[-1|-0.8] * WEALTH[2,1,0]
       + b5[-0.01]         * TAX[4,8,16,32,48,64,80,96]
       /
U(alt2)= b1                * BIOD
       + b2                * DIST
       + b3                * RECR
       + b4                * WEALTH
       + b5                * TAX
       /
U(mlr) = b0[-0.50]         ? -0.01*50 +0
;model(D50_W1):
U(alt1)= b1.dummy[1|0.7]   * BIOD[2,1,0]
       + b2[-0.01]         * DIST[1,2,5,10,25,50,75,100]
       + b3.dummy[1]       * RECR[1,0]
       + b4.dummy[-1|-0.8] * WEALTH[2,1,0]
       + b5[-0.01]         * TAX[4,8,16,32,48,64,80,96]
       /
U(alt2)= b1                * BIOD
       + b2                * DIST
       + b3                * RECR
       + b4                * WEALTH
       + b5                * TAX
       /
U(mlr) = b0[-1.30]         ? -0.01*50 -0.8
;model(D50_W2):
U(alt1)= b1.dummy[1|0.7]   * BIOD[2,1,0]
       + b2[-0.01]         * DIST[1,2,5,10,25,50,75,100]
       + b3.dummy[1]       * RECR[1,0]
       + b4.dummy[-1|-0.8] * WEALTH[2,1,0]
       + b5[-0.01]         * TAX[4,8,16,32,48,64,80,96]
       /
U(alt2)= b1                * BIOD
       + b2                * DIST
       + b3                * RECR
       + b4                * WEALTH
       + b5                * TAX
       /
U(mlr) = b0[-1.50]         ? -0.01*50 -1
$

2. Yes that is indeed an issue, so perhaps the syntax above will be better.

3. Constraints are not compatible with the ;fisher command, so if you need these constraints I would suggest the following syntax where a single design is optimised using an average prior for b0 across the six samples.

Code: Select all

Design
;alts = alt1*, alt2*, mlr*
;cond:
if(alt1.DIST>=2,alt1.BIOD>0 or alt1.RECR=1),
if(alt2.DIST>=2,alt2.BIOD>0 or alt2.RECR=1)
;rows = 24
;eff = (mnl,d)
;con
;model:
U(alt1)= b1.dummy[1|0.7]   * BIOD[2,1,0]
       + b2[-0.01]         * DIST[1,2,5,10,25,50,75,100]
       + b3.dummy[1]       * RECR[1,0]
       + b4.dummy[-1|-0.8] * WEALTH[2,1,0]
       + b5[-0.01]         * TAX[4,8,16,32,48,64,80,96]
       /
U(alt2)= b1                * BIOD
       + b2                * DIST
       + b3                * RECR
       + b4                * WEALTH
       + b5                * TAX
       /
U(mlr) = asc[0]
       + b2                * DISTmlr[2,50]
       + b4                * WEALTH
$

4. Attribute level balance constraints typically need to be added to numerical attributes that are not dummy or effects coded as otherwise Ngene will only select the extreme levels since this is most efficient. For dummy or effects coded attributes this is not necessary since it is most efficient to use all attribute levels more or less equally.

5. Here I see that you also noticed that you need to use 24 rows (sorry I am looking at your questions sequentially because there are so many). My last syntax seems identical to your syntax, except that I my syntax allows for checking dominance with respect to mlr whereas yours does not (since you did not specify attributes in mlr). You can add the interaction effects to the utility functions in my sytax.

Michiel

[mfa]

Dear Michiel,

many thanks for looking into my code (and questions) so thoroughly. Much appreciated.

I've carefully looked into and run your last suggested designs - both options look very good, but each has some limitations too (that I would like to ideally overcome, if possible).

The first design you suggest in your last message has the advantage of preparing a homogenous design across all split samples, while taking into account the different baselines for MLR. This is great. However, unfortunately, the fact that the ;cond property cannot be used is a limiting factor, as without it (in some scenarios) alt1 or alt2 are dominated by the MLR option.

The last design you suggest in your last message is very similar to my last suggested design. It allows to add the ;cond property (which, similar to the specification of the attributes in mlr that you suggested, avoids that alt1 or alt2 are dominated by MLR). However, your design doesn't produce a unique experimental design (the same combinations) for all split samples and it generates an uneven number of choice cards depending on the split sample group, which is not ideal for comparison purposes.

Ideally the design I'm looking for should allow to: 1) use the same choice card combinations across split samples (except for definition of baseline which changes across split samples), 2) avoid MLR dominance and 3) ensure that same number of choice cards across split samples (ideally <10) are displayed. My last suggested design (message from 13th September), which is very close to your last suggested design except for MLR specification and use of blocks, seems to me to allow to somehow achieve these (1-3) desired properties (MLR dominance in my case is avoided by using ;cond). However, the specification of the MLR seems to be the sticky point - if I specify it my way (U(mlr)=asc[0]) the design is not considering that different baseline levels are used in the different split samples (and this might be a problem if I want to later estimate different models for each split sample?). If we specify MLR in the way you suggested I end up with different combinations (and number of choice cards) depending on the split sample. Is it absolutely important to specify the different levels for DIST and WEALTH in the MLR alternative? If so, then is there a way to do this while ensuring that the above desired properties (1-3) are also met? I couldn't find a perfect solution, unfortunately, the only idea that came to mind could be to perhaps replace U(mlr)=asc(0) with another constant estimated by 'pooling' the responses (across the different split samples) from a pilot I'm about to launch.

How would you suggest to proceed given the above?

Thank you very much in advance for your help with this.

Best wishes,

M.

[Michiel Bliemer]

It has been a while ago since I looked at this and it would take be a considerable amount of time to re-read all the text and again understand the study, but some quick responses:

* The DIST and WEALTH attributes in mlr are needed to avoid dominant alternatives with mlr. You can replace the whole utility with a constant, but that means that Ngene will no longer be able to check for dominance with mlr.

* You can always generate more choice tasks and then simply manually remove the ones that you do not like. Given that you have a large number of constraints that you want the design to satisfy, you may want to impose some of them manually in case Ngene cannot satisfy all.

* To have the same choice tasks for each respondent, you could consider reducing the number of choice tasks to 8 or 10. This is quite minimal but should still be sufficient to estimate all your parameters.

Michiel

[mfa]

Thanks very much for these suggestions Michiel - yes, fully understand and sorry for not getting back earlier on this.

Best wishes,

M.