Priors, optimal design and associated syntax

[KatherineL]

Hi all

I am very new to the design of DCE and so please bear with me.

I am looking to design an online DCE for approx. 500 individuals. I will have 5 attributes, each with 3 attribute levels and I would like to present 3 alternatives per choice task. I did a small pilot study in about 100 subjects where they were presented with a similar DCE. Although I have decided to change one of the attributes as I didn’t find one of them very informative. All attribute levels have an implied order (e.g. low, adequate or optimal nutrition content (nut: 0, 1, 2) and cost (cost: $5, 10, 15)).

I thought as a minimum I could include information on whether the priors werepositive or negative (fifth prior was based on literature). I investigated the foldover option and also the blocking option but was unable to get blocks of 12 tasks and the first choice task always included identical meal alternatives. The syntax I used was:

design
;alts = mealA, mealB, mealC
;rows = 36
;orth = sim
;block = 3
;model:
U(mealA) = b1 + btaste*taste[0,1,2] + btime*time[5,15,30] + bnut*nut[0,1,2] + bcost*cost[5,10,15] + bqual*qual[0,1,2] /
U(mealB) = btaste*taste + btime*time + bnut*nut + bcost*cost + bqual*qual /
U(mealC) = btaste*taste + btime*time + bnut*nut + bcost*cost + bqual*qual $

So I instead used the design detailed below.

design
;alts = mealA, mealB, mealC
;rows = 12
;eff = (mnl,d)
;model:
U(mealA) = b1[0] + btaste[0.001]*taste[0,1,2] + btime[-0.001]*time[5,15,30] + bnut[0.001]*nut[0,1,2] + bcost[-0.001]*cost[5,10,15] + bqual[0.001]*qual[0,1,2] /
U(mealB) = btaste*taste + btime*time + bnut*nut + bcost*cost + bqual*qual /
U(mealC) = btaste*taste + btime*time + bnut*nut + bcost*cost + bqual*qual $

The MNL efficiencies for this second design were as follows
D error 0.024422
A error 0.078445
B estimate 99.98175
S estimate 496916.634033

So my questions are:

1) Can I still use priors in this new study even though I only have prior data for 4 of the 5 attributes? These attributes were dummy coded so I have coefficients and SE for each attribute level. If I can, how could I incorporate this into the syntax?
2) If I can’t use the priors then I thought that indicating the sign would be better than nothing. I thus made the values very close to zero (e.g. taste[0.001]), is this correct?
3) Should I use dummy coding or effects coding instead of treating attribute levels as “continuous”? Would I then use syntax like this for example: btaste.effects[0.001|0.002]*taste[0,1,2]?
4) Why was the first design not creating 12 choice tasks per block? Any why did the first choice task include three identical meal alternatives?
5) Is the second design acceptable? How can I improve it? I appreciate that the S estimate must be very large given that my priors are so close to zero.
6) I want to include a two-step opt out option, where I present a forced choice task followed by a question about whether they would actually eat that meal (Y/N). It this acceptable or should this be incorporated into the experimental design and then followed by a force choice question?

Thank you in advance and I look forward to hearing from you.

Best wishes
Katherine

[Michiel Bliemer]

1) You can provide priors for the attributes that appear in your pilot study. Please use identical utility specifications for model estimation and design in order to be able to transfer priors. You will need to use dummy or effects coding for nut, taste, etc. Please note that the LAST level in Ngene is the reference level, so you may want to set bnut.dummy[0.01|0.02]*nut[1,2,0], where 0 = low nutrition, 1 = adequate nutrition, and 2 = optimal nutrition. Further, instead of using 0.01 and 0.02 you want to use Bayesian priors directly obtained from your pilot study, e.g. bnut.dummy[(n,0.02,0.01)|(n,0.04,0.03)]*nut[1,2,0] where 0.03 and 0.04 are the estimates for the dummy coded parameters, and 0.01 and 0.03 are their standard errors.

2) Yes choosing them close to zero is a good choice.

3) Yes see 1)

4) There does not exist a simultaneous orthogonal design with 36 rows for your dimensions, only 54 rows. Note that you should use a sequential orthogonal design since your alternatives are unlabelled, so you can change ;orth = sim to ;orth = seq and Ngene will be able to find a design with 36 rows. The fact that three the same alternatives result in the first choice task is an artefact of a simultaneous orthogonal design. Instead of using an orthogonal design, I think you are better off using an efficient design. Since you provide attribute level orderings by providing priors, you can let Ngene also remove dominant alternatives by setting ;alts = mealA*,mealB*,mealC* (i.e. adding an asterisk after each alternative name).

5) Use a Bayesian efficient design with priors from your pilot study. Set ;eff = (mn,d,mean) and set ;bdraws = gauss(2) or gauss(3). Yes your S-estimates are large because of your small (and likely unrealistic) priors.

6) You can specify two models in Ngene at the same time, one with forced choice (3 alts) and one with unforced choice (additional optout). You can do this by defining ;model(forced) .... ;model(unforced)... with ;alts(forced) = mealA,mealB,mealC, and ;alts(unforced) = mealA,mealB,mealC,optout... etc, and then optimise the design with something like ;eff = 2*forced(mnl,d) + unforced(mnl,2). This is pretty advanced and I would say not strictly necessary. If you optimised for the forced choice only I think you will already get a quite good design.

Michiel

[KatherineL]

Hi Michiel

Thank you very much for your informative reply. I've tried to take on board your feedback in the design below, although I have a few follow up queries:

1. What did you mean by " Please use identical utility specifications for model estimation and design in order to be able to transfer priors. " Does this mean that I can only use the priors if I keep the attributes the same between experiments? Which would mean that I would not be able to use them as the fifth attribute was not the same. See example below where I used the priors from the four attributes from the previous pilot study and added direction (positive) only priors to the attribute qual (quality).

2. I will be using Qualtrics to deliver the survey and DCE. My understanding is that the best way to do that is to manually create the choice tasks based on the experimental design using radio buttons. If I use the blocked orthogonal design I would only need to manually create 36 choice tasks, block these 36 into 3 blocks to get 12 choice tasks per block, and then assign my 400 participants to receive one of the three blocks. Could I add blocks to the Bayesian design to do this? When I try to do this below it comes up with an error message.

3. In my pilot study I presented 4 meal alternatives(A-D). I did this because I had read that it is preferable to have more alternatives than number of attribute levels (i.e. 3 levels so should have 4 meal alternatives). Do you think I should consider adding a fourth meal alternative or does the experimental design not require this? If I do use 3 meal alternatives, does that mean that I am unable to use the priors from the study with 4 meal alternatives or does is not matter?

Looking forward to your feedback and thank you in advance.

Best wishes
Katherine


? Bayesian efficient design that includes blocking - this gives an error message once I try to add blocking. Have I got to tweak this somehow to create blocks of 12 choice tasks?
design
;alts = mealA*, mealB*, mealC*
;rows = 36
;block = 3
;eff = (mnl,d,mean)
;bdraws = gauss(2)
;model:
U(mealA) = b1[0.09] + btaste.dummy[(n,0.38,0.08)|(n,0.20,0.08)]*taste[2,1,0] + btime.dummy[(n,-0.06,0.08)|(n,-0.33,0.08)]*time[30,15,5]
+ bnut.dummy[(n,1.11,0.09)|(n,0.63,0.09)]*nut[2,1,0] + bcost.dummy[(n,-0.50,0.08)|(n,-0.16,0.08)]*cost[15,10,5] + bqual.dummy[(n,0.02,0.01)|(n,0.01,0.005)]*qual[2,1,0] /

U(mealB) = btaste*taste + btime*time + bnut*nut + bcost*cost + bqual*qual /
U(mealC) = btaste*taste + btime*time + bnut*nut + bcost*cost + bqual*qual $


? Bayesian design - this creates a design but I don't think I can manually create choice tasks with this as I don't know how to assign my 400 participants to it
design
;alts = mealA*, mealB*, mealC*
;rows = 12
;eff = (mnl,d,mean)
;bdraws = gauss(2)
;model:
U(mealA) = b1[0.09] + btaste.dummy[(n,0.38,0.08)|(n,0.20,0.08)]*taste[2,1,0] + btime.dummy[(n,-0.06,0.08)|(n,-0.33,0.08)]*time[30,15,5]
+ bnut.dummy[(n,1.11,0.09)|(n,0.63,0.09)]*nut[2,1,0] + bcost.dummy[(n,-0.50,0.08)|(n,-0.16,0.08)]*cost[15,10,5] + bqual.dummy[(n,0.02,0.01)|(n,0.01,0.005)]*qual[2,1,0] /

U(mealB) = btaste*taste + btime*time + bnut*nut + bcost*cost + bqual*qual /
U(mealC) = btaste*taste + btime*time + bnut*nut + bcost*cost + bqual*qual $


? sequential design - this creates a blocked design that I can use to manually create 36 choice tasks but it may not be the most efficient?
design
;alts = mealA*, mealB*, mealC*
;rows = 36
;orth = seq
;block = 3
;model:
U(mealA) = b1[0.09] + btaste.dummy[(n,0.38,0.08)|(n,0.20,0.08)]*taste[2,1,0] + btime.dummy[(n,-0.06,0.08)|(n,-0.33,0.08)]*time[30,15,5]
+ bnut.dummy[(n,1.11,0.09)|(n,0.63,0.09)]*nut[2,1,0] + bcost.dummy[(n,-0.50,0.08)|(n,-0.16,0.08)]*cost[15,10,5] + bqual.dummy[(n,0.02,0.01)|(n,0.01,0.005)]*qual[2,1,0] /

U(mealB) = btaste*taste + btime*time + bnut*nut + bcost*cost + bqual*qual /
U(mealC) = btaste*taste + btime*time + bnut*nut + bcost*cost + bqual*qual $

[Michiel Bliemer]

1) You can always set priors to zero if the attribute did not appear in your pilot study. I mainly mean to say that if you want to use priors for a dummy coded variable, you need to estimate a dummy coded variable and not a continuous variable.

2) Yes you can do this. The reason why the syntax did not work is not because of blocking, but rather that you apply dominance constraints and this makes it more difficult for Ngene to find a design using the default swapping algorithm. If you switch to the modified federov algorithm then it will work. Simply add something like:

;alg = mfederov(candidates = 1000)

3) I have not heard of a rule that you need more alternatives than the number of attribute levels, that sounds strange to me. You can ignore that rule. I think that 3 alternatives is more than enough as otherwise your choice tasks become very complex. Priors are not depends on the number of alternatives since your alternatives are unlabelled. Btw I notice that you specify a constant for mealA, is there a reason there is a constant there? So that means that the alternatives are not unlabelled?

[KatherineL]

Hi Michiel

Thank you so much for your prompt reply.

Thank you for pointing out the constant - I had forgotten that unlabelled experiments shouldn't have constants so I have removed that.

The below design seems to work, which is great! Do you have any further suggestions or do you think this is good enough?

? Bayesian design that includes priors, blocking and algorythm to allow for (*) to avoid dominance
design
;alts = mealA*, mealB*, mealC*
;rows = 36
;block = 3
;eff = (mnl,d,mean)
;bdraws = gauss(2)
;alg = mfederov(candidates = 1000)
;model:
U(mealA) = btaste.dummy[(n,0.38,0.08)|(n,0.20,0.08)]*taste[2,1,0] + btime.dummy[(n,-0.06,0.08)|(n,-0.33,0.08)]*time[30,15,5]
+ bnut.dummy[(n,1.11,0.09)|(n,0.63,0.09)]*nut[2,1,0] + bcost.dummy[(n,-0.50,0.08)|(n,-0.16,0.08)]*cost[15,10,5]
+ bqual.dummy[(n,0.02,0.01)|(n,0.01,0.005)]*qual[2,1,0] /

U(mealB) = btaste*taste + btime*time + bnut*nut + bcost*cost + bqual*qual /
U(mealC) = btaste*taste + btime*time + bnut*nut + bcost*cost + bqual*qual $

Thanks again!

Best wishes
Katherine

[Michiel Bliemer]

I think it looks fine.

You could replace (n,0.01,0.005) with 0.01 and (n,0.02,0.01) with 0.02 since these distributions are so narrow that drawing from these distributions is not useful. This will make evaluating designs 4 times faster, i.e. Ngene is then doing 2^8 draws instead of 2^10 draws for each Bayesian design evaluation.

Michiel

[KatherineL]

Hi Michiel

Thanks for that.

I replaced that:

design
;alts = mealA*, mealB*, mealC*
;rows = 36
;block = 3
;eff = (mnl,d,mean)
;bdraws = gauss(2)
;alg = mfederov(candidates = 1000)
;model:
U(mealA) = btaste.dummy[(n,0.38,0.08)|(n,0.20,0.08)]*taste[2,1,0] + btime.dummy[(n,-0.06,0.08)|(n,-0.33,0.08)]*time[30,15,5]
+ bnut.dummy[(n,1.11,0.09)|(n,0.63,0.09)]*nut[2,1,0] + bcost.dummy[(n,-0.50,0.08)|(n,-0.16,0.08)]*cost[15,10,5]
+ bqual.dummy[0.02|0.01]*qual[2,1,0] /

U(mealB) = btaste*taste + btime*time + bnut*nut + bcost*cost + bqual*qual /
U(mealC) = btaste*taste + btime*time + bnut*nut + bcost*cost + bqual*qual $

Just one last query, I realise that I used the order [2,1,0] for levels but notice you indicated a different order in a previous reply. Should I change this to [1,2,0] and swap the priors. i.e:

design
;alts = mealA*, mealB*, mealC*
;rows = 36
;block = 3
;eff = (mnl,d,mean)
;bdraws = gauss(2)
;alg = mfederov(candidates = 1000)
;model:
U(mealA) = btaste.dummy[(n,0.20,0.08)|(n,0.38,0.08)]*taste[1,2,0] + btime.dummy[(n,-0.33,0.08)|(n,-0.06,0.08)]*time[15,30,5]
+ bnut.dummy[(n,0.63,0.09)|(n,1.11,0.09)]*nut[1,2,0] + bcost.dummy[(n,-0.16,0.08)|(n,-0.50,0.08)]*cost[10,15,5]
+ bqual.dummy[0.01|0.02]*qual[1,2,0] /

U(mealB) = btaste*taste + btime*time + bnut*nut + bcost*cost + bqual*qual /
U(mealC) = btaste*taste + btime*time + bnut*nut + bcost*cost + bqual*qual $

This creates the following table. For the taste attribute, for example, the levels are sufficient (0), good (1) and very good (2) and the priors are ref, 0.02 (SE 0.08), 0.38 (SE 0.08). Apologies if this is a silly question but how do these three levels and priors correspond to the d0 and d1 for taste below?

Prior btaste(d0) btaste(d1) btime(d0) btime(d1) bnut(d0) bnut(d1) bcost(d0) bcost(d1) bqual(d0) bqual(d1)
Fixed prior value 0.2 0.38 -0.33 -0.06 0.63 1.11 -0.16 -0.5 0.02 0.01
Sp estimates 24.334172 9.798166 11.72332 331.149298 2.54471 1.4041 42.082737 4.881516 4508.059401 11182.197358
Sp t-ratios 0.397327 0.626158 0.572441 0.107707 1.228675 1.654082 0.302137 0.887113 0.029192 0.018535
Sb mean estimates 40.412213 11.224257 14.03766 1520.342702 2.716314 1.435941 93.912623 5.282329 4530.308583 11240.527183
Sb mean t-ratios 0.39682 0.624128 0.570538 0.143003 1.225495 1.648867 0.301417 0.884131 0.029121 0.018488

Best wishes
Katherine

[Michiel Bliemer]

You do not need to swap them, as long as the priors and levels are in the order (with the same base level) then it gives the same result.

If you have b.dummy[a|b]*x[1,2,3] then d0 refers to the parameter for level 1 (with prior a) and d1 refers to the parameter for level 2 (with prior b).

Michiel

[KatherineL]

Hi Michiel

Great! Thank you very much for your time and advice.

Best wishes
Katherine