Minimize choice card received per person for online survey

[mushi88]

Dear Michiel and the ChoiceMetrics Team,
Hello! I have a question regarding the maximum limits of rows/blocks. I am designing an online survey to gather information about people's preferences on different environmental policies. We are expecting a large sample (10k+) but want to minimize the number of choice cards given to each participant.

I have two questions:

1. We’re wondering if you have recommendations or references for a structured way to minimize the number of choice cards received per person (i.e., maximize the number of rows and blocks), or is this just a choice made by the designer? For example, I have designed the survey with 75 rows and 25 blocks), so each person receives only 3 choice cards to minimize fatigue. Are there any issues with this design and general approach? Our code is copied below.

2.Are there constraints regarding the number of levels for a specific attribute. Since our sample size is likely quite large, would it be a problem if we had, say, 11 levels for one attribute, either in the design or analysis stage?

We plan is to obtain priors through a D-efficient design that optimizes MNL and then use these priors to design a Bayesian MNL model. We will estimate the final results using RPL. The attached code represents our preliminary design before obtaining priors.

Thank you in advance!

Code: Select all


Design
;alts=option1*, option2*, opt-out
;rows=75
;eff=(mnl,d)
;alg=swap(stop=total(5000000iterations))
;block=25

;model:
U(option1)=
b2.dummy [0.00001|0.00002]*A[1,2,0]+
B3.dummy [0.00001|0.00002|0.00003|0.00004|0.00005]*B[1,2,3,4,5,0]+
B4.dummy [0.00001|0.00002|0.00003]*C[1,2,3,0]+
b5.dummy [0.00001|0.00002|0.00003|0.00004|0.00005|0.00006] *D[1,2,3,4,5,6,0] + b6[-0.00001]*price[5,25,50,75]/

U(option2)=b2*A+b3*B    +b4*C     +b5*D +b6*price/
U(opt-out)=asc
$


[Michiel Bliemer]

The number of choice sets given to a decision maker is your choice. If you use too few then there may be a learning effect, while if you use too many then there may be a fatigue effect. I don't think that there is much risk of fatigue with 8 or less choice tasks since each choice task in your case is relatively simple.

There are no constraints on the number of levels, 11 levels is fine but if categorical it will require estimating a lot of parameters of dummy variables.

Michiel

[mushi88]

Got it. Thank you so much Michiel!

[mushi88]

Hi Michiel,

following up on the previous question, we are interested in examining the interaction of C and D so I created 18 more interaction terms. I am wondering if you have suggestions on:

1) whether such interaction is encouraged at the design stage? or we should focus the design on variables A-D and use interactions in final model analysis?

2) For the next stage design (after I acquired the priors), would you recommend to keep the interaction priors as [0], or used the coefficient from the conditional login model as a fixed prior?

Thank you!

Code: Select all

 Design
;alts=option1*, option2*, opt-out
;rows=75
;eff=(mnl,d)
;alg=swap(stop=total(500000iterations))
;block=25

;model:
U(option1) =
     b2.dummy [0.00001|0.00002]                                      * A[1,2,0]
   + b3.dummy [0.00001|0.00002|0.00003|0.00004|0.00005]              * B[1,2,3,4,5,0]
   + b4.dummy [0.00001|0.00002|0.00003]                              * C[1,2,3,0]       
   + b5.dummy [-0.00006|-0.00005|0.00004|-0.00003|-0.00002|-0.00001] * D[1,2,3,4,5,6,0]
   + b6[-0.00001]                                                    * E[5,25,50,75] 
   + i1[0] * C.dummy[1] * E.dummy[1] 
   + i2[0] * C.dummy[1] * E.dummy[2] 
   + i3[0] * C.dummy[1] * E.dummy[3] 
   + i4[0] * C.dummy[1] * E.dummy[4] 
   + i5[0] * C.dummy[1] * E.dummy[5] 
   + i6[0] * C.dummy[1] * E.dummy[6] 
   + i7[0] * C.dummy[2] * E.dummy[1]
   + i8[0] * C.dummy[2] * E.dummy[2] 
   + i9[0] * C.dummy[2] * E.dummy[3] 
   + i10[0] * C.dummy[2] * E.dummy[4]
   + i11[0] * C.dummy[2] * E.dummy[5]
   + i12[0] * C.dummy[2] * E.dummy[6] 
   + i13[0] * C.dummy[3] * E.dummy[1] 
   + i14[0] * C.dummy[3] * E.dummy[2] 
   + i15[0] * C.dummy[3] * E.dummy[3] 
   + i16[0] * C.dummy[3] * E.dummy[4] 
   + i17[0] * C.dummy[3] * E.dummy[5]
   + i18[0] * C.dummy[3] * E.dummy[6] / 

U(option2) = 
     b2 * A
   + b3 * B 
   + b4 * C 
   + b5 * D
   + b6 * E
   + i1 * C.dummy[1] * E.dummy[1] 
   + i2 * C.dummy[1] * E.dummy[2] 
   + i3 * C.dummy[1] * E.dummy[3]
   + i4 * C.dummy[1] * E.dummy[4]
   + i5 * C.dummy[1] * E.dummy[5] 
   + i6 * C.dummy[1] * E.dummy[6] 
   + i7 * C.dummy[2] * E.dummy[1] 
   + i8 * C.dummy[2] * E.dummy[2] 
   + i9 * C.dummy[2] * E.dummy[3]
   + i10 * C.dummy[2] * E. dummy[4] 
   + i11 * C.dummy[2] * E.dummy[5] 
   + i12 * C.dummy[2] * E.dummy[6] 
   + i13 * C.dummy[3] * E.dummy[1] 
   + i14 * C.dummy[3] * E.dummy[2] 
   + i15 * C.dummy[3] * E.dummy[3] 
   + i16 * C.dummy[3] * E.dummy[4] 
   + i17 * C.dummy[3] * E.dummy[5] 
   + i18 * C.dummy[3] * E.dummy[6] /

U(opt-out)=b1[0] 

$


[Michiel Bliemer]

1. I generally only include interactions if I believe that they are important to include in the final model estimation. In most cases I simply try some interaction effects during model estimation and see what comes out as statistically significant, but I do not optimise for interactions in the design. Including them at the design stage ensures that you will be able to estimate all of them (although with 75 rows it is likely that you can even without considering them at the design stage), but it does take some of the Fisher information away from the main effects.

2. Interaction effects of dummy coded variables often are not statistically significant in a pilot study and as such as often quite unreliable. If they are too unreliable then you could leave them as 0, but if some of them turn out to be sufficiently reliable you can include those as non-zero priors.

Michiel