2-in-1 design

[peyman_07]

Hi Michiel,

First of all, I would like to appreciate your massive support. My question is about a particular design with two stages. At the first stage, respondents are supposed to choose an alternative based on the first set of attributes. Then, more attributes (the second sets of attributes) are shown to them while the previous attributes are still available with the same attribute levels and they are asked to make their second choice. The purpose is being able to compare respondents' choices when more information is provided. In this case, I think there are two ways of design:

1- Creating two separate designs in a way that the first design is based on the first set of attributes and the second one is based on the second set of attributes. Then, for the first stage of the survey, we just show the choice sets of the first design and after their first decision, for the second stage, we add the choice sets of the second design to the first one shown to respondents.
2- Producing one single design with all attributes (two sets of attributes). Then, for the first part, we remove the second set of attributes from the choice set and then after the first choice we add them and ask respondents to make their second choice.

I was wondering which design is preferred while minimizing the correlation between two sets of attributes? If there is a more efficient design, I would be grateful if you let me know.
Thanks.

[Michiel Bliemer]

I believe that I answered a similar question from you earlier this year.

Both options you propose would likely work, but the designs may not be very efficient. I have not tried such a design before.

If I would do it, I would optimise for both designs simultaneously, for example using syntax like the one below, where model m1 contains the initial 2 attributes, and model m2 adds a third attribute while keeping the other ones the same.

Code: Select all

design
;alts(m1) = alt1, alt2
;alts(m2) = alt1, alt2
;rows = 12
;eff = m1(mnl,d) + m2(mnl,d)
;model(m2):
U(alt1) = b1 * x1[1,2,3] + b2 * x2[1,2,3] + b3 * x3[1,2,3] /
U(alt2) = b1 * x1[1,2,3] + b2 * x2[1,2,3] + b3 * x3[1,2,3]
;model(m1):
U(alt1) = b1 * x1[1,2,3] + b2 * x2[1,2,3] /
U(alt2) = b1 * x1[1,2,3] + b2 * x2[1,2,3]
$


Michiel

[peyman_07]

Thanks Michiel for your reply. Just ran the example design but the problem is that this model does not keep the same attribute levels for the common attributes (e.g., b1 and b2). We do not want to change the attribute levels of the first choice set given that the idea is to understand whether the choice of respondents changes when more information is shown. Is there any way of keeping the same attribute levels for the common attributes in these two models?

Thanks.

[Michiel Bliemer]

That is exactly what this syntax does. The generated design is for the second set of questions including all attributes. For the first set of questions, you simply remove the last attribute(s). The levels of both sets of questions for the first attributes remain the same. The design is optimised simultaneously for both sets, which I understood is what you want.

Michiel

[peyman_07]

Brilliant! I applied this method to my survey which is about the taxi drivers' ride acceptance behaviour. The idea is that respondents (taxi drivers) receive a ride request in their mobile application and they just need to decide whether to accept or decline it (a binary choice) based two sets of attributes specified in model 1 and model 2 in the following syntax:

Code: Select all

design

;alts(m1) = ACC, DEC
;alts(m2) = ACC, DEC

;rows=24
;block=6
;eff = m1(mnl,d,mean) + m2(mnl,d,mean)
;bdraws=halton(250)
;alg = mfederov

;model(m2):

U(ACC)= tr.dummy[(n,-0.548,0.14)]*TR[1,2]+rt.dummy[(n,0.558,0.478)|(n,0.866,0.515)]*RT[1,2,3]+wt[0.00001]*WT[0,5,10,15]+dr.dummy[0.000001]*DR[1,2]+rr[(n,0.638,0.311)]*RR[3,4,5]+pt[-0.001]*PT[5,10,15,20]+sp[0.0001]*SP[0,1.5,3]+lo.dummy[0.0001]*LO[1,2](7-10,14-17)+pl.dummy[-0.00001]*PL[1,2]+tcg[(u,-0.05,0)]*TCG[0,20,40]+gt[(n,0.4,0.129)]*GT[0,2,4]+tf[(u,0.05,0.2)]*TF[5,15,25]/

U(DEC)= ASC0

;model(m1):

U(ACC)= tr.dummy[(n,-0.487,0.15)]*TR[1,2]+rt.dummy[(u,0,0.5)|0]*RT[1,2,3]+wt[(n,0.0373,0.261)]*WT[0,5,10,15]+dr.dummy[0.000001]*DR[1,2]+rr[(u,0,0.4)]*RR[3,4,5]+pt[-0.001]*PT[5,10,15,20]+sp[(u,0,0.2)]*SP[0,1.5,3]+lo.dummy[(u,0.1,0.375)]*LO[1,2](7-10,14-17)+pl.dummy[-0.00001]*PL[1,2]/

U(DEC)= ASC0

$



I estimated this model and have some followup questions about the model specifications and the Ngene outputs. I would be so grateful if you answer them.

1) Have the utility functions been formulated correctly for designing and estimating this binary logit model?

2) The D-efficient value (Fixed) is 0.35 and 0.894 for model 1 and model 2, respectively. Are they logical, especially for m2?

3) The S-efficient value is 6105517061421.31. Does it mean that I need to have this number of observations - as a minimum - to have all parameters significant? If so, given that it is not feasible for me to collect this number of observations, what could I do?

4) Just checked the utility of each alternative in each choice set. In all cases, the utility of the second alternative is 0. I think this is because we only have context variables and there is only one component in the utility function of DEC which is ASC0. Is this the potential reason for this? If so, I was wondering if this may cause any problem later in the model estimation?

5) In the model average approach in Ngene, it is not possible to impose any constraint on attributes, right? If so, in case there are a few infeasible combinations, is it possible to change the attribute levels of an attribute manually?

Many thanks for your massive support,
Peyman.

[Michiel Bliemer]

1. Yes. Another way of writing them is in the syntax below, where I have moved the constant to ACC so you can omit specifying DEC (it defaults to zero). Note that you forgot to specify a prior for ASC0. This prior is very important, especially in model m2. If you set it to zero, then in m2 everyone will choose alternative ACC because gt and tf add very positive utilities and therefore choice probabilities become almost 1-0, leading to extremely inefficient designs. You need to set appropriate priors in order to generate efficient designs, otherwise you need to default ALL of them to zero. You can never leave ASCs as zero unless all other priors are also zero. I assume you estimated a model based on pilot data, so you should have your ASCs, which will be different for each of the two models.

Code: Select all

design

;alts(m1) = ACC, DEC
;alts(m2) = ACC, DEC

;rows=24
;block=6
;eff = m1(mnl,d,mean) + m2(mnl,d,mean)
;bdraws=gauss(3)
;alg = mfederov

;model(m2):

U(ACC)= ASC0
      + tr.dummy[(n,-0.548,0.14)]                 * TR[1,2]
      + rt.dummy[(n,0.558,0.478)|(n,0.866,0.515)] * RT[1,2,3]
      + wt[0.00001]                               * WT[0,5,10,15]
      + dr.dummy[0.000001]                        * DR[1,2]
      + rr[(n,0.638,0.311)]                       * RR[3,4,5]
      + pt[-0.001]                                * PT[5,10,15,20]
      + sp[0.0001]                                * SP[0,1.5,3]
      + lo.dummy[0.0001]                          * LO[1,2](7-10,14-17)
      + pl.dummy[-0.00001]                        * PL[1,2]
      + tcg[(u,-0.05,0)]                          * TCG[0,20,40]
      + gt[(n,0.4,0.129)]                         * GT[0,2,4]
      + tf[(u,0.05,0.2)]                          * TF[5,15,25]

;model(m1):

U(ACC)= ASC0
      + tr.dummy[(n,-0.487,0.15)]                 * TR[1,2]
      + rt.dummy[(u,0,0.5)|0]                     * RT[1,2,3]
      + wt[(n,0.0373,0.261)]                      * WT[0,5,10,15]
      + dr.dummy[0.000001]                        * DR[1,2]
      + rr[(u,0,0.4)]                             * RR[3,4,5]
      + pt[-0.001]                                * PT[5,10,15,20]
      + sp[(u,0,0.2)]                             * SP[0,1.5,3]
      + lo.dummy[(u,0.1,0.375)]                   * LO[1,2](7-10,14-17)
      + pl.dummy[-0.00001]                        * PL[1,2]

$

2. See above, if you specify the prior for ASC0 correctly the D-error for m2 will be more appropriate, now it is problematic.

3. Since not all of your priors are realistic, such as such as lo.dummy being set to 0.0001, your S-estimates cannot be computed and should be ignored. ONLY if you have reliable priors with a certain level of significance you can use S-estimates.

4. Yes, and you have assumed ASC0 = 0. You need to set an appropriate prior for ASC0. In my syntax I have moved ASC0 to ACC, but that does not change the choice probabilities, they remain the same. It is finer having one utility set to zero because all utilities are relative to each other.

5. Correct, Ngene does not support constraints in a model averaging design. Yes you can make small changes to attribute levels manually as long as you do not create perfect correlations between attributes.

Michiel

[peyman_07]

Thanks Michiel for your detailed response. Just one more point about ASC0. For M1, ACS0 = -0.259 (p-value=0.83) and for M2, ASC0=2.18 (p-value=0.36). Both of them are insignificant. If we include them, the utility of DEC is positive in M2 which makes sense but it is negative for M1 which seems to be weird to me. I mean even if ASC0=0 in M1, the situation is better. May I know what you think? What would you suggest to deal with this issue?


Thanks.
Peyman.

[Michiel Bliemer]

Why is an ASC of negative weird? If you have mostly positive or negative utility components, the ASC will either be positive or negative to balance the other alternative. You need to look at the choice probabilities in Ngene, the sign of the ASC is irrelevant. I could add 10 utilities to both alternatives to make the ASC positive and still obtain the same choice probabilities. The value of the ASC also depends on which level you make the base level in dummy coding, so the ASC really does not mean much except to obtain appropriate choice probabilities.

[peyman_07]

Here are the choice probability for both models:

Code: Select all

MNL probabilities (m2)      
Choice   acc   dec
1   0.998127   0.001873
2   0.932267   0.067733
3   0.99863   0.00137
4   0.935535   0.064465
5   0.761722   0.238278
6   0.739696   0.260304
7   0.732078   0.267922
8   0.992198   0.007802
9   0.466139   0.533861
10   0.727582   0.272418
11   0.997743   0.002257
12   0.760241   0.239759
13   0.998127   0.001873
14   0.886507   0.113493
15   0.77512   0.22488
16   0.361041   0.638959
17   0.692118   0.307882
18   0.997397   0.002603
19   0.701854   0.298146
20   0.993161   0.006839
21   0.900885   0.099115
22   0.997632   0.002368
23   0.720973   0.279027
24   0.632113   0.367887

      
MNL probabilities (m1)      
Choice situation   acc   dec
1   0.882571   0.117429
2   0.719806   0.280194
3   0.876369   0.123631
4   0.73145   0.26855
5   0.821712   0.178288
6   0.7511   0.2489
7   0.764136   0.235864
8   0.901099   0.098901
9   0.750915   0.249085
10   0.879848   0.120152
11   0.824769   0.175231
12   0.701406   0.298594
13   0.905037   0.094963
14   0.840104   0.159896
15   0.870851   0.129149
16   0.726314   0.273686
17   0.679939   0.320061
18   0.847742   0.152258
19   0.716363   0.283637
20   0.88257   0.11743
21   0.64325   0.35675
22   0.783298   0.216702
23   0.787763   0.212237
24   0.657459   0.342541



Please note that the ASC0 is insignificant in both models. For M1, ACS0 = -0.259 (p-value=0.83) and for M2, ASC0=2.18 (p-value=0.36). It should be noted all of the priors except two in m2 are insignificant. Would you please let me know your thoughts?

Thanks.
Peyman.

[Michiel Bliemer]

You can see that the choice probabilities do not make sense for m2. This is because of the ASC set to zero. Your estimates indicaste that the value is large, despite it being insificant because your sample size was small.

You have two choices:

* You set appropriate non-zero priors for all parameters
* You set all priors to zero

If your priors are very unreliable and even of the wrong sign, then I think it is better to set them all to zero. Alternatively, you collect more data in your pilot study to obtain more reliable parameter estimates.

Unreliable estimates are also a result of the fact that you have many (too many?) attributes, and perhaps also the choice task is difficult to understand or may not make sense to respondents. Focus groups and pilot tests give important insights.

Michiel