Bayesian Priors from Pilot Study into Design

[Michiel Bliemer]

To answer your second questions,

(1) You should not use wtp if your priors are near-zero since you are dividing by a very small number and the WTP values are meaningless. You should only use WTP optimsation when your priors are properly set based on a pilot study

(2) You should put all attributes and levels in the status quo alternative since you also want to check for dominance with respect to the Default alternative. I have done that in the syntax below. It does not matter where you put the constant, either in both A and B or in Default.

There are some issues with your syntax. The main issue is with respect to your conditional constraints. They imply that cl=0.2 always needs to appear with co=2, which creates multicollinearity and you cannot estimate the model. I have rewritten your conditional constraints into reject constraints (since the modified Federov algorithm is needed to use the require command to fix the attribute levels of the dummy coded attributes in Default). I have commented out these constraints since leaving them in leads to an Undefined D-error since the model is not estimable.

Code: Select all

Design

;alts = A*, B*, Default*
;rows = 24
;block= 6 ,minsum

;eff = (mnl, d)
;con

;alg = mfederov
?;reject:
?A.cl = .5, A.co = 2,
?A.cl =  1, A.co = 2,
?B.cl = .5, B.co = 2,
?B.cl =  1, B.co = 2
;require:
Default.s = 0,
Default.co = 0,
Default.po = 0

;model:
U(A) =       b1[.001]                * cl[.2,.5,1]?(6-10,6-10,6-10)           
           + b2.dummy[0.001|-0.002]  * s[1,2,0]   
           + b3.dummy[0.001|-0.002]  * co[1,2,0] 
           + b4.dummy[0.001|0.002]   * po[1,2,0] 
           + b5[-.001]               * pr[0, 10, 20, 50, 75, 100]?(3-5,3-5,3-5,3-5,3-5,3-5)
           /
U(B)       = b1 * cl
           + b2 * s
           + b3 * co
           + b4 * po
           + b5 * pr
           /
U(Default) = b0[-0.001]
           + b1 * cl_default[.2]
           + b2 * s
           + b3 * co
           + b4 * po
           + b5 * pr_default[0]
$ 

Michiel

[jamalm]

Michiel,

Thank you much for answering all my questions and also looking over my codes. I relaxed the constraint and re-ran the design. What's surprising to me is that Ngene does not find a level balanced design in what should have been a fairly simple task to achieve (4 attributes @ 3 levels and 1 @ 6 levels). Will this issue of level imbalance resolves once the Bayesian design is made with the actual priors from the pretest?

Further, the following does lead to a level balanced design and it is faster than the suggested design with ;alg = mfederov. I am not sure why I should include the dummy variables in the U(default) equation as their base value of zero results in zero utility anyways. However, it is evident that there is a difference between the design you suggested and the one below. Any further guidance is highly appreciated.

Code: Select all

Design 
;alts = A*, B*, Default*
;rows = 24
;block=6 ,minsum
;eff = (mnl, d)
;con

?;cond:
?if(a.cl=[.5,1], a.co=[1,0]),
?if(b.cl=[.5,1], b.co=[1,0])

;bdraws=sobol(2000)
;model:
U(A) =
            b1[.001]    * cl[.2,.5,1]            +           
            b2.dummy[0.001|-0.002]  * s[1,2,0]   +           
            b3.dummy[0.001|-0.002]  * co[1,2,0]  + 
            b4.dummy[0.001|0.002]   * po[1,2,0]  +
            b5[-.001] * pr[0, 10, 20, 50, 75, 100] 
          /
U(B) =
            b1  * cl  +             
            b2 * s    + 
            b3  * co  + 
            b4* po    +
            b5* pr
          /
 U(Default) =
             b0[-0.001] +
             b1 * cl_default[.2]
$

-Jamal

[Michiel Bliemer]

Dominant alternatives occur more often if:

* the number of attributes is small
* the number of attribute levels is small
* the number of alternatives is large

In your case, the number of attributes and levels is small, and with 3 alternatives (including the status quo) it is very likely that there will be dominant alternatives if dominance constraints are not imposed. In my syntax I have imposed dominance constraints (using the priors you have provided), so there should be no choice tasks with any dominance. If there still exists dominance, then your priors are not set properly. Bayesian priors would not resolve the issue.

Michiel

[jamalm]

I can only find priors for a few of the attributes and not all of them. Per Ngene's manual, I opted to include near-zero priors for all attributes and not a mixture of near-zero and literature priors.

Further, the following does lead to a level balanced design and it is faster than the suggested design with ;alg = mfederov. I am not sure why I should include the dummy variables in the U(default) equation as their base value of zero results in zero utility anyways. However, it is evident that there is a difference between the design you suggested and the one below. Any further guidance is highly appreciated.

Code: Select all

Design 
;alts = A*, B*, Default*
;rows = 24
;block=6 ,minsum
;eff = (mnl, d)
;con

?;cond:
?if(a.cl=[.5,1], a.co=[1,0]),
?if(b.cl=[.5,1], b.co=[1,0])

;bdraws=sobol(2000)
;model:
U(A) =
            b1[.001]    * cl[.2,.5,1]            +           
            b2.dummy[0.001|-0.002]  * s[1,2,0]   +           
            b3.dummy[0.001|-0.002]  * co[1,2,0]  + 
            b4.dummy[0.001|0.002]   * po[1,2,0]  +
            b5[-.001] * pr[0, 10, 20, 50, 75, 100] 
          /
U(B) =
            b1  * cl  +             
            b2 * s    + 
            b3  * co  + 
            b4* po    +
            b5* pr
          /
 U(Default) =
             b0[-0.001] +
             b1 * cl_default[.2]
$

-Jamal

[Michiel Bliemer]

If you do not put in all attributes in the Default alternative, Ngene will not be able to check for dominance between A and Default and B and Default, see my previous comment. Therefore, Default could become a dominant alternative.

This is shown in the warning that Ngene produces when I run your syntax:

Warning: Two alternatives were specified for alternative dominance checking, but do not have the same priors, and so will not be checked. 'a', 'default'
Warning: Two alternatives were specified for alternative dominance checking, but do not have the same priors, and so will not be checked. 'b', 'default'

The design generated with my syntax will not have dominant alternatives. If your design does not have any dominant alternatives then that's fine, but I suspect there may be some.

Michiel

[jamalm]

That makes sense, thanks a bunch!
-Jamal

[jamalm]

Hello Michiel,

Following your earlier advice, I have revised my pilot design to what follows.

Code: Select all

Design

;alts = A*, B*, Default*
;rows = 24
;block= 4 ,minsum

;eff = (mnl, d)
;con

;alg = mfederov
;require:
Default.cl = .4,
Default.re = .6,
Default.po = 0,
Default.jo = 0

;model:
U(A) =       b1[.001]                        * bv[.02,.5,1] (7-9,7-9,7-9)           
           + b2.dummy[0.001|0.002]   * cl[.7,1,.4]     
           + b3.dummy[0.001|0.002]   * re[.8,1,.6]           
           + b4.dummy[0.001|0.001]   * po[1,2,0]
           + b5.dummy[0.001|-0.002]  * jo[1,2,0]
           + b6[-.001]                          * pr[0, 10, 20, 50, 75, 100] (3-5,3-5,3-5,3-5,3-5,3-5)
           + b7[-0.002]                        * cl*re
           /
U(B)       = b1 * bv
           + b2 * cl
           + b3 * re
           + b4 * po
           + b5 * jo
           + b6 * pr
           + b7 * cl*re
           /
U(Default) = b0[-0.001]
           + b1 * bv_default[.02]
           + b2 * cl
           + b3 * re
           + b4 * po
           + b5 * jo
           + b6 * pr_default[0]
$


I have a few questions.

1- Ngene gives me 4 lines of warning that it won't check for alt. dominance. Is that because I have an interaction variable in both U(A) & U(B) and not in the U(Default)? How can I have Ngene check for alt. dominance?

2- I originally had my cl & re attributes as continuos (see below), but Ngene wouldn't find a (relatively) balanced design. I even tried to add (7-9,7-9,7-9) to the end of their lines, but it wouldn't find any designs.

Code: Select all

              + b2[.001] * cl[.4,.7,1]             
              + b3[.001] * re[.6,.8,1]                 
     

since the difference between the status quo level and the other two levels are equal (.3 for cl & .2 for re), I thought to convert them to dummy variables. So, cl would have the following three levels: "30% increase from the status quo level", "60% increase from the status quo level", and "no change". Can I leave the actual levels (rather than changing them to 1,2,0) in the design and convert the betas to dummies similar to how I have included cl & re in my original codes above? I like converting them to dummies better as Ngene includes all the levels in the final design; I am unsure if the codes are correct, especially because I cannot find any similar codes in the Ngene manual.

2-1 Related to my second question: My other concern would be that I will not be able to estimate a continuous MWTP anymore, but rather MWTP at three levels only. Is that right?

3- Do you see any red flags in my codes?

4- A generic question: Is there any gain in having a large number of rows? I believe 12 rows should do here.

Thanks much in advance for your help.
Jamal

[Michiel Bliemer]

Hi Jamal,

1. Dominance can only be checked if for unlabelled alternatives, which need to have the same utility function (same parameters and attributes, but not necessarily same levels). So indeed you need to add the interaction effect in Default.

2. It is quite challenging for Ngene to find a design that satisfies attribute level balance constraints, so there are two options: (i) convert to dummy coding, or (ii) use a soft constraint in the optimisation route, something like:

;eff = 2*(mnl,d) + 1*(imbalance)

which minimses the D-error (multiplied by 2) plus a measure for attribute level imbalance (see Ngene manual). You can play with the multipliers to make the design more balanced but less efficient. I suggest you first give this a try to see if you are happy with the attribute level balance in this design. You can run the syntax below where I also added the interaction effect in Default and added require constraints.

Code: Select all

    Design

    ;alts = A*, B*, Default*
    ;rows = 24
    ;block= 4 ,minsum

    ;eff = 2*(mnl, d) + 1*(imbalance)
    ;con

    ;alg = mfederov
    ;require:
    Default.bv = 0.02,
    Default.cl = .4,
    Default.re = .6,
    Default.po = 0,
    Default.jo = 0,
    Default.pr = 0

    ;model:
    U(A) =       b1[0.001]               * bv[.02,.5,1]
               + b2.dummy[0.001|0.002]   * cl[.7,1,.4]     
               + b3.dummy[0.001|0.002]   * re[.8,1,.6]           
               + b4.dummy[0.001|0.001]   * po[1,2,0]
               + b5.dummy[0.001|-0.002]  * jo[1,2,0]
               + b6[-.001]               * pr[0, 10, 20, 50, 75, 100]
               + b7[-0.002]              * cl*re
               /
    U(B)       = b1 * bv
               + b2 * cl
               + b3 * re
               + b4 * po
               + b5 * jo
               + b6 * pr
               + b7 * cl*re
               /
    U(Default) = b0[-0.001]
               + b1 * bv
               + b2 * cl
               + b3 * re
               + b4 * po
               + b5 * jo
               + b6 * pr
               + b7 * cl*re
    $

If this does not give you sufficient attribute level balance, then dummy coding is the best way forward. You have done this correctly in your syntax for the other attributes, so it would become b1.dummy[0.001|0.002] * bv[.5,1,.02].
A design that is optimised for dummy coding can also be used to estimate a model with linear continuous variables so that you can still obtain a single continuous MWTP as you desire. You do not need to use dummy coding in model estimation, but you can if you want to.

3. No it looks good.

4. More rows means more variation in the data. This is usually a good thing, especially if you will be estimating more advanced choice models (mixed logit, latent class), which have more parameters. I would stick with 24 rows.

Michiel

[jamalm]

Michiel,

Thank you very much for your prompt and thorough explanation as always. I have learned a great deal from my correspondence with you.

Regards,
Jamal

[jamalm]

Hi Michiel,

I presented my preliminary design at a conference recently and received some good feedback. One, in particular, was that I should change my number of alternatives per choice from "A, B, Default" to "A & Default." The concern was stemming from the literature on incentive compatibility of choices. I am planning to use the following design for the pilot study.

Code: Select all

Design
;alts = A*, Default*
;rows = 24
;block= 6 ,minsum

;eff = 2*(mnl, d) + 1*(imbalance)
;con

;alg = mfederov(candidates=15000000)
    ;require:
    Default.bv = 0.03,
    Default.cl = .4,
    Default.sr = 0,
    Default.po = 0,
    Default.jo = 0,
    Default.pr = 0

   ;model:
    U(A) =       b1[0.001]               * bv[.03,.5,1]
               + b2.dummy[0.001|0.002]   * cl[.7,1,.4]     
               + b3.dummy[0.001|-0.002]   * sr[1,2,0]     
               + b4.dummy[0.001|0.001]   * po[1,2,0]
               + b5.dummy[0.001|-0.002]  * jo[1,2,0]
               + b6[-.001]               * pr[0, 5, 15, 30, 50, 75]
               /

    U(Default) = b0[-0.001]
               + b1 * bv
               + b2 * cl
               + b3 * sr
               + b4 * po
               + b5 * jo
               + b6 * pr
    $


When I had three alternatives per choice, Ngene was happy with the default number of candidates for the mfederov algorithm. However, the lowest number of candidates with two alternative/choice design that Ngene accepts is 1.5 million now, which seems rather large. Is this of concern? Is there a better way of designing that I do not find in the Ngene manual?


Thank you for your time and help as always.

Best,
Jamal