Basic questions on a Bayesian design

[Keiko Aoki]

Dear Members,

Since I finished the pre-test, I estimated it by the MNL in Nlogit.
-----------------------------------------------------------------------------
NLOGIT
;Lhs=choice
;Choices=alt1,alt2,alt3,notbuy
; model : U(alt1,alt2,alt3)=
B
+KI*KI
+GD*GD
+EI*EI
+TE*TE+NA*NA ? dummies for attribute ri:0=no, 1=te, 2=na
+SI*SI
+KO*KO+OR*OR ? dummies for attribute es:0=ar, 1=ko, 2=or
+PL*PL
+PR*PR
/
U(notbuy)=0
$
Iterative procedure has converged
Normal exit: 6 iterations. Status=0, F= .2038222D+04

--------+--------------------------------------------------------------------
| Standard Prob. 95% Confidence
CHOICE| Coefficient Error z |z|>Z* Interval
--------+--------------------------------------------------------------------
B| 2.10233*** .17516 12.00 .0000 1.75903 2.44563
KI| .08684 .05573 1.56 .1192 -.02240 .19607
GD| .24071*** .05517 4.36 .0000 .13258 .34883
EI| .00047** .00020 2.41 .0159 .00009 .00086
ri:1 TE| .38714*** .06530 5.93 .0000 .25916 .51512
ri:2 NA| .29433*** .06582 4.47 .0000 .16533 .42333
SI| .10772** .05486 1.96 .0496 .00019 .21525
es:1 KO| .58714*** .06648 8.83 .0000 .45684 .71744
es:2 OR| .56279*** .06671 8.44 .0000 .43204 .69354
PL| -.10949** .05342 -2.05 .0404 -.21418 -.00479
PR| -.05311*** .00388 -13.70 .0000 -.06071 -.04551
--------+--------------------------------------------------------------------

Based on the estimation results, I am generating a Bayesian D efficient design
Also, to generate it, I checked several topics on the Bayesian design in this forum and the Ngene's manuals.
I generated an Bayesian design as below.
-----------------------------------------------------------------------------
Design
;alts = alt1*, alt2*, alt3*,notbuy ?unlabelled experiment
;rows = 12
;eff=(mnl, d, mean) ?Bayesian D-error
;block = 2
;bdraws = gauss(5) ?the type and number of draws for Bayesian prior parameters
;model:
U(alt1)
=ki.dummy[0.08684]*ki[1,0]
+gd.dummy[0.24071]*gd[1,0]
+ei[0.00047]*ei[163,326,489]
+ri.dummy[(n,0.29433,0.0682)|(n,0.38714,0.0653)]*ri[1,2,0]
+si.dummy[0.10772]*si[1,0]
+es.dummy[(n,0.58714,0.06648)|(n,0.56279,0.06671)]*es[1,2,0]
+pl.dummy[-0.10949]*pl[1,0]
+pr[-0.05311]*pr[9.8,14.8,19.8,24.8,29.8]/
U(alt2)
=ki.dummy*ki
+gd.dummy*gd
+ei*ei
+ri.dummy*ri
+si.dummy*si
+es.dummy*es
+pl.dummy*pl
+pr*pr/
U(alt3)
=ki.dummy*ki
+gd.dummy*gd
+ei*ei
+ri.dummy*ri
+si.dummy*si
+es.dummy*es
+pl.dummy*pl
+pr*pr
$

-----------------------------------------------------------------------------
So, I have a few elementary questions.

1. Is there each rule when I use "mean" or "median" in the the efficient design?
-That means how do I select.
2. How many numbers of gauss when the efficient design is mean?
-I learned if eff is mean, bdraws is gauss.
3. Are there any situations which a D-prior is better than a Bayesian one?
-I learned that a Bayesian is better than a D-prior thought I sometime find previous studies employed D-prior design, not Bayesian one.
4. Even if D error in the D-prior design is less than that in the Bayesian one, is the Bayesian design better?

I was wondering if you could comment in my questions.

Thank you for your time.

Best regards,
Keiko Aoki

[Michiel Bliemer]

1. You would always use 'mean', unless your Bayesian distribution is very wide, in which case extreme draws from the distribution may lead to extreme D-errors, and using 'median' would avoid such issues.

2. There are many types of Bayesian draws, my preferred ones are Gaussian quadrature and Sobol draws.
;bdraws = gauss(x) --> takes x draws per abscissa, which means that with K Bayesian priors it will do x^K draws.
;bdraws = sobol(x) --> takes x draws in total

The value of x you need depends on the number of Bayesian priors and the standard deviation of each distribution. Please consult Bliemer et al. (2008).

Bliemer & Rose & Hess (2008) Approximatino of Bayesian efficiency in experimental choice designs. Journal of Choice Modelling, 1, 98-126.

3. Using fixed priors does not require as much computation time, so it is easier to generate designs with fixed priors than Bayesian priors. Bayesian priors account for uncertainty about your priors, fixed priors do not.

4. Yes, a Bayesian efficient design is more robust against prior misspecification. This means that if your fixed priors deviate from the true parameter values (which they always do), they will lose their efficiency more quickly than a Bayesian efficient design. So the efficiency of the data collection using a Bayesian efficient design will be higher if the fixed priors deviate significantly from the true values.

Michiel

[Keiko Aoki]

Dear Professor Bliemer,

Thank you so much for answering my questions.
I checked your comments and the paper. But, I did not understand the gauss clearly.

So, I would like to ask further questions.

1. The larger R (the number of gauss), the more efficient (decreasing D error): Is that correct?
When I employ gauss(5) in 4 Bayesian priors, draws are 5*5*5*5=625.
If I want to be efficient design more, I employ that R is more than 5 (which means 6*6*6*6=1296,for example) or I employ more than 4 Bayesian priors (which means 5*5*5*5*5=3125,for example) .
However, in your paper (Table 1), whether gauss makes efficient depends on the number of Bayesian priors and distributions like normal and uniform one.
Also, in the forum, whether one becomes Bayesian priors among attributes depends on the attributes which influence the utility (which means that it is not good to increase the number of Bayesian priors easily).
Could you comment me when I use gauss?

2. Is there the best answer in the design?
I learned that the lower D error is better and that a Bayesian design is the most robust among designs.
But, I understood in the forum, which the best design depends on ones' purpose.
On the other hand, I understand that preferable designs need to estimate true preference.
Although I understood that a design is art, not science, I was not sure what is the best for the design. I think that it is good to employ the better design thought.
So, how do you generate a design if you could allow me to ask ?

3. Is my code on the Bayesian design correct?

Thank you for you time.

Best regards,
Keiko

[Michiel Bliemer]

1. No, the number of absciccas or draws has nothing to do with efficiency, it has to so with the accuracy of the computation of Bayesian efficiency. The higher R, the more accurate the Bayesian D-efficiency value will be.

2. I typically first conduct a pilot study, based on a D-efficient design with zero priors (or very small positive/negative values to indicate the sign, which in an unlabelled experiment allows avoiding dominant alternatives). Then i estimate a multinomial logit model and use these parameter estimates to set Bayesian priors for generating a Bayesian efficient design. I use this Bayesian efficient design for my main study.

3. The syntax looks good, but to make a few comments:
* Bayesian priors for ri1 and ri2 are perhaps swapped around? please check
* using 12 rows is very little for estimating so many coefficients, I would recommend ;rows = 24 and ;block = 4.
* You have 5 levels for price, which means price will not have attribute level balance. If you would use 4 levels or 6 levels it would have attribute level balance
* If you include a notbuy / optout alternative, where U(notbuy) = 0, then you MUST estimate a constant in alt1, alt2, and alt3, namely U(alt1) = const + ..., U(alt2) = const + .. , U(alt3) = const + ... (you add the same constant in all 3 alternatives). This means that you also must estimate this constant in your model and then add a prior for this constant.
* Note that your choice tasks are quite complex, you have 8 attributes and 3 alternatives (excl output), meaning that each respondent must evaluate 24 values in each choice task. Consider using alt1, alt2, nobuy.

Michiel

[Keiko Aoki]

Dear Professor Bliemer,

Thank you so much for answering my questions.

Although further questions are regarding 1, Can I ask you about a way of understanding Table 1 (Bliemer, Ross, and Hess, 2008)?
Now, I have understood that Table 1 shows which better for distributions based on abscissas Rk and weights Wk(r) for k bayesian parameters.
So, I am wondering that number of weight has one to five regarding Rk. How do I interpret them.
Also, for example, if the number of weight in the normal distribution is higher than that in the uniform one, should I set the normal distribution in the Bayesian priors which I set?

Regarding 2, I referred your procedures on a design. I appreciate you so much.

Regarding 3, I inversely labeled r1 and r2 to these coefficients in estimation results (i.e., r1=Na, r2=TE). Thank you.
I will discuss your advice like the number of alternatives and rows with my co-others.
Also, regarding opt-out option, I misunderstood the explanation in the manual. So, I generated the Bayesian design again based on your explanations. Is that correct?

Some comments would be appreciated if you could.

Best regards,
Keiko
----The new design as below:
Design
;alts = alt1*, alt2*, alt3*,notbuy ?unlabelled experiment
;rows = 12
;eff=(mnl, d, mean) ?Bayesian D-error
;block = 2
;bdraws = gauss(5) ?the type and number of draws for Bayesian prior parameters
;model:
U(alt1)
=b[2.10233]
+ki.dummy[0.08684]*ki[1,0]
+gd.dummy[0.24071]*gd[1,0]
+ei[0.00047]*ei[163,326,489]
+ri.dummy[(n,0.29433,0.0682)|(n,0.38714,0.0653)]*ri[1,2,0] ?1=Na,2=teion
+si.dummy[0.10772]*si[1,0]
+es.dummy[(n,0.58714,0.06648)|(n,0.56279,0.06671)]*es[1,2,0] ?1=kousei-nashi,2=oraganic
+pl.dummy[-0.10949]*pl[1,0]
+pr[-0.05311]*pr[9.8,14.8,19.8,24.8,29.8]/
U(alt2)
=b
+ki.dummy*ki
+gd.dummy*gd
+ei*ei
+ri.dummy*ri
+si.dummy*si
+es.dummy*es
+pl.dummy*pl
+pr*pr/
U(alt3)
=b
+ki.dummy*ki
+gd.dummy*gd
+ei*ei
+ri.dummy*ri
+si.dummy*si
+es.dummy*es
+pl.dummy*pl
+pr*pr
$

[Michiel Bliemer]

1. Gaussian quadrature is an extremely smart method for approximating an integral. Instead of taking random or quasi-random draws, it takes very specific draws (abscissas) and applies a weight to these draws that is specific to a distribution. This way, Gaussian quadrature can establish an accurate value for the integral with much fewer draws. If you would like to know more about how these weights are determined, then please search online for Gaussian quadrature, but note that the theory behind this is highly mathematical and not easy to understand. The values from Table 1 in our paper comes from the literature, we did not compute these ourselves.

3. Yes the syntax looks good, the constant is in the right place. Looking at the probabilities, the optout is selected in about 5% of the cases (you can see this by clicking on Probabilities under MNL properties when you open a design), hopefully that is consistent with the data that you have.

Michiel

[Keiko Aoki]

Dear Professor Bliemer,

Thank you so much!
Regarding 1, I understand.
Regarding 3, I checked the Probabilities for notbuy alternative in Ngene and them in our pilot data. As you said, they are almost the same (0.038 in Ngene and 0.036 in the pilot for average notbuy). Fantastic!!
This probabilities as well as utilities should be useful to generate code and to apply further researches likes the bass model.

I really appreciate you.

Best regards,
Keiko