Bayesian efficient designs

[Sumitra]

Good morning,

I’m new to the forum, and incidentally also new to Ngene and DCE designs in general. Apologies if my queries seem trivial.

I am attempting to improve the current design of a DCE by using the priors from a previous similar study.
However, I am running into some issues.

For e.g., a factorial design achieves a better efficiency than the Bayesian model (see below).
1. Could it be that I have made a mistake in my syntax? For e.g. I am not sure I have interpreted how to use the utility equations, here I’ve assumed 2 generic functions and replicated them except for the intercept, is that the correct way to do it? In the case I am modeeling, we will have several alternatives and we will compare them two by do.
2. The Bayesian design generated seems to produce a dominated option, which is not informative, what could be the cause of this? We have tried tweaking the priors to narrow the range and also used a much smaller standard deviations to see if that would help. While that helped a bit, it is still not as good as the factorial design.
3. Perhaps this also could be due to the fact that the similar study we are using is not identical - that is, the levels we are using are different and we had to extrapolate their values.

Any help or tips would be greatly appreciated, especially if the syntax is incorrect.



//Factorial design: D-error 0.33//

Design
;alts = DMT1, DMT2
;rows=14
;fact
;model:

U(DMT1) = b1
+ b2 * ARR[0,1,2]
+ b3 * Prog[0,1,2]
+ b4 * Les[0,1,2]
+ b5 * NSRev[0,1]
+ b6 * SRev[0,1]
+ b7 * Nrev[0,1]
+ b8 * Freq[0,1,2,3,4]/

U(DMT2) = b2 * ARR
+ b3 * Prog
+ b4 * Les
+ b5 * NSRev
+ b6 * SRev
+ b7 * Nrev
+ b8 * Freq
$

// Bayesian design: D-error 0.69//

Design
;alts = DMT1, DMT2
;rows=14

? Bayesian MNL type model
;eff=(mnl,d,mean)
;rdraws=mlhs(250)
;model:

U(DMT1) = b1
+ b2.dummy[(n,0.23,0.05)|(n,0.5,0.05)] * ARR[0,1,2]
+ b3.dummy[(n,1,0.05)|(n,0.5,0.05)] * Prog[0,1,2]
+ b4.dummy[(n,0.23,0.05)|(n,0.42,0.05)] * Les[0,1,2]
+ b5.dummy[(n,0.26,0.05)] * NSRev[0,1]
+ b6.dummy[(n,1,0.05)] * SRev[0,1]
+ b7.dummy[(n,0.75,0.05)] * Nrev[0,1]
+ b8.dummy[(n,0.1,0.05)|(n,0.3,0.05)|(n,0.6,0.05)|(n,0.8,0.05)] * Freq[0,1,2,3,4]/


U(DMT2) = b2 * ARR
+ b3 * Prog
+ b4 * Les
+ b5 * NSRev
+ b6 * SRev
+ b7 * Nrev
+ b8 * Freq
$


Thank you very much,
Sumitra

[Michiel Bliemer]

1. There is no mistake in your syntax, but you are comparing apples and oranges. You can only compare D-errors assuming the same priors. Now you are using zero priors for the random fractional factorial design (;rows=14 together with ;fact generates 14 random choice tasks from the full factorial) and non-zero priors for the Bayesian efficient design. If you want to compare, you should set the priors for the efficient design to zero as well, or for the random design to non-zero as well.

2. You did not specify that Ngene should remove dominant alternatives. You should not change the Bayesian priors, but rather use ;alts =DMT1*, DMT2*. The * tells Ngene that these alternatives are generic and require checking for dominancy. Note that you simply got lucky that the random fractional factorial design did not generate any dominant alternatives, as such a design does not have any intelligence in generating choice tasks.

3. One should always be careful taking priors from other studies, and use conservative values (i.e. divide them by half to make them closer to zero). There could be different units used, different attribute level range, different scale parameters, etc., so using conservative values at least ensures they are not too large (which could be problematic). Too small is usually not problematic.

I think there are some changes needed in your syntax. My suggestion is below. Since you are estimating 13 parameters (I removed the constant since the study seems to have unlabelled alternatives, so that I would optimise the design for a model without constants), using 14 rows is quite little (it barely meets the degrees of freedom requirement). Since you have 2, 3, and 5 levels, I used 30 rows such that the design is attribute level balanced, and blocked it in 2 such that each respondent faces 15 choice tasks (or block it in smaller pieces if you think this is too much). You used ;rdraws, but this is from random parameters (mixed logit), you should use ;bdraws for Bayesian priors. You have a very large number of Bayesian priors (13), so this means you need to do a LOT of draws. The minimum is 2^13 = 8,192, but better would be 3^13 = 1,594,323 draws per design. The latter is too large for optimisation, so I would try to cut down the number of Bayesian priors, or use the minimum of 2 draws per dimension using Gaussian quadrature, which is done via ;bdraws = gauss(2).

This Bayesian design will give you a D-error of 0.32 or something, and it should not contain any dominant alternatives. If you replace ;eff = (mnl,d,mean) with ;fact, then it generates a random design and in my case it gives a D-error of 0.59. Clearly, since it is random, if you run it again, it gives a completely different design and different D-error.

Good luck.

Design
;alts = DMT1*, DMT2*
;rows=30
;block=2
;eff=(mnl,d,mean)
;bdraws=gauss(2)
;model:

U(DMT1) = b2.dummy[(n,0.23,0.05)|(n,0.5,0.05)] * ARR[0,1,2]
+ b3.dummy[(n,1,0.05)|(n,0.5,0.05)] * Prog[0,1,2]
+ b4.dummy[(n,0.23,0.05)|(n,0.42,0.05)] * Les[0,1,2]
+ b5.dummy[(n,0.26,0.05)] * NSRev[0,1]
+ b6.dummy[(n,1,0.05)] * SRev[0,1]
+ b7.dummy[(n,0.75,0.05)] * Nrev[0,1]
+ b8.dummy[(n,0.1,0.05)|(n,0.3,0.05)|(n,0.6,0.05)|(n,0.8,0.05)] * Freq[0,1,2,3,4]/

U(DMT2) = b2 * ARR
+ b3 * Prog
+ b4 * Les
+ b5 * NSRev
+ b6 * SRev
+ b7 * Nrev
+ b8 * Freq
$

[Sumitra]

Dear Michiel

Thank you so much for this detailed response. It has given us a lot of food for thought.

We have been working on implementing the changes you suggested and taken the feedback on for the other types of designs we tested and are analysing the results. No doubt we may have other questions

Thank you again very much,
Sumitra