Bayesian design after pilot study

[atan]

Hi all,

I am conducing an unlabelled experiment with two alternatives, 8 attributes (A-H) and 2-4 levels per attribute. These are all categorical variables that I have dummy coded. I also have a scenario with interaction terms associated with different attribute levels. I have conducted my pilot study and used STATA18 to conducted a conditional logit regression..

Code: Select all

design

;alts = service1* , service2*
;rows = 60
;block = 6
;alg = mfederov (candidates = PP Candidate set V5_Nov 2023.csv)
;eff = (mnl,d,mean)
;bdraws = gauss(2)

;require:
service1.I = service2.I                                           

;model:
U(service1) = b1.dummy[1.12|0.93|0.58] * A[3,2,1,0]
            + i1[-0.99] * A.dummy[3] * I[1,0]                                         
            + i2[-0.72]  * A.dummy[2] * I[1,0]                 
            + i3[(n,-0.91,0.37)]  * A.dummy[1] * I[1,0]
           
            + b2.dummy[0.42|0.598] * B[2,1,0]
            + i4[-0.08] * B.dummy[2] * I[1,0]
            + i5[0.03] * B.dummy[1] * I[1,0]

            + b3.dummy[(n,-0.008,0.2376)] * C[1,0]

            + b4.dummy[1.03|0.88] * D[2,1,0]           
            + i6[-1.07] * D.dummy[2] * I[1,0]                           
            + i7[-0.74] * D.dummy[1] * I[1,0]

            + b5.dummy[0.00035|(n,0.16,0.34)] * E[2,1,0]

            + b6.dummy[0.19|(n,-0.89,0.58)|(n,-0.59,0.57)] * F[3,2,1,0]
            + i8[(n,0.24,0.56)] * F.dummy[3] *I[1,0]
            + i9[1.65]  * F.dummy[2] *I[1,0]
            + i10[(n,0.46,0.72)]  * F.dummy[1] *I[1,0]
 
            + b7.dummy[0.97|0.74|1.18] * G[3,2,1,0]
            + i11[-0.43] * G.dummy[3] * I[1,0]
            + i12[0.39]  * G.dummy[2] * I[1,0]
            + i13[(n,-0.16,0.81)] * G.dummy[1] * I[1,0]

            + b8.dummy[0.67|0.41|(n,0.21,0.34)] * H[3,2,1,0]
/
U(service2) = b1  * A
            + i1  * A.dummy[3] * I
            + i2  * A.dummy[2] * I
            + i3  * A.dummy[1] * I
            + b2  * B
            + i4  * B.dummy[2] * I
            + i5  * B.dummy[1] * I
            + b3  * C
            + b4  * D
            + i6  * D.dummy[2] * I
            + i7  * D.dummy[1] * I 
            + b5  * E
            + b6  * F
            + i8 * F.dummy[3] * I
            + i9 * F.dummy[2] * I
            + i10 * F.dummy[1] * I
            + b7  * G
            + i11 * G.dummy[3] * I
            + i12 * G.dummy[2] * I
            + i13 * G.dummy[1] * I
            + b8  * H
$


I want to conduct a Bayesian efficient design for the main survey, however less than half of my co-efficients are statistically signifcant.

I have assessed and reviewed the coefficients and only assigned Bayesian priors for values that were unexpected, e.g. negative sign and magnitude in comparison to other attribute levels. In total there are 9, which is in line with what you have suggested in previous posts for Bayesian priors <=10

My B-estimates are within range, however the d-error and s-estimates are very big which is concerning. Could you possibly review my syntax and provide any suggestions on where I should relax the Bayesian priors further, or make any suggestions for this study.

Thanks,

Annie

[Michiel Bliemer]

The D-error that I can see is around 0.6 or 0.7, which looks fine and the S-estimates are also small for most parameters. The sample size estimates will of course be very large for parameters with priors very close to zero. If the true prior is indeed near-zero then this attribute level does not matter in choice and hence you will not be able to estimate a significant effect for it. This holds for any attribute that is irrelevant for choice and can simply be an outcome of your study so you should not worry about it.

If your D-error is much larger than the one I can see, then there may be an issue with your candidate set (possibly near-perfect correlations between some attributes). I do not have your CSV file so I cannot see the D-errors that you obtain.

Michiel

[atan]

Hi Michiel,

Thanks for the reply. I'm not sure how to attach the csv file for you to review.

Below is the output from one of the iterations from the Ngene syntax after running it for a couple of hours. I'm not sure what to make of the s-estimate as it is so large.

MNL efficiency measures

Fixed Bayesian mean
D error 0.60257 0.655214
A error 1.589435 1.7255
B estimates 86.171131 0.803145
S estimate 31294543.497048 34429796.166538

Thank you kindly,

Annie

[Michiel Bliemer]

That output looks good so no need to send me the CSV file.

You should NOT look at the S-estimate, you should look at the Sp estimates for each parameter separately, they are listed directly below the output that you copied here. Those Sp estimates are much more important than the overall S-estimate, which is simply the maximum of all Sp estimates.

Some of your priors are very close to zero and therefore will have an extremely large Sp estimate and you should ignore such sample size estimates. Only parameters that have a reliable prior (i.e. that are statistically significant in your pilot) will have a meaningful sample size estimate.

Michiel