How to choose the Bayesian experiment design?

[sukunta]

Dear Prof. Michiel,
I run the syntax that is

Code: Select all

design
;alts=alt1*,alt2*,alt3
;rows=6
;eff = (mnl,d,mean)
;con
;bdraws = gauss(3)
;model:
U(alt1)=b0 [(n,5.51892, 2.400485)]+b1.dummy[(n,-.1969752, 0.6256593) |(n, -1.777203, 1.348568)]*mode[1,2,0]+b2.dummy[(n, -0.0287256, 0.20309630)]*format[1,0]+b3.dummy[(n,1.368347, 1.700064)]*con_ser[1,0]+b4[(n,-.0268496, .0316639)]*time[10,20,30]+ b5 [(n,-.0297015, .0246112)]*cost[0,50,100]/
U(alt2)= b0 [(n,5.51892, 2.400485)]+b1*mode+b2*format+b3*con_ser+b4*time+b5*cost
$


I run for around 1 hour. The mean bayesian MNL D-error value between 3.35698- 0.76954.
My question is
1. Can I choose the model that has the mean bayesian MNL D-error value 0.84018 (use around 1 minute)?
2. What are the MNL efficiency measure parameters that I should be observing or be aware of?
Sincerely yours,
Sukunta

[Michiel Bliemer]

Dear Sukunta,

I notice that some of your bayesian priors have a very large standard deviation. I assume that these come from a pilot study and that you have used (n,parameter,standarderror) as Bayesian priors? Please check.

With such large standard deviations, it is quite likely that some of your draws will lead to quite extreme values the coefficients, which results in issues in generating a design because some attributes will become dominant and choice probabilties will be pushed towards 0 and 1.

One way to limit the issue is to use the median Bayesian D-error instead of the mean Bayesian D-error. Medians cannot be computed using Gaussian draws, so in this case I suggest switching to Sobol draws.

Another issue is the number of rows used. While with 6 rows you can estimate a choice model, it is really the bare minimum and there will not be much variation in your data. I suggest using 12 or 24 rows, blocking the design in 2 or 4 (such that each respondent still faces 6 choice tasks, you simply create 2 or 4 different versions of the survey questions).

For example, see the syntax below.

Code: Select all

design
;alts=alt1*,alt2*,alt3
;rows=12
;block=2
;eff = (mnl,d,median)
;con
;bdraws = sobol(5000)
;model:
U(alt1)=b0 [(n,5.51892, 2.400485)]+b1.dummy[(n,-.1969752, 0.6256593) |(n, -1.777203, 1.348568)]*mode[1,2,0]+b2.dummy[(n, -0.0287256, 0.20309630)]*format[1,0]+b3.dummy[(n,1.368347, 1.700064)]*con_ser[1,0]+b4[(n,-.0268496, .0316639)]*time[10,20,30]+ b5 [(n,-.0297015, .0246112)]*cost[0,50,100]/
U(alt2)= b0 [(n,5.51892, 2.400485)]+b1*mode+b2*format+b3*con_ser+b4*time+b5*cost
$

I would run the syntax for at least an hour (there is no harm in running it overnight). Once the Bayesian D-error has stabilised (i.e. doesn't decrease much anymore), you can stop and pick the design with the lowest Bayesian D-error. I often also look at the Sp estimates, which give you an idea about the required sample size needed for statistical significance (at 95% confidence level) for each parameter (assuming that your priors reflect the true parameter values). Note that Sp-estimates assume that a respondent faces all questions in the design, so if you have rows = 12 and block = 2, then you need to multiply the Sp-estimates by 2 to get an indication for the required number of respondents.

Michiel

[sukunta]

Dear Prof.Michiel,
Thank you so much. I will run this syntax from your suggestion (running it overnight). The choice set more than 6 are very burdened. Moreover, I have a limitation on the number of respondents (300). Thus I design to 6 rows in this model. I will look at the result of Sp estimate again and consult you again.
Sincerely yours,
Sukunta

[Michiel Bliemer]

I strongly advise against using 6 rows, you will have a lot of difficulties estimating your model because the coefficients are hardly identifiable. As I wrote in my comment, by using ;block = 2, YOU ARE ONLY SHOWING 6 CHOICE TASKS TO EACH RESPONDENT, but you have 2 versions of the survey. Most online survey instruments can easily handle that. You can randomly assign a version if you like. Having 12 different questions in your dataset will substantially improve the identifiability of your coefficients.

Michiel

[sukunta]

Dear Prof.Michiel,
The result of the running model 13 hours (6.49 AM -7.43 PM) that proposed a Bayesian median error between 0.35622-0.196023.
Sp estimate max b2(d0) = 1917.362119. My questions are
1.The sample size from this result should be = 1918 x2 ?
2. If I have a limitation the sample size around 300, Can I develop this model on 6 rows?

Sincerely yours,
Sukunta

[Michiel Bliemer]

Did you use ;rows = 6 or ;rows = 12?

Using 12 rows will improve your sample size requirements, not make it worse. While you have to multiply the Sp estimates with 2 if you use ;rows = 12 and ;block = 2, the Sp estimates will be much lower than when you use ;rows = 6.

Note that sample size estimates need to be taken with a grain of salt because they rely on the correct specification of your coefficients. If b2(d0) = -0.0287256 is the true value, then the sample size estimate will indeed be 1917*2 if you used ;rows = 12 and ;block = 2. But note that -0.0287256 is an extremely small value, i.e. this attribute level is not important in making a choice according to your priors, so if this is the true value then it will be almost impossible to estimate this parameter with a high level of statistical significance because respondents believe it is not important. However, the standard error of this dummy variable indicates that there is a high level of uncertainty about its exact value, which means that the Sp estimates for this parameter will be very unreliable and I would not worry about it.

Michiel

[sukunta]

Dear Prof.Michiel,
I use ;rows = 12 and ;block = 2.The Db-error of evalution 3446 = 0.197265.

The Sp estimate for b0 =1.33, b1(d0)= 72.49, b1(d1)=2.55, b2 (d0)= 2015.86,b3(d0)=2.82, b4=10.91, b5=2.18

. My questions are
1.When I should stop running this syntax? How many the Db -error that suitable?
2.If the sp estimate for b2 (d0) is very unreliable, Tne minimum number of respondents for the rest b estimate is 72x2?
3.If I have a 300 respondents , Can I use ;row 6 for this model?

Sincerely yours,
Sukunta

[Michiel Bliemer]

1. I already answered this question earlier: "I would run the syntax for at least an hour (there is no harm in running it overnight). Once the Bayesian D-error has stabilised (i.e. doesn't decrease much anymore), you can stop and pick the design with the lowest Bayesian D-error. " Note that the D-error is case specific and I cannot tell you what a good D-error is in your case; in one case, 0.1 is very good, in another case 0.1 is very bad. It is the same for loglikelihood values in estimation, it does not have a meaning except that higher is better. In the case of D-error, the lower the better.

2. Yes

3. I do not understand this question. If you use ;rows = 12, then you give 150 respondents the first block of 6 rows and the other 150 respondents the second block of 6 rows. Look at the blocking column in the Ngene output and give all rows with the blocking = 1 to one respondent and all rows with blocking = 2 to another respondent.

Michiel

[sukunta]

Dear Prof. Michiel,
I plan to study the external validity of the DCE. I have 2 models between
1.Row 6 ,Db median error = 0.640401, Sp estimate (except b2 (d0))=197.3
2.Rows 12, block 2 ,Db median error = 0.197265 , Sp estimate (except b2 (d0))= 72x2 = 144
I must choose 1 choice set from the model to propose the actual scenario for the respondents. Thus, the heterogeneous choice design
may be a limitation for me to set the actual scenario and 12 choice sets may be burdensome for respondents. My respondents vary in preferences, thus the heterogeneous design may not be fit for my study. My questions are
1. Can I choose the homogeneous design with 6 choice sets, Db median error = 0.640401, Sp estimate (except b2 (d0))=197.3.?
2. If I change to used MMNL, I should analyze the pilot study model in a mixed logit model?
Please, recommend me, if I do not understand well.
Sincerely yours,
Sukunta

[Michiel Bliemer]

For showing an example choice task, you can simply manually create a choice task as you would generally ignore the response to the example choice task from your survey. So you have 12 choice tasks + 1 example choice task.

As I stated twice previously, there is no need to show respondents 12 choice tasks, you show 6 choice tasks to one respondent and 6 to another respondent. I do not understand why a heterogeneous design would not be suitable for you.

I would not recommend using a design with only 6 choice tasks. It is unlikely that you can estimate an MMNL model based on a design with only 6 choice tasks, I would recommend using AT LEAST 12 choice tasks (and preferably more, e.g. 24 choice tasks in 4 blocks of 6).

Michiel