A way of decreasing D-error

[Keiko Aoki]

Dear members,

I am going to conduct online survey (# of participants will be 1000 persons) and to estimate the data by mixed logit.
Although I am beginner, I am building the code for our D-efficient design with Ngene as below.

I learned several tips of coding with Ngene based on the form.
However, the D-error does not get less than 0.18 even though the evaluation is about 42089, for example.
I understand that the less D-error, the more good.
So, I tried it again and again while changing prior.
But, it does not work.

So, I have two questions.

1. How do you judge that the design is valid ?
2. What factors does D-error decrease?

(Coding)
Design
;alts = alt1, alt2, alt3, alt4
;rows = 12
;eff=(mnl, d)
;block = 2
;model:
U(alt1) =
+oy.dummy[0.00001]*oy[1,0]
+ki.dummy[0.00001]*ki[1,0]
+yu.dummy[-0.00001]*yu[1,0]
+sh.dummy[0.00001|0.00001]*sh[2,1,0]
+es.dummy[0.00001]*es[1,0]
+pl.dummy[0.00001]*pl[1,0]
+ei.dummy[0.00001|0.00001|0.00001]*ei[3,2,1,0]
+pr[-0.00001]*pr[198,298,398]/
U(alt2)
=oy.dummy*oy
+ki.dummy*ki
+yu.dummy*yu
+sh.dummy*sh
+es.dummy*es
+pl.dummy*pl
+ei.dummy*ei
+pr*pr /
U(alt3)
=oy.dummy*oy
+ki.dummy*ki
+yu.dummy*yu
+sh.dummy*sh
+es.dummy*es
+pl.dummy*pl
+ei.dummy*ei
+pr*pr
$

Thank you for your time in advance.

Best regards,
Keiko

[Michiel Bliemer]

Your syntax looks fine and your D-error of 0.18 is normal (you can compare to the D-error of the random start design and see that it can be improved significantly). Note that D-0errors will NEVER reach zero.

Aspects that decrease D-error:

* More choice tasks per respondent
* More alternatives
* Wider range for quantitative variables such as pr
* using the modified Federov algorithm to generate the design, which removes the attribute level balance constraint

You should not play with priors to improve D-errors. Priors should either be (near) zero, or should come from a pilot study. D-error is not the only important aspect of a design, you should also consider choice task complexity.

Feedback on your design:

* I think that your syntax mostly looks fine
* 4 alternatives with 8 attributes each is a very complicated choice tasks, especially if you do the survey online. You may want to reduce the number of alternatives to reduce choice task complexity, while increasing the number of choice tasks per respondent (currently 6).
* the priors determine the order of preference in the attribute levels. you use 0.00001 for each prior in ei, indicating that there is no ordering across levels 1,2,3, If there is an ordering, you can use something like 0.00001|0.00002|0.00003 to indicate the order. Using very small positive or negative coefficients that only indicate sign/ordering of attribute levels is only useful if you are checking for dominant alternatives, otherwise you may as well set them all equal to 0. Since your design is unlabelled, I would suggest using ;alts = alt1*, alt2*, etc. Dominance checks may increase your D-errors.

If it were me I would probably use the following syntax (I have assumed a certain order for sh and ei that you want to check):

Code: Select all

Design
;alts = alt1*, alt2*
;rows = 36
;eff=(mnl, d)
;block = 3
;model:
U(alt1) =
+oy.dummy[0.00001]*oy[1,0]
+ki.dummy[0.00001]*ki[1,0]
+yu.dummy[-0.00001]*yu[1,0]
+sh.dummy[0.00001|0.00002]*sh[2,1,0]
+es.dummy[0.00001]*es[1,0]
+pl.dummy[0.00001]*pl[1,0]
+ei.dummy[0.00001|0.00002|0.00003]*ei[3,2,1,0]
+pr[-0.00001]*pr[198,298,398]/
U(alt2)
=oy.dummy*oy
+ki.dummy*ki
+yu.dummy*yu
+sh.dummy*sh
+es.dummy*es
+pl.dummy*pl
+ei.dummy*ei
+pr*pr
$

Michiel

[Keiko Aoki]

Dear Professor Bliemer,

Thank you for your quick response and perfect answers.
Especially, I am really excited about your code!
I can catch the fantastic D-error like 0.06...which I have never seen it before.
I would like you to allow me writing down your acknowledgment (including choice-metrics.com?) in our manuscript if possible.

Anyway, I realize that I must study Ngene more.
Simultaneously, I would find Ngene and this forum much earlier.

By the way, I have understood a way of decreasing D error by your explanation and code.
The important point is the number of rows and block in my situation.
So I have further questions although they might be in general index.

1. Can I properly compare between the data in the with-in design which employ divided sets in one block when I analyze?
This question may not be clear. So, I more explain as below.
For example, I code "row=36 and block=2" and make 18 sets in two groups, A and B.
In the design, I would like to make two stage like S1 and S2 per person because of providing information.
I would like to conduct the within design, not between design.
So, I am wondering that I can provide one group to participants.
That means that S1 consists of 9 set which is the former sets in the group A and S2 is the later in the group A.
In this situation, I am not sure that the comparison between S1 and S2 is valid in the analysis.
That's because I understand that each participant face each group.

2. Whether one check D-error based on the estimation results after survey.
Usually, we check D-error in making design.
I think that D-error is one of checking point which the design is proper.
So, I am wondering that rechecking D-error with estimation results is useful for showing the validity of the design.

Is that correct?

I am sorry that I wrote down long.
Thank you for your time.

Best regards.
Keiko

[Michiel Bliemer]

1. You can compare responses of respondents in different scenarios using the same choice tasks or different choice tasks. If you use the same choice tasks, then differences in responses are likely due to differences in the scenario, whereas if you use different choice tasks, differences in responses may not be exclusively due to differences in the scenario. For comparisons across scenarios, giving the same choice tasks to the same respondent in stages S1 and S2 increases the statistical power of comparison. However, there may also be a risk that respondents remember seeing the same choice tasks, whether that is a problem or not depends on the study. If you show different choice tasks in S1 and S2 you can still make a comparison, but you will need a larger sample size because you will lose some statistical power.

2. You can check the D-error after estimation but most people do not do that. It may provide information about the loss of efficiency you have had. It does not provide information about validity as any finite D-error means that the data collection is valid and a model can be estimated.

Michiel

[Keiko Aoki]

Dear Professor Bliemer,

Thank you so much for you reply.
I understood your comments.
So, I am going to consider the better design with your comments and the form.

Thank you for your time

Best regards,
Keiko