RP d-error and no. of rdraws

[miq]

The D-error obviously depends on the model and hence its value is incomparable between morels (its relative).
However, for the same model - does it also depend on the no. of random draws I specify, when I look for efficient rp design?
I've noticed that for more random draws the starting D-errors and D-errors of subsequently generated designs are quite different. Usually the more draws - the higher D-errors, but if the no. of draws is too small D-errors raise again. Can they be expected to eventually converge, no matter what the no. of random draws?

Thank you in advance for explaining.

[Michiel Bliemer]

The number of random draws determines the accuracy of the D-error for the mixed logit model. The more random draws, the more accurate this D-error will be. There is no rule that less draws will yield higher D-errors or lower D-errors, they can go either way. As always, the more draws, the more accurate, although this comes at the cost of higher computation times. When using too little draws, Ngene may find a design that SEEMS efficient, but when evaluated with a higher number of draws, actually is NOT. The minimum number of draws to be used depends on the number of random parameters (the more parameters, the more draws needed), and on the assumed standard deviations in the random parameters (the higher the standard deviations, the more draws needed).

[miq]

Thank you for explaining this Michiel.

In this case - could you help me figure out what's going wrong with this design?

Design
; alts = NH, H
; rows = 20
; block = 5
; rdraws = halton(*)
; eff=(rp,d)
; con
; model:
U(NH) = basc[n,1,4]*asc[1] /
U(H) = bpat[n,-0.6,2.4]*pat[0,1] + bpri[n,1,4]*pri[1,2,4,6] + bpen[n,-0.2,0.8]*pen[5,10,15,20]
$

It's supposed to be a design for a pilot hence a very uninformative priors.

When I evaluate this model at a small no of random draws (i.e. 100) it produces D-errors of about 50. When I increase the number of draws D-errors quickly raise: for 500 draws it's about 100, for 1000 draws it's about 150, for 2000 draws it's about 200.

If I increase the number of draws further, say to 5000 or 10000, almost all designs produced are invalid, and some early ones have D-error of about 400.

Almost all designs being invalid indicates a problem with the model and I can't figure out what it is. Utilities/probabilities for the designs witch do get generated look good. Increasing no. of random draws was not a problem in other models (perhaps ones where there were more alternatives and attributes in every one).

Any help would be appreciated, thank you in advance.

[Michiel Bliemer]

The main thing that will go wrong in this design is, that your standard deviations for the normally distributed parameters in the RP model are very high. The random draws for these parameters will therefore be all over the place, making it difficult to evaluate the design. You mention that you use very uninformative priors, but you need to be aware of the difference between a random parameters model, and Bayesian priors. You are using a (cross-sectional) RP model with fixed priors, not Bayesian. So essentially you assume very informative priors, namely the exact distribution of the normally distributed parameters. If you would like to use uninformative priors, you will have to use Bayesian priors.

Furthermore, I am not sure you really want to use the cross-sectional RP model, as you probably giving multiple choice tasks to a respondent, hence there will be correlations. Then the panel RP model (rppanel) is more approprioate. Unfortunately, this model type takes very long to compute. Therefore, I would recommend for the pilot study to use the MNL model type with Bayesian priors, and then once some priors have been obtained, generate a new design with these priors.

I cannot explain why more draws would yield higher D-errors, that should not really happen. We will look into it.

[miq]

All is clear. After thinking about it I can see why using Bayesian priors for MNL can produce different designs than RPL with fixed priors.

Thank you!