incrementally move to full RP-panel design

[k24lewis]

I am trying to generate a RP-panel efficient design using priors. However, it has been running for four days and it still hasn't found a design. You responded the following to help:

"I would suggest moving incrementally to a full RP-panel design. Add one random parameter at a time, without Bayesian priors, and perhaps use the cross sectional mixed logit model initially. As you incrementally move to the full specification, check that the choice probabilities are reasonable. There are likely some outlying values drawn from either the random parameter or Bayesian prior distributions that are leading the singular
Fisher information matrices.
Even with good priors, the design will take a very long time to run, as there are three layers of integration (random parameters, Bayesian priors, and the random parameters panel)."

I am now wondering how you incrementally move to a full RP-panel design?

This is my current design:
?A 4 levels Price
?B a; None
?C b; None
?D c; None
?E d; None
?i1 B*C
?i2 B*D
?i3 B*E
Design
;alts = alt1, alt2, alt3
;rows = 36
;block = 3
;eff = (rppanel,d, median)
;rep = 350
;rdraws = halton(200)
;model:
U(alt1) = b1[-.48912]*A[5.99,7.99,9.99,11.99] + b2[n,(n,.50189,.55349),(u, 0.29292,1.66692)]*B[0,1]+b3[n,(n,.16082,.36702),(u, 0.21337,1.17877)]*C[0,1]+ b4[n,(n,.42309,.34874),(u,0.35747,1.32161)]*D[0,1] + b5[n,(n,.61217,.33508),(u,0,0.95723)]*E[0,1] + i1[1.19208]*B*C + i2[0.29299]*B*D +i3[-0.59302]*B*E /
U(alt2) = b1*A + b2*B + b3*C + b4*D + b5*E + i1*B*C + i2*B*D + i3*B*E$



Thanks!
Karen

[Michiel Bliemer]

It is always tricky doing an RPPANEL design and we typically advise people against this, since optimising for panel random parameter models is in many cases almost impossible due to the high dimensions. You are doing a Bayesian RPPANEL design, which is even harder.

You have 4 random parameters (for which you do 200 draws), you have 8 Bayesian priors (for which you will need at least 1000 draws or so), and you need to create a sample (you specified 300 repetitions). This means that for each design evaluation, you will need to do 200*1000*350 = 70 mln calculations. It will take very long for Ngene for evaluate even a single design.

That you do not get any output after 4 days is likely due to a problem with your priors. I think your main problem is that your Bayesian priors are very wide, since for the standard deviations you have a very high limit of the uniform distribution. This means that it will draw your normally distributed random parameters from a very wide distribution, which easily creates dominant attributes and hence little information in the design.

I would suggest to find a Bayesian efficient design for the MNL model, and evaluate under RPPANEL.

So do something like this:
;eff = my_mnl_model(mnl,d)
;model(my_mnl_model):
U(alt1) = b1[(n,..,..)]*A + b2[(n,..,..)]*B + ...
...
;model(my_rppanel_model):
U(alt1) = b1[n,..,..]*A + b2[n,..,..]*B + ...

This way, Ngene will only optimise for the first (MNL) model, but once you click on a design, you can evaluate its efficiency under the second (RPPANEL) model.

[k24lewis]

Thanks! I decided to get rid of the Bayesian restriction and it is running much quicker now. I now have another question. I have made some adjustments to the previous code and now get a decent design with a D-error, etc. However, for D-optimality it returns 0%. Since I'm doing an efficient rppanel design, does it matter if I get 0% D optimality for OOD since I'm not using OOD?

Also, occasionally a design will generate where alternative 1 and alternative 2 are the same. How do I prevent this from happening?

Thanks!

[Michiel Bliemer]

You can ignore the D optimality percentage, it is not relevant here.

In order to remove identical alternatives, dominant alternatives, and repeated choice tasks you need to tell Ngene that your alternatives are unlabelled/generic. You do that by putting an asterisk behind the alternative names, like this:

;alts =alt*, alt2*

If Ngene has trouble finding a design you may need to decrease the number of rows (which i think would be a good idea anyway, 36 is a very large design and you now assume in your panel model that one respondent faces all 36 questions; this means it is multiplying 36 probabilities, which can lead to numerical problems). Also, you can use the mfederov algorithm instead of the default algorithm.

[k24lewis]

Thanks! I have reduced my design to 2 blocks and 12 choices each. I have also added the asterisks after the alternatives. How can I make the design assume each respondent only sees 12 choices (1 block)? We only want respondents to see 12 choice sets each.
By the way, I'm now working with the following for my design:

Design
;alts = alt1*, alt2*, alt3
;rows = 24
;block = 2
;eff = (rppanel,d)
;rep = 500
;rdraws = halton(200)

;model:
U(alt1) = b1[-.4]*A[5.99,6.99, 7.99,8.99, 9.99,10.99] + b2[n,0.5,.3]*B[0, 1, 2]+b3[n,0.3,.3]*C[0,1]+ b4[n,.45,.3]*D[0,1] + i1[.15]*B*C + i2[.15]*B*D /
U(alt2) = b1*A + b2*B + b3*C + b4*D + i1*B*C + i2*B*D $

Thank you so much for your help!

[Michiel Bliemer]

Ngene creates blocking columns AFTER the design has been created. This holds for all designs. So when calculating the D-error it assumes that a single respondent faces 24 choice tasks. The fact that you give only 12 choice tasks to a single respondent does not matter for the MNL model, but it is slightly inconsistent for the RPPANEL. This is not a big issue and I would be happy to split the design in 12 using the blocking column that Ngene generates.

There are still several issues with your model and design.

1) alt3 is dominant. The choice probabilities for alt1 and alt2 are near zero. You need to add a constant to your model, since alt3 is a labelled alternative (namely the no-choice I assume)

2) Your design is heavily constraint since you are asking for 24 rows without dominant alternatives, while you have only very few levels per attribute. It is still possible to find a design, but optimising for RPPANEL will take too long. It takes a few thousand design evaluations for Ngene to find a feasible design, this would take a very long time with RPPANEL. Further, for the default swapping algorithm it is quite difficult to find good designs that are so heavily constraint. Therefore, I suggest using the syntax like the one below. It now optimises for the MNL model, but when you open a design, it will also report the D-error for the RPPANEL model. You will see that this value is likely much better than you will be able to find by running Ngene for days trying to find an efficient RPPANEL design directly.

Design
;alts = alt1*, alt2*, alt3
;rows = 24
;block = 2
;eff = (mnl,d)
;rep = 500
;rdraws = gauss(3)
;alg = mfederov
;model:
U(alt1) = b0[2.5] + b1[-.4]*A[5.99,6.99, 7.99,8.99, 9.99,10.99] + b2[n,0.5,.3]*B[0, 1, 2]+b3[n,0.3,.3]*C[0,1]+ b4[n,.45,.3]*D[0,1] + i1[.15]*B*C + i2[.15]*B*D /
U(alt2) = b0 + b1*A + b2*B + b3*C + b4*D + i1*B*C + i2*B*D $

Good luck.