Getting 0% OOD Optimality for my design script

[neeraj85]

Hi,

I have recently begun using Ngene (licensed version) for my experimental design. I have been trying to build an OOD design for my scenario.

Following is the script:

Design
? This will generate an orthogonal optimal in difference fractional factorial design (OOD)
;alts = alt1, alt2
;rows = 75
;orth = ood
? ;block = 5
;model:
U(alt1) = b1 + b2 * tt[0.8, 1.0, 1.2, 1.4, 1.6] + b3 * tts[0.3, 0.4, 0.5, 0.6, 0.7] + b4 * sn[0.75, 1.0, 1.25, 1.50, 1.75] + b5 * vr[0.75, 0.875, 1.0, 1.125, 1.25] /
U(alt2) = b2 * tt + b3 * tts + b4 * sn + b5 * vr $


I am getting a 0% OOD-Optimality value for this design which I am unable to figure out.
I even tried changing number of rows to: 25, 50, 75, 100, 125 and 150 beyond which it started giving an error “OOD Design doesn’t exist”.


Also, I tried running the same script with 4 levels for each attribute. Its working fine and I am getting an OOD-Optimality value of 94.49%


Can you please let me know where the issue is.

Thanks
Neeraj

[johnr]

Hi Neeraj

I think the problem lies with the number of attribute levels you have. The program is able to locate an OOD if you have 4 levels.

For example, this works.

Design
? This will generate an orthogonal optimal in difference fractional factorial design (OOD)
;alts = alt1, alt2
;rows = 48
;orth = ood
? ;block = 5
;model:
U(alt1) = b1 + b2 * tt[0, 1, 2, 3] + b3 * tts[0, 1, 2, 3] + b4 * sn[0, 1, 2, 3] + b5 * vr[0, 1, 2, 3] /
U(alt2) = b2 * tt + b3 * tts + b4 * sn + b5 * vr $

We will need to explore if this in an internal programming issue, or whether indeed the result is correct in this case with 5 levels.

John

[Michiel Bliemer]

I also had a quick look, and I think a suitable OOD for these design dimensions just may not exist. OOD is quite restrictive, so good designs only exist in special cases.

To confirm, I tried to locate an OOD using a different syntax and a different algorithm:

Design
;alts = alt1, alt2
;rows = 75
;eff = (mnl,d)
;orth = seq
;model:
U(alt1) = b1 + b2.orthn * tt[0.8, 1.0, 1.2, 1.4, 1.6] + b3.orthn * tts[0.3, 0.4, 0.5, 0.6, 0.7] + b4.orthn * sn[0.75, 1.0, 1.25, 1.50, 1.75] + b5.orthn * vr[0.75, 0.875, 1.0, 1.125, 1.25] /
U(alt2) = b2 * tt + b3 * tts + b4 * sn + b5 * vr $

So I am searching for a D-optimal design using a sequential orthogonal array and orthonormal coding, which should lead to an OOD design as well. This also returns a D-optimality of 0% (although the D-error clearly exists). So I suspect no suitable OOD exists for these design dimensions.

[neeraj85]

Thanks a lot for your reply.

IDEA: I thought of conducting a pilot survey by generating the OOD design. The prior parameter values thus obtained would then be fed as an input to build a D-Efficient design.

Given that OOD designs don't exist for 5 levels per attribute, will it be okay to roll out my pilot survey based on the sequential orthogonal design (;orth = seq)?


Thanks
Neeraj

[Michiel Bliemer]

Definitely.
You could even generate a D-efficient design using zero priors, which would perhaps be an even better design for your pilot study.
If you know the signs of the coefficients, you could even put in very small priors (like 0.000001 and -0.00001) and use ;alts = alt1*,alt2*, such that Ngene can automatically get rid of strictly dominant alternatives.
Also, consider removing the constant (b1) in case your utility functions are unlabelled/generic, as then the constant does not have a meaning.

[johnr]

Hi Neeraj

OODs are optimised under the assumption of an unlabelled experiment, there is no no choice alternative, that an MNL model be used, with locally optimal zero priors, and further that the attributes will be orthonormally coded. Experience has suggested that these designs may also be optimal in many cases under different coding schemes keeping the remaining assumptions constant, however this appears to be a case by case finding. You can use the sequential orthogonal design approach and locate an efficient (though it might not be optimal - you can only know if you test all designs, the beauty of OODs is that Street and Burgess mathematically were able to prove what the optimal D-error would be under the above assumptions) to do the same thing. It may not be as good as an OOD in terms of optimality, however hopefully the assumptions that are used in generating the design won't hold in reality anyway. That is, you are assuming the parameters are zero in generating the design, however in the pilot you would hope that the parameters are not zero. So optimality in this case is not the be all and end all.

I would suggest you go with the approach you suggest.

John

[neeraj85]

Hi,

I would like to add that I am designing an Unlabelled Experiment.

1. I tried generating a sequential OOD design using the following script:
Design
;alts = alt1, alt2
;rows = 75
;orth = seq
;block = 5
;model:
U(alt1) = b1 + b2.orthn * tt[0.8, 1.0, 1.2, 1.4, 1.6] + b3.orthn * tts[0.3, 0.4, 0.5, 0.6, 0.7] + b4.orthn * sn[0.75, 1.0, 1.25, 1.50, 1.75] + b5.orthn * vr[0.75, 0.875, 1.0, 1.125, 1.25] /
U(alt2) = b2 * tt + b3 * tts + b4 * sn + b5 * vr $

Following are my observations, after executing the script once:
1. OOD optimality is still ZERO
2. Out of the 75 tasks generated, around 20 of them are dominant choice tasks.
3. Most of the values in the Pearson Product Matrix are less than 0.2

The results from this approach doesn't seem encouraging. Should I go ahead in trying out multiple runs of this script?


2. Alternately, I am also trying the approach mentioned by Prof. Bliemer. Following is the script for the same:

Design
;alts = alt1, alt2
;rows = 75
;eff = (mnl,d)
;block = 5
;model:
U(alt1) = b2[-0.001] * tt[0.8, 1.0, 1.2, 1.4, 1.6] + b3[-0.002] * tts[0.3, 0.4, 0.5, 0.6, 0.7] + b4[-0.003] * sn[0.75, 1.0, 1.25, 1.50, 1.75] + b5[-0.004] * vr[0.75, 0.875, 1.0, 1.125, 1.25] /
U(alt2) = b2 * tt + b3 * tts + b4 * sn + b5 * vr $

The execution is currently on-going and I have reached D-error ~ 0.27

Is the process right? Also, should I wait until the D-error drops below 0.1??


3. There's one more thing that I wish to ask now. I will like to fit a Panel mixed logit model on the data obtained from the full scale survey. So is it okay to estimate the prior values using MNL model here in this script?


Thanks
Neeraj

[Michiel Bliemer]

1. Yes that is exactly what I got from my run, you will still get a 0 D-optimality (see my earlier message) and it does not avoid dominant alternatives. I would not suggest this syntax.

2. This syntax does not remove dominant alternatives. The following syntax will, where I have added * to the alternative names:

Design
;alts = alt1*, alt2*
;rows = 30
;eff = (mnl,d)
;block = 2
;model:
U(alt1) = b2[-0.001] * tt[0.8, 1.0, 1.2, 1.4, 1.6] + b3[-0.002] * tts[0.3, 0.4, 0.5, 0.6, 0.7] + b4[-0.003] * sn[0.75, 1.0, 1.25, 1.50, 1.75] + b5[-0.004] * vr[0.75, 0.875, 1.0, 1.125, 1.25] /
U(alt2) = b2 * tt + b3 * tts + b4 * sn + b5 * vr $

Note that it is very difficult to find a design with 75 rows without any dominant alternative. So I have set rows to 30 and block to 2. You can also use rows = 15 and no blocking.
The D-error has not much meaning really (especially with priors that are not set to real values), just the lower the better. If the D-error does not change much anymore at some point, you can stop.
For a pilot study this design will be fine. After collecting data in the pilot study, you estimate coefficients for the MNL model and if possible also for the panel mixed logit model (but you may not have enough data). Then in the next stage you use the priors from the MNL model to optimise the design, while you use the coefficients for the panel mixed logit model to evaluate the design, see several discussions here on the forum.

[neeraj85]

Thanks a lot Professor.

Regarding the number of rows, I am a bit curious to know that why several previous works opted for more than, say 40 rows and blocking them into smaller chunks?
Is it because they lead to a more efficient design?

Or is it a trade off between fewer dominant choice tasks (using rows=15; block=0) & more efficient design (using rows=75; block=5)?


I'll be grateful if you can clarify this query of mine.


Thanks
Neeraj

[Michiel Bliemer]

People using orthogonal designs have to use many more rows, because orthogonal designs tend to only exist with many rows. Because there are too many rows, they have to block them.

Efficient designs do not need a lot of rows, so you can simply ask for less rows. The number of rows needed depends on the number of coefficients that you want to estimate. In your case, you need a minimum of 5 rows. So 15 is more than enough. But some people feel more comfortable having a bit more variation in their data, so they opt for more rows and block the design. But increasing the design also means that you will include less efficient choice tasks, so the efficiency per choice task will actually go down. Variation in the data is only needed when you think you will estimate a lot more parameters later on, such as all kinds of interaction effects, or estimate nonlinear effects by dummy coding the coefficients etc. In that case it is best to immediately specify all interactions and dummy coded variables in the utility function and find an efficient design for this utility function.