Ngene crashed

[Yuanyuan Gu]

Dear Michiel and John

Many thanks for your responses so far. They have been extremely helpful.

I wanted to run a sequential design with some two way interactions. Below is my syntax:

design
;alts = alt1, alt2
;rows = 16
;orth = seq
;block=2
;model:
U(alt1) = b1*A[0,1,2,3,4,5,6,7] + b2*B[0,1,2,3] + b3*C[0,1,2,3] + b4*D[0,1,2,3,4,5,6,7] + b5*E[0,1,2,3] + b6*A*B + b7*B*C/
U(alt2) = b1*A + b2*B + b3*C + b4*D + b5*E + b6*A*B + b7*B*C$

Ngene crashed when I ran the code above.

If I delete the line ';block=2' then I can get a design but the two interaction terms are highly correlated with the main effects within the same alternatives which is not ideal.

So is blocking not allowed in this case?

I went on using the 'foldover' option. It allowed using blocking and also gave me a design whose interaction terms are uncorrelated with the main effects within the same alternatives.

Can I confirm with you that this design can be used to estimate coefficients of interaction terms? Also, can I use my own blocking and ignore 'foldover block''?

Many thanks,
Yuanyuan

[Michiel Bliemer]

Orthogonal designs are very hard to find, especially if you include multiple interaction effects and blocking. I am not sure why the blocking column would be problematic, but I think you have made a good decision by looking at fold over designs and removing the blocking column. Note that 16 choice tasks may be too little for an orthogonal design, probably 32 will give you more freedom.

Is there a reason why you want to create an orthogonal design? They are very hard to generate and are usually not very efficient. Efficient designs are much more flexible, can do blocking, and can take interactions into account more easily. Note that orthogonal designs are only optimal for linear regression model and not for logit models.

You can use your own blocking column if you want, Ngene merely tries to minimise the correlations and ensure that each level appears more or less equally within each block.

If you are estimating a logit model, I would suggest the following syntax:

Code: Select all

design
;alts = alt1, alt2
;rows = 16
;orth = (mnl,d)
;block=2
;model:
U(alt1) = b1*A[0,1,2,3,4,5,6,7] + b2*B[0,1,2,3] + b3*C[0,1,2,3] + b4*D[0,1,2,3,4,5,6,7] + b5*E[0,1,2,3] + b6*A*B + b7*B*C/
U(alt2) = b1*A + b2*B + b3*C + b4*D + b5*E + b6*A*B + b7*B*C$


Once you have obtained priors for the parameters, you can further improve the efficiency of the design.

[Yuanyuan Gu]

Hi Michiel

Many thanks for the prompt reply.

Efficient design was my first choice as there are a lot of constraints to impose (orthogonal design cannot impose constraints). However, our research team has decided to use dummy coding for all attributes and estimate the effects of several two way interactions. These two requirements have made it difficult to use efficient design because:

(1) It requires a large number of rows and thus a lot of blocking. And
(2) It takes a long time to get a design.

For example, to only interact A (8 levels) with B (4 levels) will lead to 21 interaction terms and the minimal number of rows will be 45.

The research team also want to examine other interactions including BxC, AxD, BxD, CxD and DxE which made it so hard to get an efficient design from Ngene. By constrast, orthogonal design seems to deal with interactions very easily by using the foldover option.

Michiel, is there any smart way to deal with interactions of dummy coded attributes in efficient design?

Many thanks,
Yuanyuan

[Michiel Bliemer]

I think you completely misunderstand experimental design theory. Orthogonal designs do not guarantee that interactions will work with dummy coding, they only work in linear models. Secondly, 16 rows will not be enough to satisfy the degrees of freedom in order to be able to estimate all interaction coefficients when you dummy code. You are correct that you will have many coefficients in the model if you include dummy coding in an efficient design, but you HAVE to specify exactly the same model as you will be estimating. You have now specified a linear model and therefore the design only guarantees that you can estimate a linear model with its interactions.

I strongly suggest reading into experimental design theory for stated choice surveys as otherwise you will risk collecting data and not being able to estimate models. I am happy to help if you have any further Ngene questions.

[Michiel Bliemer]

Note that there is no need to take a foldover of an edficient design, foldover designs are mainly useful for orthogonal designs. Efficient designs take all correlations into account automatically since they are based on estimating logit models. If you can find an efficient design, then you can estimate the model you have specified (including dummy coding and interaction effects). There is further no problem with having many rows, you can always block the design.

Finally, note that estimating dummy coefficients usually requires large sample sizes, especially if you have many. Interactions of dummy variables need even larger sample sizes. It is typically difficult to estimate these models and obtain statistically significant estimates.

[Yuanyuan Gu]

Many thanks Michiel. I agree that the foldover design won't allow me to estimate so many dummy coded interactions and to estimate them will also need a very large sample size. I am quite new to design and just started to understand the differences between all these designs. I learned a lot from your responses to my questions and in general the discussions in this forum.

Back to the first question I raised, i.e. using blocking will make Ngene crash in the case of orthogonal design using interactions. This problem remained when I increased the number of rows and reduced the interaction term to only one. I therefore suspect there might be a bug in the searching algorithm used by Ngene.

[Michiel Bliemer]

Thanks, I have already given your syntax to our software developer to have a look what is going wrong. There may well be a bug somewhere. I appreciate you pointing this out, and once I know more I will let you know.

[Yuanyuan Gu]

My pleasure Michiel. I have been playing with different designs using Ngene today and found the orthogonal design always led to multiple solutions, i.e., for the same code using othorgonal design, each time I pressed the 'run' button I got different designs. Is this expected?

Then I am thinking if I can run ';orth = seq' and ';eff = (mnl, d)' simultanously as shown below:

design
;alts = alt1, alt2
;rows = 16
;orth = seq
;eff = (mnl, d)
;model:
U(alt1) = b1*A[0,1,2,3,4,5,6,7] + b2*B[0,1,2,3] + b3*C[0,1,2,3]/
U(alt2) = b1*A + b2*B + b3*C$

Surprisinly I got a design but what type of design is this? Is it an orthogonal design with the smallest D error?

Many thanks!

[Michiel Bliemer]

There exist thousands of orthogonal designs, one can relabel the levels and remain orthogonal. So Ngene just creates a random orthogonal design since you did not specify which orthogonal design you prefer.

When you specify ;eff as well, then it will select the most efficient orthogonal design.