Design with two-way interactions

[Andrew]

Hello,

we are trying to generate a design with a specific number of two-way interactions.

The design has
- 8 attributes using effects coding (1 attr with 2 levels, 7 attr with 3 levels)
- 3 alternatives
- 72 rows blocked in 6 versions (that is 12 tasks per respondent).

Attribute 1 is supposed to be categorial (label 1, label 2).
The design should allow us to estimate interactions between the two level of attribute 1 and all levels of the remaining 7 attributes.

So far the design is specified as:

Code: Select all

Design                   
;alts = alt1, alt2, alt3                   
;rows = 72
;block = 6                           
;eff = (mnl,d,mean)
;model:                   
U(alt1) =
b1.effects * attr1 [0,1]   +                   
b2.effects[(n,0.8,0.1)|(n,0.01,0.02)] * attr2 [1,6,12] +                   
b3.effects[(n,0.1,0.1)|(n,0.01,0.02)] * attr3[0,1,2] +           
b4.effects[(n,0.6,0.1)|(n,0.01,0.02)] * attr4[0,1,2] +
b5.effects[(n,0.2,0.1)|(n,0.01,0.02)] * attr5 [0,1,2] +           
b6.effects[(n,0.6,0.1)|(n,0.01,0.02)] * attr6 [0,1,2] +           
b7.effects[(n,0.4,0.1)|(n,0.01,0.02)] * attr7[0,3,9] +   
b8.effects[(n,0.5,0.1)|(n,0.01,0.02)] * attr8[2,4,6] +
i1 * attr1.effects[0] * attr2+
i2 * attr1.effects[0] * attr3 +
i3 * attr1.effects[0] * attr4 +
i4 * attr1.effects[0] * attr5 +
i5 * attr1.effects[0] * attr6 +
i6 * attr1.effects[0] * attr7 +
i7 * attr1.effects[0] * attr8 /                     

U(alt2) =
b1.effects * attr1 +
b2.effects * attr2 +                   
b3.effects * attr3 +           
b4.effects * attr4 +
b5.effects * attr5 +           
b6.effects * attr6 +           
b7.effects * attr7 +   
b8.effects * attr8 +
i1 * attr1.effects[0] * attr2+
i2 * attr1.effects[0] * attr3 +
i3 * attr1.effects[0] * attr4 +
i4 * attr1.effects[0] * attr5 +
i5 * attr1.effects[0] * attr6 +
i6 * attr1.effects[0] * attr7 +
i7 * attr1.effects[0] * attr8 /                                                 

U(alt3) =
b1.effects * attr1 +
b2.effects * attr2 +                   
b3.effects * attr3 +           
b4.effects * attr4 +
b5.effects * attr5 +           
b6.effects * attr6 +           
b7.effects * attr7 +   
b8.effects * attr8 +
i1 * attr1.effects[0] * attr2+
i2 * attr1.effects[0] * attr3 +
i3 * attr1.effects[0] * attr4 +
i4 * attr1.effects[0] * attr5 +
i5 * attr1.effects[0] * attr6 +
i6 * attr1.effects[0] * attr7 +
i7 * attr1.effects[0] * attr8   $


We explicitly specified interactions between level 1 of attr1 and each remaining attributes.
Are the interaction terms specified correctly in the design? Will the given design allow us to estimate each interaction effect between level 2 of attr1 and the other attributes? Or would we have to specify each level of attribute 1 with each single level of the other attributes
Prior to this, we already specified additional 7 interactions (i8 to i14) using level 2 of attr1 and remaning attributes but this already seemed to be too complex and NGENE did not come up with a design.

Many thanks.

Best,
Andrew

[Michiel Bliemer]

1. Yes the interaction terms are properly specified.

2. I tried and was able to add the interaction effects between level 2 of attr1 and the other attributes to 2 alternatives, this works fine. When you add it to the third, it seems that your model is no longer properly specified, there may be some perfect correlations or other cancellations, not sure. You can try to generate some data in Excel and see if you can estimate the model.

3. Note that you are doing only 200 Halton draws for 14 Bayesian coefficients. This will just give you a quite random design since it does not cover the prior distribution space at all. You need a minimum of 2^14 = 16,384 Gaussian Bayesian draws to make any sense (by setting ;bdraws = gauss(2)), but preferably you want 3 Gaussian draws (;bdraws = gauss(3)), although this will result in 4,782,969 draws per design. Usually it is best to not specify more than 10 Bayesian coefficients by simply setting some to a fixed prior.

[Andrew]

Many thanks, Michiel.

I have a few follow up questions.

To 2. You wrote, the model were no longer properly specified and there may be some correlations when you add to the third, what do you mean by that? Does this effect vanish when we increase the draws? In the meantime we simulated some data for aggregate logit in Excel and were able to estimate the model including interaction effects.

To 3. Why would we need 3 Gaussian draws for our design? Setting ;bdraw= gauss(2) works fine. After using 3 Gaussian draws Ngene came up with a warning (Something went unexpectedly wrong.), aborted the whole process and eventually crashed. Would we do better with less Bayesian coefficients or by setting some to a fixed prior?

Thanks again.

Andrew

[Michiel Bliemer]

I was referring to checking whether all the estimaties in your model can actually be estimated, as if there are perfect correlations in the dataset then one or more coefficients cannot be estimated. If you are able to estimate in Excel, I believe it should work, not sure why it does not. I will have to ask our software engineer to have a look at what is happening with the syntax.

Of course gauss(2) will work, gauss(1) will also work, but it will not cover the distribution space well. Think about how many draws you would need to cover a one-dimensional normal distribution. Do you think 2 draws from this distribution covers it sufficiently? I would usually prefer at least 5, but this becomes problematic in high dimensions. So yes gauss(2) works, but it covers the very bare minimum of the 14-dimensional distribution space. gauss(3) is already much better, but comes at a very high computational cost. That is why it is typically not recommended to go above 10 random distributions.

[Michiel Bliemer]

Can you please confirm that the syntax below is the one that you would like to run, and that this is the model that you successfully estimated?

Code: Select all

    Design                   
    ;alts = alt1, alt2, alt3                   
    ;rows = 72
    ;eff = (mnl,d,mean)
    ;model:                   
    U(alt1) =
    b1.effects * attr1[0,1]   +                   
    b2.effects[(n,0.8,0.1)|(n,0.01,0.02)] * attr2[1,6,12] +                   
    b3.effects[(n,0.1,0.1)|(n,0.01,0.02)] * attr3[0,1,2] +           
    b4.effects[(n,0.6,0.1)|(n,0.01,0.02)] * attr4[0,1,2] +
    b5.effects[(n,0.2,0.1)|(n,0.01,0.02)] * attr5 [0,1,2] +           
    b6.effects[(n,0.6,0.1)|(n,0.01,0.02)] * attr6 [0,1,2] +           
    b7.effects[(n,0.4,0.1)|(n,0.01,0.02)] * attr7[0,3,9] +   
    b8.effects[(n,0.5,0.1)|(n,0.01,0.02)] * attr8[2,4,6] +
    i1 * attr1.effects[0] * attr2+
    i2 * attr1.effects[0] * attr3 +
    i3 * attr1.effects[0] * attr4 +
    i4 * attr1.effects[0] * attr5 +
    i5 * attr1.effects[0] * attr6 +
    i6 * attr1.effects[0] * attr7 +
    i7 * attr1.effects[0] * attr8 +
    i8 * attr1.effects[1] * attr2+
    i9 * attr1.effects[1] * attr3 +
    i10 * attr1.effects[1] * attr4 +
    i11 * attr1.effects[1] * attr5 +
    i12 * attr1.effects[1] * attr6 +
    i13 * attr1.effects[1] * attr7 +
    i14 * attr1.effects[1] * attr8

/                     

    U(alt2) =
    b1.effects * attr1 +
    b2.effects * attr2 +                   
    b3.effects * attr3 +           
    b4.effects * attr4 +
    b5.effects * attr5 +           
    b6.effects * attr6 +           
    b7.effects * attr7 +   
    b8.effects * attr8 +
    i1 * attr1.effects[0] * attr2+
    i2 * attr1.effects[0] * attr3 +
    i3 * attr1.effects[0] * attr4 +
    i4 * attr1.effects[0] * attr5 +
    i5 * attr1.effects[0] * attr6 +
    i6 * attr1.effects[0] * attr7 +
    i7 * attr1.effects[0] * attr8 +
    i8 * attr1.effects[1] * attr2+
    i9 * attr1.effects[1] * attr3 +
    i10 * attr1.effects[1] * attr4 +
    i11 * attr1.effects[1] * attr5 +
    i12 * attr1.effects[1] * attr6 +
    i13 * attr1.effects[1] * attr7 +
    i14 * attr1.effects[1] * attr8  /

    U(alt3) =
    b1.effects * attr1 +
    b2.effects * attr2 +                   
    b3.effects * attr3 +           
    b4.effects * attr4 +
    b5.effects * attr5 +           
    b6.effects * attr6 +           
    b7.effects * attr7 +   
    b8.effects * attr8 +
        i1 * attr1.effects[0] * attr2+
    i2 * attr1.effects[0] * attr3 +
    i3 * attr1.effects[0] * attr4 +
    i4 * attr1.effects[0] * attr5 +
    i5 * attr1.effects[0] * attr6 +
    i6 * attr1.effects[0] * attr7 +
    i7 * attr1.effects[0] * attr8 +
    i8 * attr1.effects[1] * attr2+
    i9 * attr1.effects[1] * attr3 +
    i10 * attr1.effects[1] * attr4 +
    i11 * attr1.effects[1] * attr5 +
    i12 * attr1.effects[1] * attr6 +
    i13 * attr1.effects[1] * attr7 +
    i14 * attr1.effects[1] * attr8
$


I have done a few more experiments, and looked at the Fisher information matrix and the asymptotic variance-covariance (AVC) matrix. Since you have 29 (!) coefficients that you would like to estimate, these matrices are very large, and the numbers in the AVC matrix are very large, suggesting that this model is ill-defined and that you will need extremely large sample sizes in order to be able to estimate this model. Can you confirm that you were able to estimate the parameters of this model using simulated data in which all parameters are statistically significant?

Michiel