Attribute's Levels always compared with the same levels

[michele.mazza]

Dear all

I was trying to use the below design for the pilot stage of a project where I do not have information on priors. I have noticed that despite this is a fairly simple design with little constraints, some levels of attribute B alt1 tend to appear with some other specific levels of attribute B alt2 (and vice versa). This is affecting the results I am getting while I simulate the design. This still hold even if I am doubling the number of rows or introduce a constraint to avoid dominant choices.

Is there any way to make sure that most combinations of levels of the same attribute are covered in the design (avoiding for instance that B(alt1) fourth level mostly appears against B(alt2) first level ?)?

Please see below the syntax and the output file.

Design

;alts = alt1, alt2
;rows = 24
;block = 4
;eff = (mnl,d)
;model:

U(alt1) = b1 [0.0] * A[1,2](11-13,11-13) + b2 [0.0] * B[1,2,3,4](5-7,5-7,5-7,5-7) + b3 [0.0] * C[1,2](11-13,11-13) + b4 [0.0] * D[1,2](11-13,11-13) /
U(alt2) = b1 [0.0] * A[1,2](11-13,11-13) + b2 [0.0] * B[1,2,3,4](5-7,5-7,5-7,5-7) + b3 [0.0] * C[1,2](11-13,11-13) + b4 [0.0] * D[1,2](11-13,11-13)
$

Ch alt1a alt1b alt1c alt1d alt2a alt2b alt2c alt2d Block
1 2 2 2 2 1 3 1 1 1
2 1 4 2 2 1 1 1 1 2
3 2 1 1 1 1 4 2 2 4
4 1 3 2 2 2 2 1 1 3
5 2 3 1 1 1 2 2 2 2
6 1 3 1 1 2 2 2 2 1
7 1 2 1 1 2 3 2 2 2
8 1 4 1 1 2 1 1 2 4
9 2 4 1 2 1 1 2 1 1
10 2 1 1 1 1 3 2 2 2
11 1 2 1 2 2 3 2 1 4
12 2 4 2 2 1 1 2 1 4
13 2 1 2 2 1 4 1 1 3
14 1 1 1 2 2 4 2 1 1
15 2 3 2 1 1 2 1 2 3
16 1 2 2 1 2 3 1 2 1
17 1 3 2 2 2 1 1 1 3
18 2 4 2 1 1 1 1 2 1
19 1 4 1 1 2 1 2 2 4
20 2 1 2 1 1 4 1 2 4
21 2 2 1 2 2 4 2 1 2
22 1 1 2 2 2 4 1 1 3
23 2 3 1 2 1 2 2 1 3
24 2 3 2 1 2 3 2 2 2


Thanks for your help

Michele

[Michiel Bliemer]

The output generated is correct, as it is more efficient to compare the fourth level with the first level, than to compare the fourth level with the third level (there is a larger trade-off in the former case), so this design would be more efficient and there is no problem in the fact that not all combinations appear equally.

Is there a reason why you need different attribute combinations to appear? Usually this is only important for estimating for example interaction effects. In that case, you can explicitly include these interaction effects in your utility function.

You could also use an optimal orthogonal design (OOD), see the work of Street and Burgess. They use design generators to determine the levels, and the command in Ngene is ;orth = ood (but you have to remove the ;eff command, their designs are only efficienct under zero priors). In such a design, attribute levels cycle, so for attribute B you have 1 compared to 2, 2 compared to 3, 3 compared to 4, and 4 compared to 1. Note that this is not most efficient from a statistical point of view, as smaller trade-offs are made.

[michele.mazza]

Thanks Michiel for your feedback.

I appreciate that from a statistical point of view showing (and comparing) the extreme levels results in a greater trade-off and more efficient design. However for many of our projects we would like respondents to be able to see several different combinations, as this would be closer to what they see in the real world.

Is there any way to have most attribute’s levels combination to appear at least a reasonable amount of time, within the efficient design space?

We would rather not use an optimal orthogonal design unless necessary as we also think that an efficient design is superior.

Regards,
Michele

[Michiel Bliemer]

The only way to get different combinations in an efficient design is by changing the formulation of the utility function. I think that, repeating my previous message, including interaction terms may yield different interactions within the design. You can try this. You are essentially asking to make the design less efficient, and Ngene does not contain routines to do this, so it should be induced by the model itself.

Looking at your syntax, you currently have used priors equal to zero. It is well-known that an orthogonal design in that case is actually most efficient. So unless you are going to use non-zero priors, you are best off using an orthogonal design.

[Michiel Bliemer]

If you add ;orth = seq to your syntax, Ngene will generate a design than is orthogonal AND efficient. It generates the design below. I think this may be useful for your purposes. Orthogonality guarantees that each combination of attribute levels appears an equal number of times (this is the definition of orthogonality). Clearly, non-orthogonal designs are more efficient (with non-zero priors), but then it will not include all attribute level combinations.

alt1.a alt1.b alt1.c alt1.d alt2.a alt2.b alt2.c alt2.d Block
2 1 1 2 1 2 2 1 4
2 3 2 2 1 4 1 1 2
1 2 2 1 2 3 1 2 4
1 4 1 1 2 2 1 2 1
2 1 2 1 2 3 1 1 2
2 3 1 1 1 2 1 2 2
2 2 1 1 1 3 2 1 1
2 4 2 1 1 3 1 2 3
1 2 1 2 2 4 2 1 2
1 4 2 2 2 1 2 2 3
1 1 2 2 2 4 1 1 1
1 3 1 2 2 1 2 1 3
1 3 2 1 1 1 1 1 3
1 1 1 1 2 4 2 2 4
2 4 2 2 1 1 1 1 4
2 2 1 2 1 3 2 1 1
1 4 1 2 2 3 2 2 2
1 2 2 2 2 1 1 2 1
1 3 2 1 1 2 2 2 2
1 1 1 1 1 4 2 2 4
2 4 1 1 2 2 2 1 3
2 2 2 1 1 1 2 2 1
2 3 1 2 2 2 1 1 4
2 1 2 2 1 4 1 2 3

[Michiel Bliemer]

Actually ;orth = sim will generate an orthogonal design in which all attribute level combinations appear an equal number of times, but a simultaneous orthogonal design is less useful (and less efficient) for an unlabelled experiment. So hopefully the sequential orthogonal design is sufficient for your purposes.

[michele.mazza]

Thanks Michiel, what you said is very useful.

We used to use orthogonal designs in the past but we had issues when we had to include more than 4 or 5 attributes (with several levels each).

Do you know whether using orth = seq (or = sim) would involve putting a "cap" on the number of attributes and levels we can include in an experiment?

Thanks,
Michele