Partial profile design and frequencies of level appearance

[Andrew]

Dear Ngene users,

we created a partial profile design using the instructions in the manual (section 8.10). We know that level balance is not necessary, but we found a few very unbalanced two-way frequencies in the final design. In example, one attribute A has 6 levels, another attribute B has 3 levels. When each level of attribute A is combined with each level of attribute B, there should be 15 combinations of each pair (for a perfect level balance or on average). But in our design a few level pairs appear very often, e.g. 29 times, in contrast to other pairs which appear less often, e.g. only 5 times. The combination with 5 appearances seems to be underrepresented in the design and the discrepancy to other pairs seems very high. We don't want to estimate interaction effects. But do we ran into any problem here?

The design has following specifications:
3 unlabelled alternatives
4 attributes (1 attr with 6 levels, 1 attr with 4 levels, 2 attr with 3 levels)
120 rows
10 blocks
probable sample size n=300

Appreciate any thoughts on this.

Thank you very much!
Andrew

[Michiel Bliemer]

It may seem strange, but this is an artefact of efficient designs where information is maximised. It often happens in efficient designs that certain combinations of attribute levels are over-represented, especially for models with only main effects (and no interaction effects) and where attributes are linearly coded (instead of dummy or effects coded). Certain combinations of attribute levels simply generate more information and hence are considered 'optimal' in an efficient design. Therefore, there is no problem, this is normal.

Ngene optimises for the model that you have specified, and if you do not intend to include interactions or nonlinearities then the design is fine. However, if you believe that you may want to estimate interactions or nonlinearities at a later stage, then you should specify them in the utility function and optimise the design again. A small number of attribute level combinations in the design is efficient for a model without interactions and nonlinearities, but could be problematic for a model with interactions and nonlinear effects. Therefore, in case of doubt, it is best to specify the most complex utility functions that you may consider estimating.

If for whatever reason you prefer to have attribute level combinations appear more equally, then you need to use an orthogonal design (which by definition ensures that each pair of attribute levels appears an equal number of times).

If you would like to improve attribute level balance in the design, you can impose constraints on the number of times each attribute level should appear within the design (e.g., b1*x1[1,2,3](2-5,2-5,2-5), such that each of the levels appears between 2 and 5 times within the design).

Michiel

[Andrew]

Michiel,
thank you very much. This was already very helpful.
I forgot to mention that we use effects coding and the mfederov algorithm. We were not sure if we could apply level count constraints. Instead we used the "soft level imbalance constraint" (manual p.140), e.g. ;eff = (mnl,d, mean) + 0.75*(imbalance). But in terms of two-way level appearance we did not find much improvement.

We are not expecting interaction effects or planning estimating them. But, as you wrote, in case of doubt we would like to be prepared. Would you please take a look at the design below? The imbalance constraint is left out here. First we generated a regular factorial design with 15000 rows, then 5400 rows were randomly chosen for the candidate set and overlaps manually added. We ran the design with priors and stopped the evaluation process after 10 hours and got 3 designs.

Code: Select all

Design
;alts = alt1*, alt2*, alt3*
;rows = 180
;eff = (mnl,d,mean)
;bdraws= halton(6000)
;block = 15
;alg = mfederov(candidates = D:\designs\design project files\candset.csv)
;model:

U(alt1) =
b1.effects[(n,0.30,0.05)|(n,0.10,0.05)|(n,-0.40,0.05)|(n,0.30,0.05)|(n,0.10,0.05)] * att1[1,2,3,4,5,6] +
b2.effects[(n,0.20,0.05)|(n,0.10,0.05)] * att2[1,2,3] +
b3.effects[(n,0.40,0.05)|(n,0.10,0.05)|(n,-0.10,0.05)] * att3[1,2,3,4] +
b4.effects[(n,0.20,0.05)|(n,0.10,0.05)] * att4[1,2,3]/

U(alt2) =
b1 * att1 +
b2 * att2 +
b3 * att3 +
b4 * att4 /

U(alt3) =
b1 * att1 +
b2 * att2 +
b3 * att3 +
b4 * att4
$



Thank you very much again.
Andrew

[Michiel Bliemer]

If you are using effects coding then I do not think there can be any issues (and most people do not use interactions for effects coded variables). I do not see anything wrong with the syntax, should be fine I think.
I believe that the mfederov algorithm only reports a new design after cycling through all candidates, so while it generates only 3 designs, it will likely have evaluated a lot more and only reported the best designs after all possible swaps within the candidate set with all rows in the design.

Note that if the D-error is finite, the model parameters are identifiable and hence you can estimate the model with the data. If there was an issue with the levels that would prevent you from estimating the model, then the D-error would be infinite. Since you are using effects coding, this for example means that all attribute levels appear in the design at least once, as otherwise it would not be possible to estimate the model.

Michiel

[Andrew]

Michiel,
this was of great help and makes Ngene a bit more understandable again - thank you very much.
Kind regards,
Andrew

[Andrew]

Dear Michiel,
I have a follow-up question. I have desperately tried to find a design which results in a nearly perfect 1-way- and 2-way level balance. In the above posted syntax I changed the number of rows, magnitude of priors and the algorithm as well. I get designs which are perfectly 1-way balanced for main effecsts. However, eventually there are always level combinations which would be underrepresented. Due to the use of effects coding it would be cumbersome, I guess, to specify all possible interaction effects in the syntax.

Is there any solution or workaround to get a design which has a nearly perfect 1-way and 2-way level balance?

Thank you very much!
Andrew

[Michiel Bliemer]

I assume that you mean orthogonality of strength 1 ( = attribute level balance) and orthogonality of strength 2 ( = balancedness of pairs of attribute levels)? Such orthogonality can only be obtained through orthogonal arrays, which are very restrictive.

I do not see a way how to impose constraints on pairs of attribute levels, this is not something that is commonly done.

The only way I believe would be to include all the interactions, e.g.
i1[0]*x1.dummy[1]*x2.dummy[1] + i2[0]*x1.dummy[1]*x2.dummy[2] + ... etc

Michiel

[Andrew]

Yes, attribute level balance and balancedness of pairs of attribute levels is exactly what I meant. We try to achieve near-perfect balance in 1-way and 2-way frequencies in a design for special circumstances where we can only apply counting analysis (surely not the most superior solution) using Excel for estimation of individual preferences. Therefore we would like to make sure that all attributes and pairs of attribute levels appear nearly equally often in the final design. I really appreciate your suggestion and gonna add all possible interactions in the design syntax.

Michiel, thank you very much again for the great support.

Kind regards,
Andrew