Attribute Level Balance

[pattyp12]

Hello,

Last week I posted a question on the NGENE part of this forum and Michiel gave me a quick and thorough answer. I have a follow-up question, but I thought it might be wise to post it on this part of the forum because it concerns choice designs instead of the NGENE software.

I have an unlabeled choice experiment about environmental benefits with a status quo and two different environmental project options. I have a big constraint that the levels of the environmental attributes in the status quo cannot dominate the environmental project options for the sake of plausibility.

Michiel suggested that I use a Modified Federov algorithm because I have a large design, but he mentioned that this algorithm relaxes the attribute level balance criteria. My question is: how necessary is it to have attribute level balance in a choice design? Most of the literature I have found says that attribute level balance is desirable because it ensure many observations for each attribute level, but I can't seem to find anything that examines whether it is absolutely necessary to have reliable parameter estimates.

I generated a design with the Modified Federov algorithm and noticed a pattern that the middle attribute levels seemed to be the ones under represented in the design as opposed to the more extreme upper and lower values. I would think that the upper and lower values might be more important to look at than the middle values anyway so I'm not sure if there is a problem or not.

Any advice would be greatly appreciated!

Thanks,
Pat

[johnr]

Hi Pat

Thanks for the question. You are correct that there is not much in the literature on this topic which may be a historical quirk. For orthogonal designs, balancedness implies that each attribute will be orthogonal not only to each other, but also in a linear model, to the intercept. Further, imbalance may be considered a generalised form of non-orthogonality which again for linear models means a loss of efficiency.

For non-linear models, it comes down to the assumptions you make when generating the design. If you assume zero priors and an MNL model, then the model approximates a linear model and the above will likely hold. If however you assume non-zero priors (and any model type), then the above is unlikely to hold, and imbalance is likely to be less of an issue as orthogonality becomes less important.

Where it might become an issue is if you wish to use non-linear coding such as dummy or effects coding. If a level appears infrequently over the design, then the (non-linear) design matrix may be quite sparse (i.e. the code 1 may appear very few times in a column), which may result in a singular matrix when it comes time to invert the Fisher matrix in order to calculate the AVC matrix. This problem is likely to be worse if you wish to calculate the interaction effects of non-linear coded variables as non-linear coded variables as you may end up with a column of all 0s which definitely cannot be estimated.

The point to note is that if one looks at optimal D-efficient designs for the MNL under non-zero priors, Kaninnen (2002) (and Toner and Fowkes et al. in the late 1980s early 2000s but using a different optimality criteria) found that letting the attributes be treated as continuous levels allows one to locate the optimal design. In doing so, balancedness is completely lost for that attribute (Kaninnen allows one attribute to be continuous whist Toner and Fowkes et al. let all of them be continuous). Note however that they treat continuous attributes as continuous so this cannot be done if all your attributes are qualitative. Anyway, the point is that for non-zero priors, balancedness may be an impediment to locating the optimal design.

I would suggest that the above post may be a somewhat controversial view of the world and one that reviewers may have a hard time conceding, but this is because all the training in design theory involves dealing with linear models.

John

[pattyp12]

Hi John,

Thanks for your response! I find it interesting that this is not discussed more in the literature - maybe most researchers do their best to have attribute level balance in their designs.

All of our attributes are quantitative measures and we do not plan on using any effects coding so I think we should be able to use the Modified Federov algorithm for our design.

Thanks again,
Pat

[johnr]

Hi Pat

Nothing surprises me anymore unfortunately. I really believe it is historical. Many in the research community don't realise that designs are simply the result of the assumptions we make. That orthogonal designs are outputs from assumptions of linear models. And orthogonal designs are also an output of assumptions for non-linear models - i.e., MNL model, zero local priors, d-efficiency. These are simply assumptions, but the important point, the properties of a design are outputs of these assumptions.

One day people will recognise that designs are outputs and not inputs of the process. That day I fear is not today.

John