Help with Ngene BWS Case 2 Profile Experiment Design

[Dash76]

Dear Sir or madam,

I am planning a Best-Worst Scaling (BWS) profile (case2) experiment with ten attributes: five with five levels and five with four levels. I need assistance with two issues please:

Is there a way to create a design in Ngene with only single profiles? The program seems to require specifying multiple alternatives, despite needing a single-profile design.

I attempted to create an OMEP design with the above attributes and levels in Ngene, but received an error stating no design is available. My syntax works with fewer attribute levels, leading me to wonder if the issue lies with the quantity and variation of levels.

Any help would be greatly appreciated. Thank you.

[Michiel Bliemer]

Ngene is developed to generate designs for multi-profile choice experiments, it is not designed for BWS but maybe it can be used for that.

What Ngene does allow is an opt-out, so you could do:

design
;alts = alt1, optout
...
;model:
U(alt1) = ...
$

In other words, you do not need to specify the optout alternative, only the alt1 alternative.

Orthogonal designs mainly exist for attributes with only 2 or only 3 levels, but there exist some mixed array designs with combinations of different levels. An orthogonal array of 5^4 4^4 does not exist, so you would need to try to change the number of levels to find one where the orthogonal array does exist. So yes, the issue is the number of attributes (10) as well as the number of levels (4 and 5). An orthogonal design exists for 4^10, so that would mean making all attributes 4 levels. Also, an orthogonal design exists for 5^10, so that would mean making all attributes 5 levels. The mix of 4 and 5 levels is the main issue.

Michiel

[Dash76]

Dear Sir, thank you ever so much for this. That's extremely helpful.

[Dash76]

Dear Professor,

I'm using Ngene to design a best-worst scaling case 2 experiment and am employing the workaround you suggested (which works very effectively - thank you). My goal is to elicit weights for 10 attributes: half with four levels, and the other half with five levels. My primary objective is precise parameter estimation, while my secondary goal is exploring and evaluating heterogeneity.

I intend to produce a design of approximately 400 profiles divided into 20 blocks, each containing 20 profiles, for a sample size of 1,200 respondents. While Ngene ensures overall level balance and near-zero between-attribute correlations for the total design, I'm encountering some issues at the block level:

1. Within-Block Level Balance: Ngene creates designs with excellent overall level balance. However, there's an imbalance within each block. Does Ngene aim to achieve as much 'within block level balance' as possible, or does it just try to achieve this across the entire survey? I selected 20-task blocks believing this would aid in maximising ‘within block level balance’ given the attribute levels, but I'm reconsidering if this doesn't assist with that goal (as 20 tasks is a lot for one person to complete).

2. Between-Attribute Correlations: Similarly, despite Ngene achieving near-zero correlations across the design, individual blocks exhibit larger correlations, sometimes beyond the [-0.5 to 0.5] range. Does Ngene try to minimize these correlations within blocks?

In relation to points 2 and 3 above, is there a strategy to enhance within-block level balance and reduce attribute correlations? For instance, would a 200 choice set survey, folded over to create 400 choice sets, be beneficial?

Finally I wonder if I could ask about avoiding 'Easy Choices' please. I aim to exclude profiles with either: a) a top-level attribute without any other top or second-top attribute, or b) the lowest level attribute without any other lowest or second-lowest attribute. Is it realistsic for me to expect to find such a design? And would the most effective approach simply be to extensively run Ngene and then sift through generated designs until I find one devoid of such choices?

Thank you for your guidance on these matters.

[Michiel Bliemer]

1. If your design is orthogonal it may be possible to get attribute level balance within each block, but only if the specific design exists. If your design is not orthogonal, it is not possible to achieve attribute level balance within each block. Note that attribute level balance within a block is not required and it does not affect estimation results, it is merely a nice feature if it is possible.

2. No designs, not even orthogonal designs, can have zero correlations within each block. Orthogonal designs can only ensure zero correlations across the entire design. Having correlations within a block is not important as you never estimate models using only a single block, you will always pool all the blocks during model estimation. It is typically not even possible to estimate a model using only a single block because it does not contain sufficient information.

It is quite difficult to find suitable orthogonal designs, Ngene merely looks up orthogonal designs from a library. Perhaps a foldover would work, I am not sure, you can try and analyse the correlations. I never worry about correlations, most data used in choice modelling is correlated data, it is very rare that uncorrelated data is used (and there are not really any benefits of it).

If you would like to avoid certain combinations of attribute levels, then you cannot use an orthogonal design. An efficient design allows imposing constraints and you could avoid profiles that are undesirable. Efficient designs have more correlations though, and you seem to find correlations quite important. The alternative strategy is indeed as you say, you create an orthogonal design and remove the profiles that you do not like. However, this breaks orthogonality and increases correlations. But as said, I am not worried about correlations (as long as there is no perfect correlation), so it is an acceptable practice.

Michiel

[Dash76]

Dear Professor, thank you very much for you valuable insights. That's all extremely helpful. All the best.