Planning a Preference Survey

[CMA]

I am planning a preference survey using a Federov algorithm.
I create the following program, where A-C are all categorical variables.

Code: Select all

Design
;alts = alt1*, alt2*
;rows = 20
;eff = (mnl, d) + 2 * (imbalance)
;alg = mfederov(stop=total(1000000 iterations))
;block = 2
;model:
   U(alt1) = b1[0.1].dummy[0|0|0]   * A[1,2,3,4]
        + b2[0.1].dummy[0|0|0|0] * B[1,2,3,4,5]
        + b3[0.1].dummy[0]       * C[1,2]
   /
   U(alt2) = b1 * A
        + b2 * B
        + b3 * C
$


How do I set the number before imbalance?
How do I set a prior value? (Source: coefficient? utility?)

Regards.

[Michiel Bliemer]

You do not need to include a balance objective if you dummy code everything since the design will automatically have a high attribute level balance.
Your syntax is incorrect, below is the correct syntax. Note that the last level is the reference level when dummy coding, so the priors that I set, 0.01, 0.02, etc., indicate that level 4 > 3 > 2 > 1 in terms of preference.

Code: Select all

Design
;alts = alt1*, alt2*
;rows = 20
;eff = (mnl, d)
;alg = mfederov(stop=total(1000000 iterations))
;block = 2
;model:
U(alt1) = b1.dummy[0.01|0.02|0.03]      * A[2,3,4,1]
        + b2.dummy[0.01|0.02|0.03|0.04] * B[2,3,4,5,1]
        + b3.dummy[0.01]                * C[2,1]
        /
U(alt2) = b1.dummy                      * A
        + b2                            * B
        + b3                            * C
$

Priors typically come from a pilot study. If you have not yet done a pilot study, you can use near-zero priors as I have done in the script that indicate the preference order of the attribute levels. If the preference order is not clear you can use 0.

Michiel

[CMA]

Thanks for responding.
There was a change in the research plan, so the syntax was modified.
Preference order is not clear, so prior is set to 0.

Code: Select all

Design
;alts = alt1*, alt2*
;rows = 20
;eff = (mnl, d)
;alg = mfederov(stop=total(1000000 iterations))
?;block = 2
;model:
U(alt1) = b1.dummy[0|0|0]   * A[1,2,3,0]
        + b2.dummy[0|0|0|0] * B[1,2,3,4,0]
        + b3.dummy[0|0]     * C[1,2,0]
        /
U(alt2) = b1.dummy          * A
        + b2                * B
        + b3                * C
$

I have a few questions.
Is there no ".dummy" after b2 and b3 in U(alt2)?
Should the rows be set so that the D-error is less than 0.1?
Since Prior is 0, B estimate is 100 and S estimate is 0. I do not know the minimum sample size required, but how many is good?

Regards.

[Michiel Bliemer]

1. You already defined b1, b2, and b3 as dummy coefficients in alt1, so it does not matter what you write in alt2, you can write b1, or b1.dummy, or b1.dummy[0|0|0], it is all the same.

2. No, you set your rows such that it satisfies the minimum degrees of freedom and such that you have enough variation in your data. The minimum number of rows is 9 for your model, and I usually multiply this with at least 2 or 3 to get sufficient variation.

3. There is no general rule for the sample size that you need since every study is different and it also depends on how important your attributes are for choice (if they are not very important then even with thousands of respondents you may not be able to get statistically significant parameter estimates). If you do a pilot study you will get priors and you can do a sample size calculation. Otherwise, you can let your budget dictate the number of respondents. Your experiment looks quite simple, so I guess 100+ would be enough. I generally have ~1000 respondents for my choice experiments, but they are more complex and have more parameters.

Michiel

[CMA]

Since the minimum number of rows is 9, I will plan on 18.
I found the preference order for b3 only. I created a syntax but I get an error.

Code: Select all

Design
;alts = alt1*, alt2*
;rows = 18
;eff = (mnl, d)
;alg = mfederov(stop=total(1000000 iterations))
?;block = 2
;model:
U(alt1) = b1.dummy[0|0|0]     * A[1,2,3,0]
        + b2.dummy[0|0|0|0]   * B[1,2,3,4,0]
        + b3.dummy[0.01|0.02] * C[1,2,0]
        /
U(alt2) = b1                * A
        + b2                * B
        + b3                * C
$

Error: The modified Federov candidate set size of 2000 could not be achieved. The percentages of candidates that failed are: 97.6% due dominance, 0% due constraints, and 2.4% due repeated alternatives. The candidate set size has been adjusted from 2000 to 1140.

Is it possible to set preference order for b3 only?

Regards.

[Michiel Bliemer]

If you use ;alg = mfederov(candidates=1000) you avoid the error message, but you will get another warning because Ngene is trying to avoid dominant alternatives, but that is not possible if all priors are zero except for one attribute. You must use all zero priors, or you must use non-zero priors for at least 2 attributes. But if the preference order of your attribute levels are not known for the others, then there is no issue with dominance and setting 0.01 and 0.02 for C is not necessary, so just set them to 0.

Michiel

[CMA]

Thank you.
I will create a syntax with all prior set to zero.
I have an additional question based on your previous answer.
I would like to refer to it when I write my literature.

> The minimum number of rows is 9 for your model, and I usually multiply this with at least 2 or 3 get sufficient variation.
Q. Is it judged by the rule of thumb? Please let me know if you have any reference.

> Your experiment looks quite simple, so I guess 100+ would be enough.
Q. Please let me know if there is a reference here to.

Regards.

[Michiel Bliemer]

There are no scientific references, this is based on my experience in conducting choice experiments in the past 20 years.

But the fact that you need to use a larger number of choice tasks in your design rather than close to minimal is quite logical as with a small number of rows you are giving your experimental design little flexibility and if you ever decide to add an interaction effect a nonlinear term then you cannot estimate the model because your data is not rich enough. Therefore, to be on the safe side, I recommend in all my courses on conducting choice experiments that you increase the number of rows and based on experimentation I found that a minimum of 2x or 3x the minimum works well. There is generally no harm in choosing a large number of rows, but there could be issues with choosing a too small number of rows.

In Bliemer and Rose (in press) we mention that "The number of choice tasks is often set to at least two or three times the minimum size to have sufficient degrees of freedom." This is a chapter in the Handbook of Choice Modelling that is due to appear in 2024, you can download the draft here:

https://www.dropbox.com/scl/fi/e23en4r06m18vgrf0ic5b/Bliemer-and-Rose-2023-Designing-and-Conducting-Stated-Choice-Experiments.pdf?rlkey=713snl7hwvbozuswonzc9exd2&dl=0

Michiel