Advice on Creating a Candidate Set for Partial Profile

[Andrew]

Hello,

I would like to create a partial profile design with 13 attributes, each with 5 levels and 3 unlabelled alternatives. Only 6 of the attributes are to be displayed in the final version. The design creation works well overall, but I would like to avoid a particular issue, if possible. For the candidate set, I have first created random designs using Ngene (see below syntax 1) and Stata (Stata allows me a larger number of rows). I then create systematic overlaps for 7 attributes in each of the rows across all three alternatives. However, in the random designs, I still encounter a lot of overlaps where an attribute level overlaps between two alternatives (e.g. attribute 1 overlaps only between alternative 1 and alternative 2). Creating the partial profile design with the candidate set works without any problems (see below syntax 2). But of course, the final design shows a large number of the above-mentioned overlaps between two of the three alternatives for the 6 attributes displayed.

I would like to minimize the overlap of attribute levels between two alternatives. Can anyone suggest a method that is most effective to create a candidate set that minimizes such overlaps between two alternatives?

Thank you in advance for your help!
Andrew


(1) Syntax for a random design using Ngene:

Code: Select all

Design
;alts = alt1*, alt2*, alt3*
;rows = 10000
;fact
;model:
U(alt1) =
b1.effects[0|0|0|0]    * x1[1,2,3,4,5] +
b2.effects[0|0|0|0]    * x2[1,2,3,4,5] +
b3.effects[0|0|0|0]    * x3[1,2,3,4,5] +
b4.effects[0|0|0|0]    * x4[1,2,3,4,5] +
b5.effects[0|0|0|0]    * x5[1,2,3,4,5] +
b6.effects[0|0|0|0]    * x6[1,2,3,4,5] +
b7.effects[0|0|0|0]    * x7[1,2,3,4,5] +
b8.effects[0|0|0|0]    * x8[1,2,3,4,5] +
b9.effects[0|0|0|0]    * x9[1,2,3,4,5] +
b10.effects[0|0|0|0]   * x10[1,2,3,4,5] +
b11.effects[0|0|0|0]   * x11[1,2,3,4,5] +
b12.effects[0|0|0|0]   * x12[1,2,3,4,5] +
b13.effects[0|0|0|0]   * x13[1,2,3,4,5]
/

U(alt2) =
b1  * x1 +
b2  * x2 +
b3  * x3 +
b4  * x4 +
b5  * x5 +
b6  * x6 +
b7  * x7 +
b8  * x8 +
b9  * x9 +
b10 * x10 +
b11 * x11 +
b12 * x12 +
b13 * x13
/

U(alt3) =
b1  * x1 +
b2  * x2 +
b3  * x3 +
b4  * x4 +
b5  * x5 +
b6  * x6 +
b7  * x7 +
b8  * x8 +
b9  * x9 +
b10 * x10 +
b11 * x11 +
b12 * x12 +
b13 * x13
$


(2) Syntax for the partial profile design using the candidate set:

Code: Select all

Design
;alts = alt1*, alt2*, alt3*
;rows = 120
;eff = (mnl,d)
;alg = mfederov(candidates = excel_design_random.csv)
;block = 1
;model:
U(alt1) =
b1.effects[0|0|0|0]    * x1[1,2,3,4,5] +
b2.effects[0|0|0|0]    * x2[1,2,3,4,5] +
b3.effects[0|0|0|0]    * x3[1,2,3,4,5] +
b4.effects[0|0|0|0]    * x4[1,2,3,4,5] +
b5.effects[0|0|0|0]    * x5[1,2,3,4,5] +
b6.effects[0|0|0|0]    * x6[1,2,3,4,5] +
b7.effects[0|0|0|0]    * x7[1,2,3,4,5] +
b8.effects[0|0|0|0]    * x8[1,2,3,4,5] +
b9.effects[0|0|0|0]    * x9[1,2,3,4,5] +
b10.effects[0|0|0|0]   * x10[1,2,3,4,5] +
b11.effects[0|0|0|0]   * x11[1,2,3,4,5] +
b12.effects[0|0|0|0]   * x12[1,2,3,4,5] +
b13.effects[0|0|0|0]   * x13[1,2,3,4,5]
/

U(alt2) =
b1  * x1 +
b2  * x2 +
b3  * x3 +
b4  * x4 +
b5  * x5 +
b6  * x6 +
b7  * x7 +
b8  * x8 +
b9  * x9 +
b10 * x10 +
b11 * x11 +
b12 * x12 +
b13 * x13
/

U(alt3) =
b1  * x1 +
b2  * x2 +
b3  * x3 +
b4  * x4 +
b5  * x5 +
b6  * x6 +
b7  * x7 +
b8  * x8 +
b9  * x9 +
b10 * x10 +
b11 * x11 +
b12 * x12 +
b13 * x13
$


[Andrew]

Addendum: I added the reject property in the first syntax above and it seems to work. I got a random design with 10.000 rows without any overlaps. Even though Ngene sometimes crashes on my computer. Not sure if some efficiency has been lost. Could this be a proper solution?

In my opinion, the number of overlaps over 2 alternatives was very high bevore. However, such a design would probably reduce the cognitive load on the respondents in contrast to a design without any overlaps. Any advice on this?

Code: Select all

;reject:
alt1.x1 = alt2.x1,
alt1.x1 = alt3.x1,
alt2.x1 = alt3.x1,
alt1.x2 = alt2.x2,
alt1.x2 = alt3.x2,
alt2.x2 = alt3.x2,
alt1.x3 = alt2.x3,
alt1.x3 = alt3.x3,
alt2.x3 = alt3.x3,
alt1.x4 = alt2.x4,
alt1.x4 = alt3.x4,
alt2.x4 = alt3.x4,
alt1.x5 = alt2.x5,
alt1.x5 = alt3.x5,
alt2.x5 = alt3.x5,
alt1.x6 = alt2.x6,
alt1.x6 = alt3.x6,
alt2.x6 = alt3.x6,
alt1.x7 = alt2.x7,
alt1.x7 = alt3.x7,
alt2.x7 = alt3.x7,
alt1.x8 = alt2.x8,
alt1.x8 = alt3.x8,
alt2.x8 = alt3.x8,
alt1.x9 = alt2.x9,
alt1.x9 = alt3.x9,
alt2.x9 = alt3.x9,
alt1.x10 = alt2.x10,
alt1.x10 = alt3.x10,
alt2.x10 = alt3.x10,
alt1.x11 = alt2.x11,
alt1.x11 = alt3.x11,
alt2.x11 = alt3.x11,
alt1.x12 = alt2.x12,
alt1.x12 = alt3.x12,
alt2.x12 = alt3.x12,
alt1.x13 = alt2.x13,
alt1.x13 = alt3.x13,
alt2.x13 = alt3.x13


[Michiel Bliemer]

I am glad you found a way to do what you want. For me, I generally use two methods of creating candidate sets with overlap:

1) A speadsheet in Excel where I create a random design with a large number of rows (say, 100,000) and then simply remove the ones I do not want. For example, for each row I calculate the amount of overlap, then sort the design on the overlap column and then remove the rows above or below it.

2) I write a script in Matlab (or R or Python) that goes through a large number of random choice tasks and keeps the ones that satisfy my criteria.

Note that a candidate set of 2000-5000 is in most cases sufficient. A candidate set of 10,000 rows is fine but the modified Federov algorithm will be very slow.

Regarding cognitive burden, since you have 3 alternatives, this could indeed be an issue and you really want to do some pre-testing and ask respondents to think aloud and give feedback on complexity. Is there a reason why you need 3 alternatives and not 2?

Michiel

[Andrew]

Dear Michiel,

Thank you very much, that was very helpful. I have now written a script in Stata that gives me great flexibility in creating the candidate set. I actually got the candidate set down to 2000. Ngene was a bit slow at 10000 and above and crashed occasionally (which is probably due to my hardware).

We do a Best-Worst Scaling Case 3 to get a larger amount of information with a smaller sample. Hence the 3 choice alternatives. The first pretests were promising.

Kind regards,
Andrew

[Michiel Bliemer]

Note that there is criticism of BWS case 3 because modelling worst choice as the opposite of best choice is often a problematic assumption, see for example the work of Stephane Hess and John Rose. A more acceptable approach would be to do best-best instead of best-worst, where you first ask about the best alternastive, and then ask about the best choice out of the remaining two alternatives. You get the same amount of information, but in a way that is more acceptable within the choice modelling community.

Michiel

[Andrew]

We keep this in mind. In each choice set, the best choice should be dynamically hidden, leaving only the remaining alternatives for the best-best choice. Otherwise this would possibly increase the completion time. We will check whether we can implement this with SE.
Thank you very much!

Andrew