Candidate set & overlapping

[psalazar]

Hi, I have the following design with five attributes (only one categorical, and the rest numerical) and two alternatives:

Design
;alts = progA*, progB*
;rows = 24
;block = 2
;eff = (mnl,d)
;alg = mfederov(candidates = candidate_set_pilot3_10nocost.csv)
;model:
U(progA) = b0[0.000001]
+ byol[0.000001] * yol[0.5,1,3]
+ bqol[0.000001] * qol[20,40,60]
+ bexp.dummy[0.000001|0.000002] * exp[0,1,2]
+ bsize[0.000001] * size[50,5000,10000]
+ bequ[0.000001] * equ[25,50,75]
/
U(progB) = byol * yol
+ bqol * qol
+ bexp * exp
+ bsize * size
+ bequ * equ
$

Since I want all choice tasks to have overlapping on two attributes, I am using a candidate set created in Stata. The candidate set is evenly distributed in terms of the overlappings per attribute: 7,776 rows overlap on "yol", 7,776 rows overlap on "qol", 7,776 rows overlap on "exp", etc..(total # rows: 19,440).

When I ask Ngene to create a D-efficient design (with priors close to zero), and using the candidate set indicated above, it gives me a design with no choice set overlapping on "exp" (D-error< 0.0009).

I have run the design with different numbers of rows and blocks, and I have also tried imposing restrictions on the number of times each level shows up. I have also used a candidate set with overlapping on just one attribute, in each choice set. Yet, I obtain choice sets with no overlap on "exp", or one at most (i.e when using 24 rows).

Do you know why is this happening? I would appreciate very much your help with this.

Many thanks!

Best wishes,

Pamela.

[Michiel Bliemer]

Hi Pamela,

There may be two explanations:

1. The modified Federov algorithm simply starts with the top rows and it tries consecutive rows for improvement but keeps the current rows if no improvement is found. Perhaps it never finds an improvement by inserting in any the last 7776 rows (which have overlap on 'exp'). So you could try to randomise the order of the rows in your candidateset to see if that actually changes anything.

2. I believe that overlap in exp leads to more loss of efficiency than having overlap in yol or qol for the reason that exp is dummy coded and has two parameters. Since estimation of two parameters requires more information, it would be more efficient to have no overlap in exp, while estimating the single parameters for yol and qol is much easier and therefore these attributes are 'sacrificed' to have overlap so that more information can be obtained for exp.

If explanation 2 is correct, then you can consider also dummy coding yol and qol just for the sake of the experimental design (since you are using zero priors anyway). Please keep me updated as I am interested to understand this as well.

Michiel

[psalazar]

Dear Michiel,

Thank you very much for your prompt and kind response!
Apologies for the late reply; It took me a while to find a design I could work with. I tested them using synthetic data before responding (I simulated different behaviour patterns).

I tried both options, and explanation 2 seems most likely. When I randomise the rows of the candidate set and use the same design, I still obtain only one choice set (out of 24) with overlaps in “exp” .

If I treat “QoL” and “YoL” as dummy variables, I get 5 (out of 24) choice sets with overlap on “exp”, but in “size” we have no choice set comparing 5.000 (i.e. middle level) with any of the other two levels. I added specified an attribute level count constraint (i.e. 8-8, 8-8, 8-8) on “Size” and “Equ” (the only two continuous variables in the model) and get a better design: I now have 11 choice sets (out of 24) that overlap on “Exp”, although on “Size” I only got five choice sets with overlap (D-error: 0.021268).

I also tried a design in which all variables are treated as categorical, and even though overlaps are more evenly distributed across attributes, the D-error is greater (D-error=0.50961). I am therefore using the design in which only “YoL”, “QoL” and “Exp” are treated as categorical, in addition to the attribute level count constraint on “Size” and “Equ”.

Something that I have noticed is that, when working with partial profile designs, choice sets with strongly dominant alternatives are not always avoided when I ask Ngene to compute a D-efficient design. I easily solved it by eliminating choice sets with dominant alternatives when constructing the candidate set, but I found it a bit weird. I run at least 10 different models and always got 1, 2 or sometimes 3 choice sets (out of 24) with a dominant alternative.

Thanks again for your suggestion! It worked out and I am now moving to run the pilot

Best wishes,

Pamela.

[Michiel Bliemer]

A few responses:

* It is correct that Ngene will often not use the middle levels of numerical attributes since this is less efficient (making larger trade-offs between extreme levels provides more information), so if you require some degree of attribute level balance you indeed need to impose attribute level constraints as you have done.

* The D-error of a design where the model has categorical variables cannot be compared to the D-error of a design where the model has numerical variables. So there should not be an issue in using the design where you effects coded everything.

* Constraints (including dominance constraints) are currently not applied when you read in an external candidate set, it is assumed that all constraints are applied when creating the candidate set. I agree that this may be confusion and not well documented, so for the upcoming release version of Ngene we will be applying all constraints also when an external candidate set is used.

Michiel

[psalazar]

Dear Michiel,

Thank you very much for your responses.

On your second point, may I ask you why I should not compare both models, and which paper/book would you suggest for me to expand on this?

Thanks again for all your guidance!

Best wishes,

Pamela.

[Michiel Bliemer]

The D-error is based on the covariance matrix of the parameter estimates. The covariance matrix with effects coding is entirely different than the covariance matris with numerical attributes, for example with effects coding the numnber of parameters is much larger and the attribute values are -1, 0, or 1. The D-error is the determinant of the covariance matrix, and if the covariance matrices are entirely different, they should not be compared. You can compare different designs for the SAME model based on the D-error.

A similar argument holds for comparing loglikelihood values to assess model fit. You can LL values for the SAME DATA SET across different models, but you should not compare LL values across different data sets.

Michiel

[psalazar]

Thank you very much, Michiel!