Efficient design with blocking - Deletion in 1 block

[djourdain]

Dear All,

For a choice experiment, we have used Ngene to prepare a WTP-efficient design leading to 12 scenarios. To avoid overloading respondents with too many choice situations, we requested two blocks. Upon observation of the choice proposed, one of the proposed choice situation was not making sense, and we decided to eliminate it from the final design.

Therefore, the final design of our survey was to interview 200 respondents with 6 choice situations, and 200 respondents with 5 choice situations.

Are there severe consequences when eliminating one choice situation in one block? Or, as we saw it practiced in at least one published articles, is this an acceptable practice?

A reviewer is asking us to declare the efficiency of our design before and after the elimination of this choice situation; Is there a simple way to do this?

Any help and indication of literature on this topic would be of real help to us!

Best,

Damien

[johnr]

Hi Damien

It is difficult to say what the consequences might be, however they are usually not as bad as what people think. It is important to note that efficiency relates to the VC matrix of the model, which is translated usually to mean that you will get more robust (lower standard errors for the) estimates. You should also note that in generating the design, you are assuming priors, and in all likelihood you got these wrong (if you knew them precisely to the 10th decimal point in the first place, then you would not have needed to do the study). So before you even start out, you are likely to loose efficiency between the design generation phase of the project and the empirical phase.

Now to answer your question - by changing the design (by deleting rows, or via some other mechanism), you will generally loose some efficiency - each choice task adds to the Fisher (how it does that depends on the model type) - so simply deleting a row will necessarily loose efficiency but if you delete a bad task (one that adds very little information to the Fisher), then the loss in efficiency will be less than the loss if you deleted a task that adds a lot of information to the Fisher. But what precisely does a loss of efficiency mean - higher standard errors. But we know that the VC matrix is divisible by N the sample size (noting that the diagonals are the parameter variances so if you take the sqrt of the variance to get the s.e., then the relationship between N and the s.e. is a sqrt function), so you can compensate for a loss of efficiency by having a larger sample size (though the sqrt function suggests that for each additional respondent you get less bang for your buck). But here's the cool thing - you might only need for example 10 respondents for your study (check your S-error for the min. theoretical sample size), and deleting a good row might increase this to say 15 (this is purely an example), yet you collected 200 respondents. So who cares - certainly not me.

The model type will play a role in this however - for the panel MMNL model for example, it requires taking the product of the probabilities to calculate the VC matrix. Deleting a row (calculating one less probability) may have a larger impact (who knows) on the Fisher than say for an MNL model which simply sums the task specific components of the Fishers (the probs are treated as if they are independent in the calculation. The model type will also impact upon the blocking of models that take the product of the probabilities. To demonstrate consider a design for with tasks which I denote 1,2,3,4. For each task, I obtain a task specific Fisher matrix, denoted F1,F2,F3,F4, which sum to give the the overall Fisher F such that F = F1 + F2 + F3 + F4, it doesn't how you block (in terms of impact upon efficiency) as F = B1(F1 + F2) +B2( F3 + F4) is equivalent F = B1(F1 + F4) +B2( F2 + F3) where B represent the blocks. If the product is used in the calculation as opposed to a summation, then that is another matter. You don't exactly multiply the task specific Fishers so my example to follows is not precisely what is happening internally, but only useful to illustrate the point. If you take products then F = B1(F1 * F2) +B2( F3 * F4) is equivalent F = B1(F1 * F4) +B2( F2 * F3). The point is, deleting a row in this case may have a larger impact depending on the blocking used! But again, if your design needs 10 or 15 respondents and you collected 200, then who cares.

Ngene allows you to evaluate designs - not just generate it. Take your original design (with blocks), save it in Excel, delete the row, and evaluate the new design. Make sure you save the design in the right format (this is discussed in the manual) and change the rows in your evaluation syntax from 12 to 11.

John

[djourdain]

Dear John,

Thank you for this very comprehensive and clear (and somehow reassuring) answer. I will look at the eval function more carefully that I had overlooked so far.

Best

Damien