Comparing GOF across two nearly identical experiments

[Matteo Corsi]

I have two datasets from two discrete choice experiments conducted in parallel, at the same time and with the same sample of respondents. The choice tasks of both experiments had in common:

- the text introducing the experiment;
- two unlabelled alternatives described using the same attributes and the same levels;
- a status-quo option.

The difference between the two datasets comes from the wording used in the survey instrument to describe the status quo option: in one case it was explicitly described, in the other it was simply "Neither A, nor B". I have estimated the same conditional logit model on both datasets. Would you consider it acceptable to compare the goodness-of-fit of the two estimates and how would you compare it?

Thanks, Matteo

[Michiel Bliemer]

No you would not compare the model fit across data sets, you should only compare model fit (of different models) within the same data set.

Instead of estimating two separate models, I suggest that you pool the data and that you add a dummy variable, i.e. D = 1 if described as "Neither A nor B" and add this as an attribute to the optout/status quo alternative. If the coefficient of this dummy variable is not statistically significant, then respondents considered the two descriptions more or less the same, while if it is statistically significant it means that the description results in differences in behaviour.

Michiel

[Matteo Corsi]

Thank you very much, Michiel.

Matteo