insignificant priors from the pilot data

[lemon]

Dear Administrators,

I got pilot data of 120 households (each household gets 6 CE cards), but not all estimates are significant. While some marginally significant parameters (up to 20% significance level) make intuitive sense, those insignificant parameters show unexpected sign. I was wondering if it is possible to combine the marginally significant priors with very small priors that are intended to reflect only signs. I am afraid in this situation, I will apply a higher weight to attributes for which I use the priors obtained from the pilot data.

I would greatly appreciate your advice. Many thanks.

Best,
Adiya

[johnr]

Hi Adiya

What type of design did you do for the pilot? With 120 respondents, that is quite a large pilot - you have 720 observations, which typically would be enough to estimate most models under normal circumstances. Given your findings, there are a few possible causes each of which will affect what comes next.

1. What type of model have you estimated on the data? It might be that there is heterogeneity in the data that you haven't accounted for properly in your model. This is unlikely to be a factor if the parameters are expected to be sign specific, as the heterogeneity might be directional, however if the parameters are not sign specific, then on average the result might be zero. Also, if sign specific, it might depend on whether you allow for heterogeneity in the model, and what distributions you assumed. If the heterogeneity is in the scale and not preferences, then that is more problematic to recover and detect. If this is the case, then it might be that some people are acting more/less deterministically in your sample which if more people are than aren't, then the parameters are likely to be drawn closer to zero. This is unlikely but may occur depending on the design you used for the pilot.

2. Did you do any qual research on the survey? It is possible that the attributes are simply not important in real life. Or you may have too many attributes and people are simplifying by ignoring most of them. With the information provided it is difficult to tell. If this is the cause, then you may need to cull the attributes used.

3. It might be that that preferences in the population are close to zero, but not zero. In this case, it will be hard to detect. The best design for this is an orthogonal design given that these designs assume zero priors and hence are optimal for this case. The problem with such cases however is that the closer the parameter is to zero, the larger the sample size you need to detect whether it is statistically different from zero, even with an orthogonal design - the orthogonal design will however generally require the smallest possible sample size in this case.

4. What design did you use for the pilot? Sort of related to 1 above, but certain designs may allow for certain types of outcomes. If you have a design with minimum overlap for example (e.g., Street and Burgess type designs), then respondents can often select alternatives based on a level in the design (that is, act lexicographically). In this case, the logit model will not be appropriate as the model assumes that the respondents are trading off the attributes, when in the data they are not. Please note, this is not a problem with the design. The design is simply allowing respondents to exhibit this type of behaviour. The problem is with the model which assumes that people are not exhibiting this type of behaviour. If your data has this type of problem, then you may need to try a different model or a model that allows you to recover the attribute processing rules used by respondents at a minimum.

As you can see, it is a little difficult to make suggestions without further details about what you have done already. How you proceed will probably depend on what you have done to date.

John