pilot study

[felipelobo]

Dear Ngene team,

I will start a SP study in the freight transport sector. I aim to have a sample of up to 50 companies. I would like to clarify some doubts.

1) How should I proceed with the sample size in the pilot study? Can I have a sample of 5 companies in the pilot study?

2) In the pilot study, should all estimated coefficients be significant?

3) With the parameters estimated with the data collected in the pilot study, Is there any distribution that you recommend to develop the efficient design for the final questionnaire?

4) The model has an attribute ("Risk") that is presented as a percentage to respondents (eg, risk level has three levels: 10%, 20%, 30% for three alternatives). In Ngene, can we write this way: Risk [0.1,0.2,0.3]? I think I can use it that way and use the same numbers (0.1,0.2,0.3) to calibrate the discrete choice model later.

Thank you for your attention.
Regards,
Felipe Souza

[Michiel Bliemer]

1) Yes, usually one uses 10% of the final sample.

2) No, and with a sample of 5 it may be unlikely that all coefficients will be significant.

3) If the signs of all coefficients are sensible, then I recommend using normally distributed Bayesian priors (n,mean,stdev) where mean equals the estimated coefficient and stdev is the standard error that comes out of the estimation process. If the standard error is very large (i.e., the coefficient is not significant), then you can still use a normal distribution, but I would recommend using the median D-error instead of the mean D-error to avoid computational issues with extreme draws, i.e. ;eff = (mnl,d,median). For coefficients with the wrong sign, you may want to use a zero mean or a uniform Bayesian prior such as (u,lower,upper).

4) Yes, you want to use the levels that you also use in model estimation.

Michiel

[felipelobo]

Dear Professor Bliemer,

Thank you for your answer.

Research with companies is difficult due to the restrictions they have in answering questionnaires. Thus, a pilot study would already be difficult to apply due to the sample restrictions. My model has 6 attributes. 2 attributes are specific to my country, and I don't find any previous parameters in the literature.

1) Is it possible to develop an efficient design for the final questionnaire without applying the pilot study? In the model I would leave two attributes with only the negative sign (-0.001) and the other attributes with parameters found in the literature.

2) One of the attributes in the model is called "Freight Price". I found a parameter for this attribute in an article developed in Indonesia. For attributes related to the local currency of a specific country, how can I proceed to use the parameter of a study from another country? Is any procedure necessary?

Thank you for your attention.
Regards,
Felipe Souza

[Michiel Bliemer]

1. One typically either uses all non-zero priors or one uses zero priors for all parameters, it is usually not a good idea to mix zero values (or near-zero values) with values from the literature. Also, note that scaling issues exist when taking values from the literature. The safest option is to use zero priors, or near-zero values. If you feel you have the expertise, you could use expert judgement, see Bliemer and Collins (2016).

Bliemer and Collins (2011) On determining priors for the generation of efficient stated choice experimental designs. Journal of Choice Modelling, 21, 10-14.

2. I generally do not use any values from the literature because of scale issues. I sometimes use ratios of coefficients, like willingness-to-pay (WTP), and then scale values according to expert judgement. I have never used WTP values from a different country.

Determining priors is an art, not a science, and are merely best guesses so you can do what you want. But it is also where most people go wrong in selecting inappropriate priors, and bad priors can lead to very inefficient designs, even less efficient than using zero priors. Therefore I generally do not recommend people setting priors manually unless they know what they are doing.

Michiel

[felipelobo]

Dear Professor Bliemer,
Thanks for your response. I had already read your article suggested in the last post. However, I don't think I have the capacity to judge the prior values.
Due to the sample restrictions for conducting a pilot study which procedure do you suggest to develop the final questionnaire?
This is the current code with zero priors. 2 factors do not have prior values in literature.

Code: Select all

design
;alts = 1,2,3
;rows = 10
;eff = (mnl,d)
;model:
U(1) = b01+b1* FactorA1[1,2,3]+ b2* FactorB1[480,600,720] + b3 * FactorC1[251,314,376] + b4 * FactorD1[1,2,3] + b5 * FactorE1[0,0.15]+b6 * FactorF1[0,1] /
U(2) = b02+b1* FactorA2[1,2,3]+ b2* FactorB2[424,530,636] + b3* FactorC2[183,229,275] + b4* FactorD2[1,2,3] + b5* FactorE2[0,0.15,0.3]+b6* FactorF2[0,1]/
U(3) =         b1* FactorA3[1,2,3]+ b2* FactorB3[540,675,810] + b3* FactorC3[934,980,1176] + b4*FactorD3[1,2,3] + b5* FactorE3[0,0.15]+b6* FactorF3[0,1]
$

Thank you for your attention.
Regards,
Felipe

[Michiel Bliemer]

If it is not possible to use expert judgement then I would recommend using zero priors as you really want to avoid using the wrong priors.

Is there a reason why you use values such as 251, 314, and 376? These values put a significant cognitive burden on the respondent, values such as 250, 300 and 350 are much easier to interpret for respondents, but perhaps you have a specific reason why you use these precise values.

Michiel

[felipelobo]

Dear Professor Bliemer,

Thanks for the information.

For to the final questionnaire (without carrying out a pilot study), do you recommend using low value sign (-0.001 or + 0.001) if I know the sign of each factor?

In Factor B, there are three levels. I am using a pivot design. The central value is the reference. Do you recommend to change the values?

Regards,
Felipe Souza

[Michiel Bliemer]

Using small positive or negative values is only useful when you are checking for dominant alternatives in case your alternatives are generic, e.g. ;alts = alt1*, alt2*, alt3*

But in your syntax you have not specified them as generic. If they are, please add an asterisk and use very small positive/negative values, e.g. 0.000001 and -0.000001, as priors, so Ngene can remove choice tasks with dominant alternatives.

Since you are doing a pivot design, you will need to make sure that you use rounded values in the survey instrument. It is not so important which values you use for optimisation, it is more important which values you show to respondents as you need to reduce their cognitive burden.

Michiel

[felipelobo]

Dear Professor Bliemer,

Thank you very much for your answers. I apologize for the excess of questions. I would like to clarify some doubts.

In the case of a final questionnaire developed from a design with zero priors approach, what are the implications for the research? Could this be a reason for criticism / rejection of the study?

Do you know any article that developed a final questionnaire with zero priors due to limitations (limited sample, and inability to judge priors in the context of the country where the study is being conducted)?

Thank you very much for your attention.
Regards,
Felipe Souza

[Michiel Bliemer]

Using zero priors will not lead to a rejection of the study. Many people are using random choice tasks or orthogonal designs, so using an efficient design (even if using zero priors) is already more advanced and definitely acceptable in the literature.

It is quite common to use zero priors, and most of the experimental designs based on the work of Street & Burgess implicitly assume zero priors, but I am unable to refer you to specific papers.

Michiel