Pilot design

[Tbh2017]

HI,

I am a new user of the NGENE software and was hoping for some input after reading the manual and going through the posts in this forum. I am going to conduct a DCE - an efficient design based on a pilot for which I will obtain estimated. The DCE will be distributed to 2000 respondents, and we expect a response rate of min. 50%. The topic is screening - preferences. I have two screening programmes A and B. The attributes and levels are equal across the two alternatives. The pilot study will be an orthogonal design and I tried to design it according to the manual and previous comments in this forum. Will this design be ok for such pre-study? I may need to look at heterogeneity in the final study (e.g. socio economic status and levels of health literacy). This is my syntax:

?Attributes
?do (1,5,20)(number of life saved when screening 1000)
?us (1,2,3) (three different examinations: blood pressure, CT scan and blood test)
?tid (10, 1, 2) (10 minutes, 1 hour and 2 hours)
?fort (5, 25, 50) (numbers that regret screening out of 1000 screened)
?unod (5, 25, 50) (numbers receiving unnessecary treatment out of 1000 screened)

Design
;alts = alt1, alt2
;rows = 12
;orth=seq
;eff = (mnl, d)
;model:
U(alt1) = b1 + b2 * do[1,5,20] + b3 * us[0,1,2] + b4 * tid[10,1,2] + b5 * fort[5,25,50] + b6 * unod[5,25,50] /
U(alt2) = b2 * do + b3 * us + b4 * tid + b5 * fort + b6 * unod $

* Do I need the bi coefficient in the U(alt1)?

This was the outoout I got:

Warning: Defaulting to prior values of zero for the following priors: 'b2, b3, b4, b5, b6'

[OrthogSwapInvert] Design found
Finished, at 10:32:48 PM, 2/22/2017


MNL efficiency measures:
D-error: 0.004313
A-error: 0.040263
B estimate: 100
S estimate: 0

Prior b2 b3 b4 b5 b6
Fixed prior value 0 0 0 0 0
SP estimates Undefined Undefined Undefined Undefined Undefined
Sp t-ratios 0 0 0 0 0

Design
Choice situation alt1.do alt1.us alt1.tid alt1.fort alt1.unod alt2.do alt2.us alt2.tid alt2.fort alt2.unod
1 5 1 1 50 5 1 0 2 25 5
2 20 1 2 5 50 5 2 1 50 25
3 1 2 2 25 50 5 0 1 50 25
4 1 2 10 5 25 20 0 10 25 50
5 1 0 2 25 5 5 1 1 50 50
6 20 2 10 25 5 1 0 10 5 25
7 5 1 1 50 50 20 1 2 5 50
8 5 2 1 50 25 20 2 10 25 5
9 20 1 2 5 5 5 1 1 50 5
10 1 0 10 5 25 20 1 2 5 5
11 20 0 10 25 50 1 2 2 25 50
12 5 0 1 50 25 1 2 10 5 25


Thank you in advance.

Tina

[Michiel Bliemer]

The syntax looks fine. Since your experiment sounds unlabelled, you do not have to design it with a constant in Alt1 so you could remove b1.

You could let go of orthogonality if you wish, which leads to a slightly more efficient design:

Design
;alts = alt1, alt2
;rows = 12
;eff = (mnl, d)
;model:
U(alt1) = b2 * do[1,5,20] + b3 * us[0,1,2] + b4 * tid[10,1,2] + b5 * fort[5,25,50] + b6 * unod[5,25,50] /
U(alt2) = b2 * do + b3 * us + b4 * tid + b5 * fort + b6 * unod $

If you are keen to keep the design orthogonal, I would increase the number of rows to 15:

Design
;alts = alt1, alt2
;rows = 15
;orth = seq
;eff = (mnl,d)
;model:
U(alt1) = b2 * do[1,5,20] + b3 * us[0,1,2] + b4 * tid[10,1,2] + b5 * fort[5,25,50] + b6 * unod[5,25,50] /
U(alt2) = b2 * do + b3 * us + b4 * tid + b5 * fort + b6 * unod $

While with 12 rows you get a near-orthogonal design, with 15 rows it is truly sequential orthogonal.

If you believe 15 choice tasks per person is too many, you may consider blocking the design. For example, you can split a design with 24 rows into two blocks (i.e. two versions of the survey, each with 12 choice tasks):

Design
;alts = alt1, alt2
;rows = 24
;orth = seq
;eff = (mnl,d)
;block = 2
;model:
U(alt1) = b2 * do[1,5,20] + b3 * us[0,1,2] + b4 * tid[10,1,2] + b5 * fort[5,25,50] + b6 * unod[5,25,50] /
U(alt2) = b2 * do + b3 * us + b4 * tid + b5 * fort + b6 * unod $

Good luck!

[Tbh2017]

Hi again.

Thanks a lot for your reply. It does not accept 15 rows but switches to 18:

[OrthogSwapInvert] Could not locate design in 15 rows. Switching to design with 18 rows.
[OrthogSwapInvert] Design found
[OrthogSwapInvert] Examining table combination: 2/5
[OrthogSwapInvert] Examining table combination: 3/5
[OrthogSwapInvert] Examining table combination: 4/5
[OrthogSwapInvert] Examining table combination: 5/5
Finished, at 12:50:30 PM, 2/23/2017

So if I use 18 rows and 2 blocks I will generate 9 choice sets per respondent.

Do you think this will be ok?

Final question: Are there any rules as to how many respondents I shall include in a pilot study using a blocked design as mine to obtain estimates for an effcient design.

Tina

[Michiel Bliemer]

You are right, it is actually an 18 row design. Yes if you block it in 2 then each respondent faces 9 choice tasks.

There is no rule of thumb for how many respondents you need for a pilot study, as this completely depends on how much each of your attributes influences choice. If they are all very important factors, you need much less than if they are all not very relevant factors. What we typically do is put the questions in Excel and send it to a few colleagues (e.g. 5 or 10). If you give all 18 questions to each colleague, this will provide up 90 to 180 choice observations, with which you may already be able to estimate your 5 coefficients, so you only need a small pilot sample. If none of the coefficients are statistically significant then know that you may need a much larger pilot sample.

Once you have obtained coefficients to use as priors for your efficient design, then Ngene can inform you about sample sizes required for your main data collection, as in order to provide such an estimate a guesstimate of each coefficient is required.

[Tbh2017]

Ok. Sorry but just to clarify. Thank you for being really helpful.. The problem here is that the population of the study (health care research) differs from my collegues since the study population will include males 65-74. The initial think aloud study revealed that 10 choice sets in this population was boderline meaning that 18 will be too many. So.. 1) wouldn`t it be a problem to test the DCE on a completely different population e.g. friends or collegues? 2) Alternatively I could block the design for the pilot (2 or 3) into 9 or 6 choice sets and send it to males similar to the populations in the study. However is it at all feasible to block the design in a pilot study and in that case will I need a much larger number of respondents? I suppose I still pool all the responses and analyse them together in spite of blocked designs?

Tina

[Michiel Bliemer]

You asked the question about the number of respondents needed, and I responded that there is no way telling until you actually start doing the pilot. So if you want to know at least something about sample size, I proposed a pre-pilot with colleagues who are usually willing to do more than 10 choice tasks. And often your colleagues may be able to imagine being another person, but this will of course only give you some rough indication and therefore is only useful to get some feedback and to maybe get an initial idea about sample size needed. There is otherwise no way to give an answer to your question how many respondents you need in your pilot study. Whether you really want to do a pre-pilot is another question, I would be happy without it. You will NOT use the data from a pre-pilot in your final model estimations of course.

You CAN include responses from both a pilot study and the main study into one data set for model estimation, including all blocks. Usually you will apply a different scale parameter (e.g. nested logit) to account for differences in the pilot design (which is often an orthogonal design) and the main design (which is often an efficient design).

Yes if you block the design, you need more respondents, it is the total number of choice observations that matters. But think about linear regression, if you are estimating 5 coefficients, would 10 observations be enough? Or 100? Or 1000? This clearly depends on how important the attributes are. For some studies, 10 observations is enough, while in others 1000 may not even be enough, as the attributes may actually not be very important to respondents. I do not know your study, you may know more from other studies, but the pilot will tell you. You can start collecting data from a small number of respondents, estimate a model, and if nothing is significant, keep collecting more until some or all of your parameters are statistically significant. These coefficients can then be used (together with their standard errors) to inform Bayesian priors in an efficient design that is given to a larger sample in the main study.

The number of blocks should be determined based on how many choice tasks you think respondents are willing to answer. If it is a face-to-face interview this number is higher than in an online survey. It also depends on how interested the respondents are in the study and other factors, and how difficult to choice tasks are. You could ask probably 6 difficult choice tasks to a respondent, and about 12 simple ones, but the more you ask, the more they may experience fatigue and no longer pay attention.

Michiel