NGene Crashing when I run a design-- cost levels too high?

[Jbenson]

Hi all!
I'm pretty new to NGene still, and I'm trying to design an unlabelled DCE with three alternatives which vary on eight attributes. It will be a blocked design, with two arms-- arm A asks about one type of medical imaging, arm B another. Each arm will have two blocks: one for uninsured patients, one for patients with health insurance (the software I'm using for the experiment doesn't easily allow pivoting, otherwise I'd use that). So my current approach is to make four designs, one for each sub-block, and present the relevant design to each participant.

The first design I'm working on is imaging service A, for uninsured patients. My betas are taken from a mmnl model from pilot study with a few participants looking at a similar scenario. The issue is that when I run the design with the true cost levels (2100,3150,4200), NGene spits out an "unexpected error" and crashes. Running it as I have below with [1,2,3] it goes fine-- is this an issue for determining dominance? The B-estimate is ~26, so it's not quite utility balanced, which makes me think the betas and levels aren't quite right. The scale is very different, and I think the cost attribute will be very dominant, especially among uninsured folks. Any guidance would be appreciated!

Code: Select all

design
;alts = Facility_A, Facility_B, Facility_C
;rows = 12
;eff = (mnl,d)
;con
;model:
U(Facility_A) = b1[1.7]  * specialty[1,2,3]
        + b2[1.5]  * pcprec[0,1]
        + b3[-0.63] * timeresult[0,24,48]
        + b4[-1.39] * cost[1,2,3]
        + b5[-0.48] * timetravel[10,20,30]
        + b6[-0.5]  * timewaitapt[1,4,7]
        + b7[1.2]   * onlinesched[1,2,3]
        + b8[1.4]   * service[1,2,3]
        /
U(Facility_B) = b1 * specialty
        + b2 * pcprec
        + b3 * timeresult
        + b4 * cost
        + b5 * timetravel
        + b6 * timewaitapt
        + b7 * onlinesched
        + b8 * service
        /
U(Facility_C) = b1 * specialty
        + b2 * pcprec
        + b3 * timeresult
        + b4 * cost
        + b5 * timetravel
        + b6 * timewaitapt
        + b7 * onlinesched
        + b8 * service
$


[Michiel Bliemer]

You need to use the exact same attribute levels in your design as you use in estimating the model in your pilot study. It is not possible to have a parameter value of 1.39 if your attribute levels in the data are 2100, 3150 and 4200. So please have a look in the data that you used to estimate your model and be consistent. Perhaps re-estimate the model by using these exact cost levels, this will lead to a much smaller parameter for cost.

You may want to consider dummy coding for any categorical variables, e.g. speciality[1,2,3] sounds like a categorical variable. if you, please use something like b1.dummy[prior1|prior2] * specialty[1,2,3] where level 3 is the reference level, and re-estimate your pilot study model using the same dummy coding to obtain values for prior1 and prior2.

Michiel

[Jbenson]

Michiel,

Thank you for the detailed reply! I've updated my categorical attributes to be dummy coded. Per your reply, I think I need to change the source of my priors, in that case, as we are looking to use different levels (and some new attributes) from the previous study, so it makes sense that those betas are not applicable here. I'm looking around for literature with similar levels and attribute combinations, and will use those as a base (assuming they are not too unreasonably large) to see where that gets me.

For the structure of the design, I'm thinking a heterogeneous pivot design is what I'm looking for, as there are two sets of costs available depending on the subject's insurance status. To create a design like this, in the manual both reference and pivot types are shown for levels. Is it possible to do this type of design when I am not able to actually pivot around a user's input value? (Or, alternatively, I'm using Qualtrics for the survey-- do you know of an established way to do pivoting in there? If so, a homogeneous pivot design would definitely be ideal, as I could pivot around the subject's estimated cost and travel time.)

Thank you so much!
Jamie

[Michiel Bliemer]

Using pivots in survey tools like Qualtrics is tricky. I generally avoid pivots by simply generating multiple designs and simply picking the appropriate design for each respondent based on their characteristics or input. I call this a "library of designs" approach, in some cases I had over 1000 designs in a "library" in the survey tool for different segments in the population.

Michiel

[Jbenson]

Michiel,

That sounds like it may be the best approach, here! In a "library of designs" method, how do you interpret the S-estimate of each design? Do you assume that you only need to exceed the sample size for one of the designs, as everything about the designs is similar except the levels being pivoted, or do you try to do some kind of quota filling / add extra padding? I don't foresee us running into an issue with getting enough participants, but I would like to have an accurate feel for how to estimate the needed sample size. Thanks so much!

Jamie

[Michiel Bliemer]

I suppose you could compute sample size estimates for each individual design and then compute a weighted average based on the number of respondents you expect in each segment/category.

Michiel

[Jbenson]

Michiel,

I've gotten a library of designs made, that is working well and will be a great basis for a pilot to improve for the final design. I've done a little bit of digging, and I think I'll be able to get a pivot design working in Qualtrics, with a little doing. I took a stab at transitioning my design over to a pivot design with a reference alternative based on the participant's own entries, however I'm running into a roadblock: I can't seem to use dummy coding and pivoting together. I have a few attributes, like type of doctor, or parking availability, which I could easily ask participants about ahead of time to set the reference alternative value, however I can't see a way to combine dummy coding and pivoting in NGene. Is this something I can do and have it be valid? In a pivot design are you able to treat categoricals as continuous to generate the pivot?

If it is helpful, here is the syntax for one of my "library of designs" designs:

Code: Select all

design
;alts = Facility_A*, Facility_B*, Facility_C*
;rows = 12
;eff = (mnl,d)
;con
;model:
U(Facility_A) = b1.dummy[0.6|0.4]  * specialty[3,2,1]
        + b2.dummy[0.4]  * pcprec[1,0]
        + b3[-0.3] * timeresult[0,24,48] ?https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6554244/
        + b4[-0.01] * cost[2100,3150,4200]
        + b5[-0.2] * timetravel[10,20,30]
        + b6[-0.4]  * timewaitapt[1,4,7]
        + b7.dummy[0.2]   * service[2,1,0]
        /
U(Facility_B) = b1 * specialty
        + b2 * pcprec
        + b3 * timeresult
        + b4 * cost
        + b5 * timetravel
        + b6 * timewaitapt
        + b7 * service
        /
U(Facility_C) = b1 * specialty
        + b2 * pcprec
        + b3 * timeresult
        + b4 * cost
        + b5 * timetravel
        + b6 * timewaitapt
        + b7 * service
$


And here is the design I am attempting to use for a pivoted version of the above:

Code: Select all

design
;alts = Facility_A*, Facility_B*, Facility_C*
;rows = 12
;eff = (mnl,d)
;con
;model:
U(Facility_A) = b1[0.6]  * specialty.ref[2]
        + b2[0.4]  * pcprec.ref[1]
        + b3[-0.3] * timeresult.ref[24]
        + b4[-0.01] * cost.ref[1000]
        + b5[-0.2] * timetravel.ref[20]
        + b6[-0.4]  * timewaitapt.ref[4]
        + b7[0.2]   * service.ref[1]
        /
U(Facility_B) = b1 * specialty.piv[-1,0,1]
        + b2 * pcprec.piv[-1,0]
        + b3 * timeresult.piv[-50%,0%,50%]
        + b4 * cost.piv[-50%,0%,50%]
        + b5 * timetravel.piv[-50%,0%,50%]
        + b6 * timewaitapt.piv[-50%,0%,50%]
        + b7 * service[0,1,2]
        /
U(Facility_C) = b1 * specialty.piv[-1,0,1]
        + b2 * pcprec.piv[-1,0]
        + b3 * timeresult.piv[-50%,0%,50%]
        + b4 * cost.piv[-50%,0%,50%]
        + b5 * timetravel.piv[-50%,0%,50%]
        + b6 * timewaitapt.piv[-50%,0%,50%]
        + b7 * service[0,1,2]
$


Thanks so much!
Jamie

[Michiel Bliemer]

I am not sure what a pivot around a dummy variable would mean. If parking available is Yes or No, then there is not really any need for a pivot as both levels are fixed. Pivots are mainly useful if the levels show to different respondents are very different, which typically happens with quantitative variables. Therefore, currently Ngene does not support pivots for categorical variables, but perhaps if I better understand how a pivot of a categorical variable would be useful we could consider implementing it.

Michiel

[Jbenson]

Michiel,

That makes sense-- my question then is how to tell NGene that there is a categorical attribute in a reference alternative that is the same across choice tasks for an individual?

Jamie

[Michiel Bliemer]

That is typically done via constraints. Suppose that altref if the reference alternative and x1 is the attribute and you want to fix to level 1:

;require:
altref.x1 = 1

You can create different designs for each population segment. For example, for a study in the UK we generated more than 1000 designs consisting of different combinations of reference attribute levels, creating a whole library of designs to choose from depending on the respondent.

Michiel