Pilot results

[Edel]

Dear Michiel,

I was hoping to get some advice on the results of my pilot study (n=12).

1. The size and sign of the coefficient for the dummy coded variable social history is unusual. My result suggests that the likelihood of a clinician choosing a more intensive medication regime would be decreased (massively) if the patient lived with others.

2. The sign for both CV event and falls are unusual. My results suggest that clinicians would be more likely to prescribe a more intensive treatment target despite the risk of falls and risk of CV events increasing. So doctors are happy to accept a greater risk of falls and CV events with the intensive regimen (as opposed to a standard regimen).

Value

intensive 0.117467
(scenario variables)
Age -0.049269
baseline BP 0.044589
lives with others 2.908938
multi morbidity -0.792211
(attributes)
cardio event 0.232641
fall event 0.301915
dig health avail -0.322193
peer support -0.158847


Any advice would be appreciated.
Kind regards

[Michiel Bliemer]

You did not provide standard errors of the various parameter estimates so I cannot assess how reliable/significant they are. If you suspect that signs are wrong then I would manually adjust the values to get reasonable priors.

What are the utility functions that you are estimating? Just to make sure that they are appropriate.

Michiel

[Edel]

Thank you Michiel,

below are my utility functions. I omitted the Beta for the interactions from the above post to keep it simple.

V1 = (Int +
cv_event*cv_int + cv_eventAge*cv_int*age +
cv_eventBasel*cv_int*basebp +
cv_eventMulti*cv_int*multi +
fall*fall_int +
fallAge*fall_int*age +
fallSoc*fall_int*sochx +
fallMulti*fall_int*multi +
nudge1*nudg_int+
dig_hlth1*digh_int+
Age1*age +
baseln_bp1*basebp +
social_hx1*sochx +
multi_morb*multi)


V2 = (cv_event*cv_sta +
fall*fall_sta +
nudge1* nudg_sta +
dig_hlth1*digh_sta)


Value Std err t-test p-value Rob. Std err Rob. t-test Rob. p-value
intensive 0.117467 7.546522 0.015566 0.987581 7.380725 0.015915 0.987302

Age -0.049269 0.102087 -0.482623 0.629363 0.099401 -0.495665 0.620131
baseline BP 0.044589 0.069477 0.641784 0.521014 0.070818 0.629634 0.528934
lives with others -2.908938 2.015011 -1.443634 0.148842 1.860034 -1.563917 0.117837
multi morbidity -0.792211 0.904055 -0.876286 0.380874 0.857134 -0.924256 0.355353

cardio event 0.232641 0.104050 2.235851 0.025362 0.105523 2.204646 0.027479
fall event 0.301915 0.134583 2.243331 0.024875 0.134005 2.253016 0.024258
dig health avail -0.322193 0.335252 -0.961046 0.336529 0.339867 -0.947997 0.343131
nudge avail -0.158847 0.321613 -0.493908 0.621371 0.315888 -0.502859 0.615063

cardio event * Age 0.016253 0.014426 1.126657 0.259887 0.013802 1.177543 0.238979
cardio event * baseline bp_155 -0.009619 0.007053 -1.363845 0.172616 0.006743 -1.426504 0.153723
cardio event * multi 0.089958 0.070142 1.282499 0.199668 0.068412 1.314928 0.188534
fall event * Age -0.000681 0.002367 -0.287672 0.773598 0.002261 -0.301085 0.763350
fall event * lives alone 0.345480 0.200930 1.719406 0.085540 0.190206 1.816344 0.069318
fall event * multi 0.019956 0.073835 0.270276 0.786948 0.070649 0.282464 0.777588

[Michiel Bliemer]

Since your experiment seems to have labelled alternatives, the first model I would estimate is a model with alternative-specific coefficients. In your case, all coefficients are generic, only the constant is alternative-specific. Secondly, you only include interaction effects for the first alternative, why not for the second alternative?

Michiel

[Edel]

Hi Michael,
Thank you. I changed my utility functions to include alternative-specific coefficients.

V1 = (Int +
cv_event_int*cv_int +
cv_eventAge*cv_int*age +
cv_eventBasel*cv_int*basebp +
cv_eventMulti*cv_int*multi +
fall_int*fall_int +
fallAge*fall_int*age +
fallSoc*fall_int*sochx +
fallMulti*fall_int*multi +
nudge1_int*nudg_int+
dig_hlth1_int*digh_int+
Age1*age +
baseln_bp1*basebp +
social_hx1*sochx +
multi_morb*multi)


V2 = (cv_event_sta*cv_sta +
fall_sta*fall_sta +
nudge1_sta* nudg_sta +
dig_hlth1_sta*digh_sta)


2. As the interaction effects are to investigate the effect of the scenario variables on the choice I only included them in one alternative.

3. When I try to create a design with all these parameters (19 rows), Ngene returns an undefined message. I can only generate a design that provides an s-estimate when I remove the interaction effects.

4. Considering all that I have learnt I am re-considering including the interaction effects in my design. I plan on removing the interaction effects but instead include alternative-specific constants.

If I remove the interaction effects the signs on some of my coefficients are still unusual but my se are very small.

intensive -1.334293 1.797693e+308

Age 0.084955 3.065261e-02
baseline bp -0.023333 4.096061e-02
lives with others -0.009784 3.754058e-01
multimorbidity -0.033923 1.464819e-01

cv event_int 0.113523 8.568666e-02
cv event_sta 0.172519 1.000710e-01
fall event_int 0.191905 1.797693e+308
fall event_sta 0.000000 3.419618e+01
dig health_int -0.779625 3.996135e-01
dig health_sta -0.062929 4.030926e-01
nudge_int 1.034822 4.286982e-01
nudge_sta -0.438214 3.887803e-01


My interpretation (of the scenario variables) is that participants are more likely to choose the intensive option when the patient is older, but less likely with increasing levels of baseline BP and multimorbidity.
My interpretation of the alternative specific variables is that the risk of cv event that accompanies the intensive alternative has less utility than the risk of cv event that accompanies the standard alternative, both are positive, so both increase the likelihood of choosing the intensive choice but the risk of cv event that accompanies the standard choice has more weight in the decision.

Apologies for all the questions. I appreciate your advice.

[Michiel Bliemer]

What I meant was that if you include fallAge*fall_int*age into V1 but not fallAge2*fall_sta*age into V2 then you are explicitly saying that age can affect the sensitivity to fall_int but age cannot affect the sensitivity to fall_sta. This may bias the parameters and possibly explain the unexpected results. So you should likely include either the same interaction in both utility functions or exclude the interaction in both utility functions.

I cannot comment on why Ngene cannot find a design without seeing the script.

Michiel

[Edel]

Hi Michiel,
Thanks I understand. The reason I think it might be impractical to include the interactions is that by adding them to both utility functions I would increase the number of parameters I am trying to estimate, ie increase the number of rows, blocks and therefore my sample size.

Could you comment on my interpretation?

And the rationale for changing the sign of the parameter estimate if it doesn't make sense, is that because with such a small sample size the estimates are quite unreliable?

Thanks again for your advice.

Kind regards,
Edel

[Michiel Bliemer]

It is difficult for me to interpret the results because I do not know that the alternatives are and it is difficult for me to match the coefficient names with attribute meanings. You talk about "intensive", I assume this is related to V1? I think your interpretation is correct, although I am not sure about "so both increase the likelihood of choosing the intensive choice" if V1 is the intensive option. I do not see a coefficient associated with "risk", but if you refer to cv event_int and cv event_sta then this increases both utilities but it increases the utility of V2 more, so for the same increase in cv event option V2 will become relatively more preferred.

It is always difficult to interpret someone else's output without being involved in the project and being familiar with the variable names, so I am hesitant to say too much about the interpretation. It may also depend on the coding scheme, e.g. whether a variable is dummy/effects coded and what the base level is, etc.

Michiel

[Edel]

Thank you, Michiel.
That has been useful.