Bayesian Priors from Pilot Study into Design

[jamalm]

Hello Michiel and All,

I collected some priors [N=30] for my labeled CE survey. I'm trying to design a Bayesian Efficient design for another round of pretest.

My Ngene syntax for the Bayesian Efficient design is what follows:

Code: Select all


Design

;alts = Green*, Solar*, Default
;rows = 36
;block=9,minsum
;eff = (mnl, d, mean)+0.75*(imbalance)
;alg = mfederov(candidates=choice_sets_value.xlsx)
;bdraws=sobol(2000)
;model:

U(Solar) =
            b01[0.000001]            +
            b1[(n,.012935 ,.0085332)] * S[0,.5,.75,1]  +           
            b2[(n,.0891674,.4010865)] * B[0,1]   + 
            b3[(n,.0300308, .1155341)]  * DP[0,8250,12000,17000,25000]  +
            b4[(n,.6927055, 1.475539)]  * FOP[-30,0,15,30,45,65,80,95,110,125,135,160,175,190,225,250]+
            b5[(n,-.0642841,.0894365)]*
     NPV[10175,10188,10219,11382,15794,16470,17000,17071,17072,18395,18408,19382,22044,22058,22088,22618,22690,23410,24013,24026,25000,25283,27967,27980,28011,28941,29173,29632,30277,30618,32774,34864,35568,36187,37173,42139]
 
  /

U(Green) =
           b02[0.000001]     +
           b1 * S  +
           b2 * B  +
           b3 * DP +
           b4 * FOP +
           b5 * NPV     
     
/
 
U(Default) =
            b1 * S_default[0] +
           b2 * B_default[0] +
            b3 * DP_default[0] +
            b4 * FOP_default[125] +
            b5 * NPV_default[23410]     
$


The first time that I ran the design, it would say "undefined." Now, the model unexpectedly closes. I am not sure whether there is a mistake in my codes that I don't catch or there is a bug within Ngene. Any suggestion would be highly appreciated.

Thank you,
Jamal

[Michiel Bliemer]

I believe that there are numerical issues caused by your priors in combination with your attribute levels. You have very large levels (such as 25000) yet you multiply it with a fairly large prior (0.03 in case of b3), which means that for level 25000 the part-worth utility is 0.03*25000 = 750. This is extremely large for a utility. Computing exp(750) in a logit model is not possible as this value is too large for the computer to calculate, you can verify by entering =EXP(750) in Excel and you will see that it causes numerical issues. Unfortunately these numerical issues cause Ngene to crash, so in the next version we will need to implement a check and generate an error message when very large part-worth utilities are detected in order to avoid such crashes. The issue is not only with DP but also with FOP and NPV, their associated prior parameters would be expected to be much closer to zero, e.g. 0.00003 for DP.

Assuming that you obtained these priors by estimating a model using pilot data, maybe something went wrong with the units of your data in your model estimation, so please carefully check the coding of the attribute levels in the data and make sure that you use exactly the same levels in the design as you used in model estimation.

Also, did you also estimate constants b01 and b02 using the pilot study data? You now set them to (near) zero, but similar to the other priors, they should come from the model estimated on the pilot study data.

Michiel

[jamalm]

Thank you Michiel for the prompt response and thorough explanation.

You are right, the levels are causing Ngene to crash. In the pilot design, I used 1 for the lowest level value, 2 for the second-lowest value, and so on rather than using the actual value of the levels. For example, the S attribute has four levels 0, 50%, 75%, and 100% and I had used 1, 2, 3, 4 instead. When I use the same approach (1, ..., n), Ngene doesn't crash. Your response worries me that I should have used the actual values in the original pretest design as well. Am I to ignore the priors from the original pretest and do another pretest with a design that has actual values incorporated in the coding?

Also, NPV is exogenously calculated based on DP, FOP, and SOP attribute levels. To avoid perfect multicollinearity, I dropped NPV from the utility function in the first pretest design. What I want to do now is to replace the SOP attribute with NPV. Am I on the right track or should I include all four attributes?

lastly, I did estimate alternative constants b01 and b02, but they had wrong (negative) signs. Based on previous discussions here, I set them to near zero.

Thanks again,
Jamal

[Michiel Bliemer]

It does not matter which design you used for your pilot or pre-test study, as long as you recode the attribute levels to their actual levels in the data for model estimation it should be fine. For generating an efficient design now, you should only use actual attribute levels to be used in model estimation and be consistent since parameters and priors are directly linked to the data.

Perfect multicollinearity in the data should of course be avoided because this would mean that the model parameters can no longer be estimated. But there are various ways to break multicollinearity, for example using transformations (e.g. logarithm) or by having a loose dependency only (i.e. allowing variations around the exogenously calculated NPV value), or making interaction terms with other attributes.

After re-estimating the model on pre-test data using appropriate attribute levels the constants will also be different. If it is has the wrong sign you can still choose to include it as a Bayesian prior, e.g. (n,0.00001,0.3), to account for the fact that there is uncertainty about its value. If it must be strictly positive, you could consider a uniform distribution, e.g. (u,0,0.5).

Michiel

[jamalm]

Michiel,

I appreciate your response. I will include the actual attribute levels for the upcoming design.

Below is MNL results based on the pilot study (N=30).

Attribute Coef. Std. Err.

NPV -.0642841 .0894365
DP .0300308 .1155341 -->Incorrect sign
fop_usd .6927055 1.475539
sop_usd 0 (omitted)
B_binary .0891674 .4010865
S .012935 .0085332
Green_constant -.3268627 .3348279 -->Incorrect sign
Solar_constant -.2725229 1.077538 -->Incorrect sign


Ngene doesn't crash if I multiply the means and standard errors of DP, FOP, and NPV by 10^-3. Can I actually do so for this design? My Ngene syntax now is what follows.

Code: Select all

Design

;alts = Green*, Solar*, Default
;rows = 36
;block=9,minsum
;eff = (mnl, d, mean)+0.75*(imbalance)
;alg = mfederov(candidates=choice_sets_value.xlsx)
;bdraws=sobol(2000)
;model:

U(Solar) =
            b01[(u,0,0.5)]            +
            b1[(n,.012935 ,.0085332)]                 * S[0,.5,.75,1]  +             
            b2[(n,.0891674,.4010865)]                * B[0,1]   + 
            b3[(n,.0000300308, .0001155341)]     * DP[0,8250,12000,17000,25000]  +
            b4[(n,.00006927055, .0001475539)]   * FOP[-30,0,15,30,45,65,80,95,110,125,135,160,175,190,225,250]+
            b5[(n,-.0000642841,.0000894365)]    * NPV[10175,10188,10219,11382,15794,16470,17000,17071,17072,18395,18408,19382,22044,22058,22088,22618,22690,23410,24013,24026,25000,25283,27967,27980,28011,28941,29173,29632,30277,30618,32774,34864,35568,36187,37173,42139]
/

U(Green) =
           b02[(u,0,0.5)]     +
           b1 * S  +
           b2 * B  +
           b3 * DP +
           b4 * FOP +
           b5 * NPV     
/
 
U(Default) =
            b1 * S_default[0] +
           b2 * B_default[0] +
            b3 * DP_default[0] +
            b4 * FOP_default[125] +
            b5 * NPV_default[23410]   
$


Also, just to be clear, should I include all four attributes and then address perfect multicollinearity in the model via transformation techniques, interaction, etc.? Lastly, do you see any red flags that should be addressed before launching the second pre-test?

Jamal

[Michiel Bliemer]

The MNL results really do not make sense with these attribute levels, there must be something wrong with the model estimation as you cannot have a parameter of 0.03 and then have these very large values for DP. Please re-estimate the model with correct attribute levels.

Dividing the parameters by 1000 will resolve the issue of Ngene crashing, but these coefficients are wrong, you cannot simply divide coefficients by 1000, you need to estimate proper coefficients. Priors should be set very carefully. If you are not sure how to set proper priors, the safest option is to use zero priors.

You cannot add all four attributes in the utility functions without addressing multicollinearity, so you will need to address it already in the experimental design phase in Ngene by applying nonlinear transformations to the data, adding interaction terms, etc. How best to address it is different for each study. You need to specify the utility functions in Ngene exactly as you intend to specify them in model estimation software. If Ngene generates an infinite (or extremely large) D-error then that is often a sign of multicollinearity. If Ngene generates a finite D-error, often below 1, then multicollinearity is not an issue.

Michiel

[jamalm]

Thank you very much for your help, Michiel. I will be in touch after re-estimating the MNL model.
Jamal

[jamalm]

You cannot add all four attributes in the utility functions without addressing multicollinearity, so you will need to address it already in the experimental design phase in Ngene by applying nonlinear transformations to the data, adding interaction terms, etc.
Michiel

Hi Michiel,
I want to circle back to your last comment on my thread. To address multicollinearity amongst a few of our attributes, per your recommendation, we took the natural log of their levels (in Ngene and Stata) and that solved the problem.
My question is now on how to calculate the MWTP for those logged variables. Is my approach below accurate?

Divide the logged attributes' coefficient by the price's coefficient multiplied by the inverse of the population mean of the original attributes. For example, let say DP is one of the logged attributes, therefore,

Code: Select all

MWTP_DP = - coef_lnDP/(coef_price*DP_avg)

Thank you,
Jamal

[jamalm]

Michiel,

I am designing a DCE pilot study in Ngene. Quick background: Every respondent will have a status quo with attributes set to current/base levels and price to 0. Below is my attempt at the initial design to be piloted to get priors for a final Bayesian design. I have a couple of generic questions.

1- Do you recommend using ;eff = (mnl, d, mean) or ;eff = (mnl, wtp(wtp1))?
2- Since U(defult)~=0, will my design change if I do not include the U(Defualt) in my codes and add the ASC coefficient (b0) to both U(A) and U(B)?

Code: Select all

Design 

;alts = A*, B*, Default ?to avoid dominant alternatives
;rows = 24
;block=6 ,minsum

?;eff = (mnl, d, mean)
;eff = (mnl, wtp(wtp1))
;wtp=wtp1(*/b5)

;cond:
if(a.cl=[.5,1], a.co=[1,0]),
if(b.cl=[.5,1], b.co=[1,0])

;bdraws=sobol(2000)
;model:

U(A) =
            b1[.001]    * cl[.2,.5,1]            +           
            b2.dummy[0.001|-0.002]  * s[1,2,0]   +           
            b3.dummy[0.001|-0.002]  * co[1,2,0]  + 
            b4.dummy[0.001|0.002]   * po[1,2,0]  +
            b5[-.001] * pr[0, 10, 20, 50, 75, 100] 
/

U(B) =
            b1  * cl  +             
            b2 * s    + 
            b3  * co  + 
            b4* po    +
            b5* pr
/

 U(Default) =
             b0[-0.001] +
             b1 * cl_default[.2] +
$   

Many thanks in advance.

Jamal
P.S. I did not want to create another thread to ask these couple of questions as the title of this thread applies.

[Michiel Bliemer]

Regarding your first question on computing marginal WTP when you have logarithmic transformations of variables, for any MWTP you need to compute the following (assuming continuous attribute levels, not dummy/effects coded levels):

WTP_k = - (first derivative of partworth utility of attribute k) / (first derivative of partworth utility of cost attribute)

So with the logarithm, the first derivative is d[ beta_k * ln(x_k) ]/d[beta_k] = beta_k / x_k.

If both attribute k and the cost attribute have a logarithmic transformation, then MWTP = - (beta_k / beta_cost) * (x_cost / x_k).

Note that x_k and x_cost refer to the levels for these attributes, hence the MWTP is specific to combinations of attribute levels, it is no longer a single value. If you want to compute the mean WTP then you need to compute WTP_k for each combination of levels and take the mean. Alternatively, you can compute the WTP for the mean attribute levels, where you use mean(x_k) and mean(x_cost), but this is not the same as the mean WTP.

I have answered your other question in my next post (on page 2 of this thread).

Michiel