Results pilot test, attribute non significant

[dce.farmers]

Dear choice modellers,


I have finished to collect the pilot data of my DCE with a sample of 24 respondents. I have therefore estimated a conditional logit (with the Nlogit software) in order to use the priors for my efficient design in Ngene for the final data collection. Here are the results I get:

Discrete choice (multinomial logit) model
Dependent variable Choice
Log likelihood function -131.77138
Estimation based on N = 144, K = 10
Inf.Cr.AIC = 283.5 AIC/N = 1.969
---------------------------------------
Log likelihood R-sqrd R2Adj
Constants only -142.6627 .0763 .0431
Note: R-sqrd = 1 - logL/Logl(constants)
---------------------------------------
Chi-squared[ 8] = 21.78269
Prob [ chi squared > value ] = .00533
Response data are given as ind. choices
Number of obs.= 144, skipped 0 obs
--------+--------------------------------------------------------------------
| Standard Prob. 95% Confidence
CHOICE| Coefficient Error z |z|>Z* Interval
--------+--------------------------------------------------------------------
SUPPINDI| .25548 .30245 .84 .3983 -.33732 .84828
SUPPCOL| .44822 .28504 1.57 .1158 -.11044 1.00689
TVXINDGP| .07871 .32121 .25 .8064 -.55086 .70827
TVXCOOP| -.01138 .33274 -.03 .9727 -.66353 .64078
COST|-.33857D-04 .2752D-04 -1.23 .2186 -.87794D-04 .20079D-04
PULVINGP| -.86147*** .28159 -3.06 .0022 -1.41339 -.30956
PULVCOOP| -.73240** .30344 -2.41 .0158 -1.32713 -.13767
TECHCOLL| .43551* .22877 1.90 .0570 -.01288 .88389
A_ALT1| 1.42733*** .45573 3.13 .0017 .53412 2.32054
A_ALT2| 1.37941*** .47549 2.90 .0037 .44746 2.31136
--------+--------------------------------------------------------------------
nnnnn.D-xx or D+xx => multiply by 10 to -xx or +xx.
***, **, * ==> Significance at 1%, 5%, 10% level.
-----------------------------------------------------------------------------


1. My two first attributes are not statistically significant: does it necessarily mean that I need to exlcude them from my model for the collection of the final data? In my opinion, my sample if too small to draw strong conclusions at that stage so I would prefer to keep at least the COST and the SUPP attributes as they are almost significant. Furthermore, I would be very limited later on for the analysis if I drop my monetary/cost attribute. If I keep these attributes, should I however increase the amount of levels for the COST attribute? The reason for the non significance may be the lack of variation.

2. Would you recommend me to estimate other models than a conditional logit at that stage or it is fine to extract my priors based on the results of the estimation of a conditional logit?

3. What other steps do you recommend me before extracting my priors?

Based on the results above my attributes and levels are:
Support type 1 attribute --> 3 levels (none,SUPPINDI= personalized, SUPPCOL=collective), dummy variable
Mode of plantation attribute --> 3 levels (individual, TVXINDGP= collective form1, TVXCOOP=collective form2), dummy variable
Cost (COST) attribute --> 4 levels (45 000 €,50000 €,55000 €,60000€), continious variable
Mode of spraying --> 3 levels (individual, PULVINDGP=collective form1, PULVCOOP=collective form2), dummy variable
Support type 2 attribute --> 2 levels (personalized, TECHCOLL=collective), dummy variable


Thanks a lot in advance for your advice!

Best,

Gaëlle

[johnr]

1. No, you don’t need to exclude them. As you state, the sample of your pilot is too small to draw conclusions of this nature. Indeed, you have done remarkably well to get the other parameters to be statistically significant with such a small sample. I would be highly encouraged by these results.

2. You have a few options moving forward. Are the parameters the correct sign? If so, you may wish to keep the design and collect the full sample. The standard errors will decrease naturally, and the parameters are likely to become more statistically significant as the sample increases. So option 1 is to do nothing, the design seems to be good, just your pilot is too small. Option 2, is you can take the parameters you have from the pilot as shown, and redo your design, only this time as a design with Bayesian priors. You use the parameter estimates as the means, and the standard errors as the variances in the Bayesian priors. This means e parameters that are not statistically significant have greater uncertainty (they are not statistically significant), and you re-optimize using the values you got from the pilot. I wouldn’t worry about different models at this stage. Given the lack of sample, you are doing very well as it is.

3. As I said, I would probably re-optimize the design using Bayesian priors taken from the model you have. I wouldn’t change anything else. Your results with only 24 respondents is really good!

John

[dce.farmers]

Hi John,

Thank you so much for your detailed answer and your encouragements!

I have an aditional question. This is about the prior I should use for the ASC/ intercept which is coded as b0[0] in my Ngene model: What estimate should I retrieve from the results I showed in my post? Is it the estimate for A_ALT1 or ALT_2? Both of them? In the literature, there is usually only one constant term which is the statu quo instead of Alt1 and Alt2.

Thank you!

Best,
Gaëlle

[Michiel Bliemer]

What are your alternatives? alt1, alt2, and statusquo? Or is it an optout alternative instead of status quo alternative?

If it is an opt-out alternative, you would have something like:

;alts = alt1*, alt2*, optout

U(alt1) = asc_alt[1.4] + b1 * suppindi + ... /
U(alt2) = asc_alt[1.4] + b1 * suppindi + ... /
U(optout) = 0

Or alternatively, you would have (this is the same behavioural model):

U(alt1) = b1 * suppindi + ... /
U(alt2) = b1 * suppindi + ... /
U(optout) = asc_optout[-1.4]

Estimating a different ASC for alt1 and alt2 is always a good idea as it accounts for some left-to-right bias (hence your alt1 ASC is slightly larger than your alt2 ASC), but for the purpose of priors for experimental design you are only interested in a single constant asc_alt or asc_optout as in the utility functions above.

If your alternative is a status quo alternative, then the utility function will not be merely zero or a constant, but it would have attributes with fixed levels. In that case, I would not expect the constant to be that large.

Michiel

[dce.farmers]

Hi Michiel,

Thank you for your answer!
Very good point, I am actually not 100% sure whether I have a status quo or an opt out alternative. What I did in SurveyEngine is that I wrote in the survey to my respondents what each attribute meant for the status quo, so for instance, that choosing the status quo meant no cost incurred, no support, no share of material etc.
However, when I wrote the code to generate the first design for my DCE before the pilote test, I did not modelize my utility fonction by setting another level 0 for all my attributes such that U (alt3) = b1dummy * support [0] + b2dummy * plantation [0] etc. So three questions:

1. Based on my explanations in the survey, is my alt3 a status quo or an opt-out option?
2. In case it would be a status quo, does it makes sense to change the utility function of alt3 in my Ngene code, at that stage/after my pilot test now, and specify it with fixed attribute level for collection of final data?
3. Is my constant for the status quo reasonnable? (see results with table below when the level of the attributes for my SQ are set to 0 in my data). If not, what should I do?

Thanks!
Best,

\G

This was my previous code:

Code: Select all

design
;alts = alt1*, alt2*, alt3
;rows = 24
;eff = (mnl,d)
;block = 4
;alg = mfederov
;reject:
alt2.cost=52000 and alt2.plantation=2,
alt2.cost=52000 and alt2.plantation=3,
alt1.cost=52000 and alt1.plantation=2,
alt1.cost=52000 and alt1.plantation=3,
alt2.cost=40000 and alt2.plantation=1,
alt1.cost=40000 and alt1.plantation=1
;model:
U(alt1)= b1.dummy[0.00001|0.00002]*support[2,3,1] ? 1= none, 2=personalized, 3=collective
                     + b2.dummy[0.0001|0.0001]*plantation[2,3,1]  ? 1=individual (base), 2= collective autonomous, 3=collective coop                 
                     + b3[-0.00001]*cost[40000,44000,48000,52000](5-7,5-7,5-7,5-7)
                     + b4.dummy[0.0001|0.0001]*spraying[2,3,1]
                     + b5.dummy[0.00002]*suivi[1,2]   
                     + b6[0.00001]*cost*plantation /
U(alt2)= b1*support
         + b2*plantation
         + b3*cost
         + b4*spraying
         + b5*suivi
         + b6*cost*plantation /

U(alt3)= b0[0]
$

The results with a constant as status quo:

-----------------------------------------------------------------------------
Discrete choice (multinomial logit) model
Dependent variable Choice
Log likelihood function -143.37839
Estimation based on N = 150, K = 10
Inf.Cr.AIC = 306.8 AIC/N = 2.045
---------------------------------------
Log likelihood R-sqrd R2Adj
Constants only -153.8090 .0678 .0357
Note: R-sqrd = 1 - logL/Logl(constants)
Warning: Model does not contain a full
set of ASCs. R-sqrd is problematic. Use
model setup with ;RHS=one to get LogL0.
---------------------------------------
Response data are given as ind. choices
Number of obs.= 150, skipped 0 obs
--------+--------------------------------------------------------------------
| Standard Prob. 95% Confidence
CHOICE| Coefficient Error z |z|>Z* Interval
--------+--------------------------------------------------------------------
Constant| -2.15633 1.94088 -1.11 .2666 -5.96038 1.64772
SUPPINDI| .24568 .29068 .85 .3980 -.32404 .81540
SUPPCOL| .47685* .27420 1.74 .0820 -.06058 1.01428
TVXINDGP| 1.28301 3.59360 .36 .7211 -5.76032 8.32635
TVXCOOP| -.01094 .34721 -.03 .9749 -.69146 .66957
COST|-.25181D-04 .3490D-04 -.72 .4705 -.93576D-04 .43214D-04
PULVINGP| -.78189*** .26400 -2.96 .0031 -1.29932 -.26446
PULVCOOP| -.65822** .28113 -2.34 .0192 -1.20923 -.10722
TECHCOLL| .37381* .21773 1.72 .0860 -.05293 .80055
COST_TVI|-.22903D-04 .7049D-04 -.32 .7452 -.16105D-03 .11525D-03
--------+--------------------------------------------------------------------
nnnnn.D-xx or D+xx => multiply by 10 to -xx or +xx.
***, **, * ==> Significance at 1%, 5%, 10% level.
Model was estimated on Apr 07, 2024 at 04:49:53 PM
-----------------------------------------------------------------------------

[Michiel Bliemer]

1. It seems a status quo alternative but you modelled it as an opt-out alternative by only including the constant. It results in the same behavioural model though, but information about the support and plantation will end up in the constant instead of in the b1 and b2 parameters.

2. I would probably use something like:
;require:
alt3.plantation = 0,
etc.

But if the status quo level ONLY appears in the status quo alternative then the model may become nonidentified and then you may indeed only be able to estimate a constant for all the reference levels together.

3. Your constant is probably fine as you did not include other attribute levels and it captures the effect of all reference attribute levels.

Note that -.25181D-04 = -0.000025181, see the footnote under the Nlogit output. Also COSTY_TVI and the standard errors require adjusting the decimal point.

Michiel

[dce.farmers]

Hi Michiel,

Thank you for your answer and spotting that I missed a D in the script ...
I have now edited my priors for the cost and cost interaction attributes accordingly.

The levels of my status quo alternative are all specific to my status quo (as being zero) EXCEPT for the first attribute as it refers to NONE/ NO support being given.

Hence I have been re-writting my script as below but I get this error message once I include the require constraint: "Error: The ';require' property contains an attribute that does not exist in the design. 'alt3.support'". I would understand this error message if the level 1 did not exist for my attribute support but I do specify it in utility function for alt1 and alt2.

1. How should I procede instead with the coding? Should I forget any require constraint?
2. Based on the fact that I realized that my cost coefficient is really close to 0, should I add another level for my cost attribute? This would be to ensure that I get a more substanital effect for the second round of data collection.

Code: Select all

design
;alts = alt1*, alt2*, alt3
;rows = 24
;eff = (mnl,d, mean)
;bdraws = gauss(2)
;block = 4
;alg = mfederov
;reject:
alt2.cost=60000 and alt2.plantation=2,
alt2.cost=60000 and alt2.plantation=3,
alt1.cost=60000 and alt1.plantation=2,
alt1.cost=60000 and alt1.plantation=3
;require:
alt3.support=1
;model:
U(alt1)= b1.dummy[(n, 0.245, 0.290)|(n,0.476, 0.274)]*support[2,3,1] ? 1= none, 2=personalized, 3=collective
                     + b2.dummy[(n,1.283,3.593)|(n,-0.011,0.347)]*plantation[2,3,1]  ? 1=individual (base), 2= collective autonomous, 3=collective coop                 
                     + b3[(n, -0.000025181,0.0000349)]*cost[45000,50000,55000,60000](5-7,5-7,5-7,5-7)
                     + b4.dummy[(n,-0.781, 0.264)|(n,-0.658,0.281)]*spraying[2,3,1]
                     + b5.dummy[(n,0.373,0.217)]*suivi[2,1]   
                     + b6[(n,-0.0000229,0.0000704)]*cost*plantation.dummy[2] /
U(alt2)= b1*support
         + b2*plantation
         + b3*cost
         + b4*spraying
         + b5*suivi
         + b6*cost*plantation.dummy[2] /

U(alt3)= b0[(n,-2.156,1.940)]
$



Thanks again for the help,
Gaëlle

[Michiel Bliemer]

1. You need to add + b1*support to the utility function of alt3.
2. The contribution to utility matters, not the cost coefficient. So -0.000025*60000 = -1.51 is definitely not close to zero.

Michiel

[dce.farmers]

Great, the code works now, thanks Michiel.

Just to double-check before I select a design and put it into Survey Engine, is my code below fine? Or would you advise me to change anything else?
I changed bdraws to halton type of distribution as I also changed mean to median now for drawing the Baysian priros.

Thanks,
\G

Code: Select all

design
;alts = alt1*, alt2*, alt3
;rows = 24
;eff = (mnl,d, median)
;bdraws = halton(200)
;block = 4
;alg = mfederov
;reject:
alt2.cost=60000 and alt2.plantation=2,
alt2.cost=60000 and alt2.plantation=3,
alt1.cost=60000 and alt1.plantation=2,
alt1.cost=60000 and alt1.plantation=3,
alt2.cost=45000 and alt2.plantation=1,
alt1.cost=45000 and alt1.plantation=1
;require:
alt3.support=1
;model:
U(alt1)= b1.dummy[(n, 0.245, 0.290)|(n,0.476, 0.274)]*support[2,3,1] ? 1= none, 2=personalized, 3=collective
                     + b2.dummy[(n,1.283,3.593)|(n,-0.011,0.347)]*plantation[2,3,1]  ? 1=individual (base), 2= collective autonomous, 3=collective coop                 
                     + b3[(n,-0.000025181,0.0000349)]*cost[45000,50000,55000,60000](5-7,5-7,5-7,5-7)
                     + b4.dummy[(n,-0.781, 0.264)|(n,-0.658,0.281)]*spraying[2,3,1]
                     + b5.dummy[(n,0.373,0.217)]*suivi[2,1]   
                     + b6[(n,-0.0000229,0.0000704)]*cost*plantation.dummy[2] /
U(alt2)= b1*support
         + b2*plantation
         + b3*cost
         + b4*spraying
         + b5*suivi
         + b6*cost*plantation.dummy[2] /

U(alt3)= b0[(n,-2.156,1.940)] + b1*support
$


[Michiel Bliemer]

I would change to

;bdraws = sobol(500)

Halton draws are usually not very good with so many Bayesian priors, and 200 draws is likely not enough.

Michiel