Bayesian dessign doubts ( efficiency, mfederov)

[catalpb2]

Dear Ngene Team,

I have some questions about the draft of my final design that I have created mostly based on my pilot study mnl estimations, although not all coefficients were significant, and I changed one that seem to have the wrong (counterintuitive) sign to 0.0001. For context, it is a study of a contract for adoption of sustainable practices. My sample will be around 200 people given budget constraints, I am hoping I can get significant coefficients although the Sp estimate is very large for certain parameters. I would highly appreciate an overall opinion on my design and efficieny meassures, as well as in the following questions:

1. In the final design I mostly get alternatives with the extreme levels of the compensation or both alternatives with the mid-level. I think I can see why this could be good for efficiency, but I am afraid that given the social context of my sample, they would decide solely on money when they have the two levels. Is there any way to “correct” this, without reducing the efficiency?

2. I tried the mfederov algorithm as it seems to be good for unlabeled alternatives, but it gets stuck on a first evaluation. When I put a high number of candidates it says that 97% fail because of dominance, bit even if I don’t it gets stuck on the first evaluation. I am afraid that this indicates an underlying issue in my design that I am not seeing. Moreover, although it is a Bayesian design, with the default algorithm after less than one thousand evaluations efficiency only improves marginally.

Code: Select all

I design
;alts = Program_A*, Program_B*, NoProgram
;rows = 12
;eff = (mnl,d,mean)
;bdraws = gauss(3)
;model: U(Program_A) =  land.effects
[(n,0.176853,0.093418 )|( n, -0.407689,0.097612 ) ]* LANDDIST [1,2,0]
 + bonus.effects[( n, 0.0001, 0.09)|( n, 0.005, 0.09)]* BONUS[1,2,0]

 + b4[(n,0.004224,0.006390)] * TRAINNING[6,18,30]
 + b5[( n, 0.16677 , 0.039614)] * COMPENSATION  [9.5,11.5,13.5] +     /
U(Program_B) =  land.effects* LANDDIST  + bonus.effects* BONUS
 + b4 * TRAINNING
 + b5 * COMPENSATION      /

U(NoProgram )= b0[( n, 0.297499, 0.468543)]

                  
   Fixed   Bayesian mean            
D error   0.069551   0.072005            
A error   0.188116   0.194609            
B estimate   42.285649   0.437932            
S estimate   101716625.151439   69778052.160138            
                  
Prior   land(e0)   land(e1)   bonus(e0)   bonus(e1)   b4   b5
Fixed prior value   0.176853   -0.407689   0.0001   0.005   0.004224   0.16677
Sp estimates   32.051998   7.129901   101716625.151439   38645.996092   238.099494   5.7846
Sp t-ratios   0.346201   0.734031   0.000194   0.00997   0.127021   0.814928
Sb mean estimates   783.054112   8.944371   69769090.35203   26538.976585   183.341272   7.439606
Sb mean t-ratios   0.340182   0.718718   0.099137   0.108058   0.191779   0.798514



Thank you very much for help!

Catalina

[Michiel Bliemer]

1. The only way to avoid this when searching for an efficient design is to use dummy/effects coding. I suggest you dummy code the compensation attribute. Of course you can still estimate a linear effect afterwards. For dummy coding you not only have to update the priors for the compensation attribute, but also of constant b0. Assuming a linear relationship you can manually compute the updated priors, but to be safe you may want to estimate a model with the dummy coding and use the estimated parameters as priors.

2. No it does not get stuck. The default number of candidates is 2000 if you do not specify anything (which is usually more than enough), so you simply have to wait until 2000 evaluations before Ngene reports the next best design as it needs to cycle through all candidates. So you simply need to be patient If you specify 500 candidates, Ngene will show the next result after evaluating only 500 candidates.

Michiel

[catalpb2]

Dear Michiel,

Thank you very much for your answer! I will try your recommendations.

Would you say it is definitely recommendable to use the mfederov command for my dessign? Or if I get a lower d-error with the default algorithm I should leave it without mfederob?

Finally, do you think the overall dessign is reasonable? it is my first project of DCE and Ngene and I don't want to have any major issues.

I really value your feedback, thanks again !

Catalina

[Michiel Bliemer]

Usually the default algorithm is fine, I generally only use the mfederov if the default algorithm is unable to find a design due to dominance or many constraints. The mfederov algorithm does not ensure attribute level balance in the design.

I think that the script looks fine and I see no issues, but you could consider increasing the number of rows to give you more variety in the data. For example, ;rows = 18 or ;rows = 24, in combination with ;block = 2. You will see that the default algorithm will struggle in generating 24 rows without any dominant alternatives. In that case, the mfederov algorithm may be a better.

Michiel

[catalpb2]

Thank you very much for your feedback and suggestions Michiel.

The reason I did not initially use blocking is because I have some latent variables and I will try to do a Hybrid Choice Model, and I was advised against blocking if my sample was too small but I will definitely take a second look at this.