Bayesian Priors from Pilot Study into Design

[Michiel Bliemer]

You can use candidates = 1000 without any problem, that works for me. It even works with candidates = 100.

Michiel

[jamalm]

I am not sure what happened, but it works fine now. Thanks for checking.

Jamal

[jamalm]

Hello, Michiel.

I conducted a DCE survey in 2018; I made the design in SAS and R at the time. The paper was published in the Energy Journal [https://www.iaee.org/energyjournal/article/3555]. I would like to re-design the experiment in Ngene using the priors from this paper and possibly do another survey in a near future. Below is my current design. I have a few questions that I'd appreciate it if you could answer.

1- What is the best way to require the water attribute level to be less than 6 when the rps attribute level is either 0.8 or 1? I have tried many different versions of ;require: and ;reject: Ngne commands, but none of them were accurate or close to the output of the following design. I understand that Ngene doesn't allow using the mfederov algorithm while using the ";cond." I am under the impression from my correspondence with you that it'd be better to use the mfederov algorithm in a design.
2- If the answer to the first question is no, should I go with this design? Are there any red flags in my code? The output of the current design looks fine, besides the S estimate that is extremely large.

Code: Select all

Design
;alts = A*, B*
;rows = 24
;block= 4 ,minsum

;eff = (mnl, d, mean)

?;alg = mfederov(candidates=200)

;cond:
if(a.rps=[.2,.5],a.water=[6,10]),
if(b.rps=[0.8,1],b.water=[1,3])   

;rep=500
;bdraws=sobol(2000)
   
   ;model:
    U(A) =    b1[(n,.0225559, .0027288)]    * rps[0.2,0.5,0.8,1]
               + b2[(n,.0358811, .0073098)]    * rpv[.05,.09,.2,.3]     
               + b3[(n,-.2872152, .0682302)]   * cr[0,1] ?0no 1yes           
               + b4[(n,-.1844927, .0323452)]   * water[1,3,6,10]
               + b5.dummy[(n,-.0677639, .1062624)|(n,.1551728, .1182273)|(n,.2258507, .1068817)]  * sm[1,2,3,0] ?1text, 2online, 3home, 0no
               + b6[(n,-.0406509,.0026012)]    * pr[0, 5, 10, 20, 30, 50]
               + b7[(n,-.0008801, .0001976)]   * rps*rpv
               /
    U(B)       = b0[(n,-.4425761, .1407202)]
               + b1 * rps
               + b2 * rpv
               + b3 * cr
               + b4 * water
               + b5 * sm
               + b6 * pr
               + b7 * rps*rpv
   $



As always, thanks for your timely reply.

Jamal

[Michiel Bliemer]

With the default swapping algorithm, you should use:

;cond:
if(a.rps=[0.8,1],a.water=[1,3]),
if(b.rps=[0.8,1],b.water=[1,3])

With the modified Federov algorithm, you should use:

;reject:
a.rps = 0.8 and a.water = 6,
a.rps = 0.8 and a.water = 10,
a.rps = 1 and a.water = 6,
a.rps = 1 and a.water = 10,
b.rps = 0.8 and b.water = 6,
b.rps = 0.8 and b.water = 10,
b.rps = 1 and b.water = 6,
b.rps = 1 and b.water = 10

Your priors look strange though. Prior values of 0.0022 and 0.035 are tiny when multiplying with levels 0.2-1 and 0.05-0.3 respectively. This indicates that both rps and rpv are essentially irrelevant in choice making. Are you sure that you are using identical attribute levels that you used in your previous study? The prior values do not seem to match the attribute levels. This results in very large sample size estimates because these priors are telling Ngene that these attributes are essentially irrelevant.

Michiel

[jamalm]

Michiel,

Thanks for your prompt response. I went back to my original codes and the levels used were not in percentage (not 0.20, 20). I fixed the levels for rps and rpv, and the sample size now looks reasonable (n<500)- thanks for catching that.

I ran both codes and the final outputs look very different. The default swapping algorithm yields a highly balanced design; whereas, the modified Federov algorithm yields a very unbalanced design. Am I under the right impression that I should use the modified Federov algorithm output?

Thanks,
Jamal

[Michiel Bliemer]

The modified Federov algorithm does not look at attribute level balance, it is a row-based algorithm and cannot achieve attribute level balance easily. So the default swapping algorithm will in most cases be best if that works, while in cases where it does not work (e.g., when too many constraints have been imposed and the swapping algorithm cannot find feasible designs) one would switch to the modified Federov algorithm. When using the modified Federov algorithm, some degree of attribute level balance is desirable. In the manual several ways are discussed to increase level balance, namely

(1) imposing hard attribute level balance constraints, e.g.

b2[(n,.0358811, .0073098)] * rpv[.05,.09,.2,.3] (4-8,4-8,4-8,4-8)

which means that each level needs to appear between 4 and 8 times (where 6 times would be perfect attribute level balance given that there are 24 rows, but perfect balance is extremely challenging for the algorithm to find)

(2) imposing soft attribute level balance constraints, e.g.

;eff = (mnl,d,mean) + 2*(imbalance)

Please refer to the manual for more information.

Michiel

[jamalm]

Thanks for the clarification. I have come to realize that the second [soft] approach is much easier to use/run than the first [hard] approach.

I have read (more like skimmed through) the Ngene manual a few times now, and I have not come across a comparison of the different algorithms one can use. The only thing that I have seen is a short paragraph on page 165 of the manual that states, "For different types of designs (e.g., orthogonal, efficient, orthogonal efficient, with constraints, etc.) different algorithms are used, including RSC (relabelling-swapping-cycling) algorithms, swapping algorithms, and Modified Federov algorithms. For efficient designs, the swapping algorithm is the default." So, I appreciate what you said about the swapping approach being best when it works.

Thanks again!
Jamal

[jamalm]

Dear Michiel and NGene team,

I use priors from two previous surveys to create the following Bayesian-efficient design. NGene stopped finding designs after 77105 iterations, which took 26 hours. The model has been running for an additional 24 hours after the 77105th design, and it doesn't seem like it's going to find any more designs. The 77105th design has a large D error (0.240785), A error (0.784889), and S estimate (377966.520818), which makes me concerned about this design.

I have two questions:

1) Are there any red flags in my design? Is there a way to make this model run faster/more efficiently?
2) Should I be concerned about the D & A errors and S estimate? Do you have any suggestions to lower them?

Thank you in advance for taking the time to respond.

Best,
Jamal

Code: Select all

Design
;alts = A*, B*, Default*
;rows = 24
;block= 4 ,minsum
;eff = (mnl, wtp(ref1))
;wtp= ref1(b0,b1,b2,b3,b4,b5,b7,b8/b6)

;con
;alg = mfederov (candidates=1000)
    ;require:
    Default.bev = 0.04,
    Default.rps = 0,
    Default.rpv = 0,
    Default.po = 0,
    Default.jo = 0,
    Default.pr = 0
;bdraws=sobol(2000)
 
   ;model:
    U(A) =    b1.dummy[(n,.2931043, .0413873)|(n,.1040212, .0522824)]   * bev[.5,.82,.04]
               + b2.dummy[(n,.0421512, .0368125)|(n,.1830361, .0427255)|(n,.2274791,   .0460439)|(n,.2815779,   .0793215)|(n, .3725451,  .063924)]   * rps[1,2,3,4,5,0]
               + b3.dummy[(n,0.0443329884, 0.0077785571)|(n,0.05689339,   0.0069986313)|(n,0.0491955064, 0.0075071036)]   * rpv[5,10,20,0]
               + b4.dummy[(n,-.1361521, .0438639)|(n,.2445789, .0439351)]   * po[1,2,0]
               + b5.dummy[(n,.2222731, .0179133)|(n,.32867245,   .0244022)]  * jo[1,2,0]
               + b6[(n, -.0144308,.0007273)]               * pr[0, 5, 15, 30, 50, 75]
               + b7[(n, .0016078, .0002654)]               * bev*rps
               + b8[(n,-.001296,  .0002125)]               * rps*rpv
               /
    U(B)       = b1 * bev
               + b2 * rps
               + b3 * rpv
               + b4 * po
               + b5 * jo
               + b6 * pr
               + b7 * bev*rps
               + b8 * rps*rpv

               /

    U(Default) = b0[(n,-.688979,   .0514445)]
               + b1 * bev
               + b2 * rps
               + b3 * rpv
               + b4 * po
               + b5 * jo
               + b6 * pr
               + b7 * bev*rps
               + b8 * rps*rpv
    $


[Michiel Bliemer]

The long computation time is because you are using a large number of Bayesian draws as you have many Bayesian priors. You can reduce the computation time by selecting only a subset of priors as Bayesian while others are kept fixed. Priors where the standard error is small, relative to the parameter value, could be kept as a locally fixed prior since the Bayesian distribution will be very narrow. You mainly want to use normally distributed Bayesian priors for parameters that have a wider distribution narrow.

I would use ;eff = (mnl,d) and not use ;wtp since for attributes with dummy coding the WTP is not simply a ratio of parameters and therefore it is not appropriate to use the ;wtp command.

The Sp-estimates reported by Ngene when using ;eff = (mnl,d) look reasonable, although of course some of them are large because some of your priors are very close to 0, e.g. b7 is very small, therefore this interaction effect does not seem to contribute much to utility and will be difficult to estimate (if your prior is correct).

Your b3 dummy coefficients are essentially all near-zero, therefore attribute rpv does not contribute much to utility and therefore becomes irrelevant. You may want to check whether you did the dummy coding correctly in model estimation. Perhaps you used levels 0,5,10,20 in model estimation (which could explain the near-zero parameter values) instead of dummy coding with values 0 or 1?

Why do you believe that 0.24 is a large D-error? D-errors are model specific and cannot be compared across studies.

Michiel

[jamalm]

Following your comments, I revised the design to what follows:

Code: Select all

Design
;alts = A*, B*, Default*
;rows = 24
;block= 4 ,minsum
;eff = (mnl, d)

;con
;alg = mfederov ?(candidates=1000)
    ;require:
    Default.bev = 4,
    Default.rps = 0,
    Default.rpv = 0,
    Default.po = 0,
    Default.jo = 0,
    Default.pr = 0
?;bdraws=sobol(2000)

   ;model:
    U(A) =     b1[.005]   * bev[4,50,82] ?continuous
               + b2.dummy[(n,.042, .037)|(n,.18, .043)|(n,.23, .046)|(n,.28, .079)|(n, .37,.064)] * rps[1,2,3,4,5,0] ?Dummy coded
               + b3[.044] * rpv[0,5,10,20] ?continuous
               + b4.dummy[(n,-.14, .044)|(n,.245, .044)]   * po[1,2,0] ?Dummy coded
               + b5.dummy[(n,.22, .018)|(n,-.33, .024)]  * jo[1,2,0] ?Dummy coded
               + b6[-.0144308]               * pr[0, 5, 15, 30, 50, 75] ?continuous
               + b7[.00161]               * bev*rps
               + b8[-.0013]               * rps*rpv
               /
    U(B)     = b1 * bev
               + b2 * rps
               + b3 * rpv
               + b4 * po
               + b5 * jo
               + b6 * pr
               + b7 * bev*rps
               + b8 * rps*rpv
               /
    U(Default) = b0[(n,-.69, .051)]
               + b1 * bev
               + b2 * rps
               + b3 * rpv
               + b4 * po
               + b5 * jo
               + b6 * pr
               + b7 * bev*rps
               + b8 * rps*rpv
    $


How does this design look? It runs rather fast when defaulting to 200 Halton Bayesian draws rather than sobol(2000) (understandable!). When I tried to run this design, Ngene sometimes give the following warning and sometimes it actually runs; I thought that was interesting

Error: There were problems generating a fractional factorial of choice tasks. For the modified federov algorithm, increasing the number of candidates might assist.

Is there a particular reason for NGene to be inconsistent here?

Thanks,
Jamal