Switching between algorithms

[miq]

Hi,

I've noticed a wired phenomenon. When I set optimizations to use the SWAP algorithm for a few hours and then switch automatically to MFederov, the D-Error of even the first iteration of MFederov is always much higher than that for the last SWAP optimization. The same happens if I stop optimization and use the last design as a starting point for the new algorithm.
The differences are large, for instance:
17494 0.239894
1 0.489672 (MFederov kicks in here)

I do not understand why is this happening - it shouldn't, right? I can provide the code that generates this if necessary.
Thank you very much in advance for any suggestions...

Best regards

Mikolaj

[Andrew Collins]

Hi Mikolaj

Yes,that does sound strange. Can you send me the code please - via contact at choice-metrics.com if you don't want to post it here.

Regards
Andrew Collins

[miq]

Hi Andrew,

Here's the code:

Code: Select all


Design
; alts(A) = alt1*, alt2*, alt3*
; alts(B) = alt1*, alt2*, alt3*
; alts(C) = alt1*, alt2*, alt3*

; rows = 16
; block = 4

; bseed=179424673
; rseed=179424673

; eff = 0.33* B(mnl,d) + 0.66*C(mnl,d,median)

; alg = swap(reset=100, resetinc=100), stop=total(14400 secs)), mfederov

? ;bdraws=gauss(3)
; bdraws=sobol(5000)

;model(A):
U(alt1) = BC[0.35]*COW[0,0.1,1.5,3] + BMC[1]*MC[0,1] + BW[0.17]*WAGE[0,0.8,2,6] + BR[1]*ROAD[0,1] + BT[-0.42]*TIME[0,0.1,0.8,2.4] + BL[-0.4]*LL[0,0.1,0.2,0.4]
+ BTL[0]*TIME*LL/
U(alt2) = BC*COW + BMC*MC + BW*WAGE + BR*ROAD + BT*TIME + BL*LL + BTL*TIME*LL/
U(alt3) = BC*COW + BMC*MC + BW*WAGE + BR*ROAD + BT*TIME + BL*LL + BTL*TIME*LL

;model(B):
U(alt1) = BC[0]*COW[0,0.1,1.5,3] + BMC[0]*MC[0,1] + BW[0]*WAGE[0,0.8,2,6] + BR[0]*ROAD[0,1] + BT[0]*TIME[0,0.1,0.8,2.4] + BL[0]*LL[0,0.1,0.2,0.4] + BTL[0]*TIME*LL/
U(alt2) = BC*COW + BMC*MC + BW*WAGE + BR*ROAD + BT*TIME + BL*LL + BTL*TIME*LL/
U(alt3) = BC*COW + BMC*MC + BW*WAGE + BR*ROAD + BT*TIME + BL*LL + BTL*TIME*LL

;model(C):
U(alt1) = BC[(n,0.35,0.18)]*COW[0,0.1,1.5,3] + BMC[(n,1,0.5)]*MC[0,1] + BW[(n,0.17,0.9)]*WAGE[0,0.8,2,6] + BR[(n,1,0.5)]*ROAD[0,1] + BT[(n,-0.42,0.21)]*TIME[0,0.1,0.8,2.4] + BL[(n,-0.4,0.2)]*LL[0,0.1,0.2,0.4]
+ BTL[(n,-0.96,0.48)]*TIME*LL/
U(alt2) = BC*COW + BMC*MC + BW*WAGE + BR*ROAD + BT*TIME + BL*LL + BTL*TIME*LL/
U(alt3) = BC*COW + BMC*MC + BW*WAGE + BR*ROAD + BT*TIME + BL*LL + BTL*TIME*LL
$



Basically what matters is model B and C. The target efficiency was calculated for MNL model with 0 priors (model B; weight 33%) and MNL model with all the parameters normally distributed (model C; weight 66%) with priors of means specified as 1/maximum variable value (with the expected sign) and priors of standard deviation specified as 0.5 of the prior of mean. This was in order to assume all variables equally important for respondents and not allow any of the attributes to dominate the others, as we had no clue at this point.

I tried it with sobol, halton and mfederov bdraws. You can of course reduce the no. of bdraws and the time after which the algorithm is supposed to change to mfederov - the result is the same.

In the end I used the swap algorithm only because I did not like unbalanced number of the attributes in the design mfederov generated (for some attributes there were many highest levels of the attributes used and very few or none middle levels; perhaps if I dummy coded them it would help but it's for the pilot and a sample will be very small anyway).


One other funny thing I've noticed with this - I had a few optimizations run simultaneously for a few days with different bdraws - gauss(3), gauss(5), sobol(5000) and halton(5000). They all used the SWAP. After that time I saved the best design of each optimization and compared them with ;eval. It turned out that depending on what bdraws I specified for evaluation (I used as many as 50000 draws or gauss with 7 abscissas, so a lot) different designs had the lowest d-error. Interestingly the ranking according to sobol and halton was pretty consistent, gauss was different. I suppose that if there are many draws - the kind of draws should not influence d-error and ranking of the designs. That's why I found this result very strange. Any ideas what might be going wrong?
I can send you the designs for comparisons if you like.

Thank you very much in advance for any help/suggestions on this.

Mikolaj

[Andrew Collins]

Hi Mikolaj

The issue is that the mfederov algorithm is not actually reading in the design from the previous algorithm. Instead, it is generating a design from scratch, which is pulled out of the candidate set that the federov algorithm uses. Going the other way, from federov to the swapping algorithm, should work, as the latter will read in the design and just swap the levels in place. I will change the initialisation of the federov algorithm so that it does actually bring across the design from the previous algorithm, likely by the next point release.

If you want to make sure that the attributes aren't too unbalanced with the federov algorithm, you can specify limits how many times each attribute level appears, using round brackets as follows:
BT[0]*TIME[0,0.1,0.8,2.4](3-5,3-5,3-5,3-5)
This is documented in the manual. However, having limits that are too restrictive can lead to problems, and this is where the federov algorithm is at its weakest.

As for the issues with the draws, yes, please send through some designs by email, and I will take a look. Mike or John might have some thoughts on this issue.

Andrew

[Michiel Bliemer]

I notice that you have 7 Bayesian priors. This is quite a lot, and therefore the Bayesian D-error will take many many draws to stabilize, particularly if you use "wide" distributions. You use a normal distribution with mean 0.17 and standard deviation of 0.9 for the WAGE attribute. This is a very large standard deviation, are you sure you do not mean 0.09? Large standard deviations also make the Bayesian D-error rather unstable, so this may also resolve some of the "problems".

Michiel

[miq]

Dear Andrew and Michiel,

Thank you very much for your help and thorough explanation - all is clear now.
Indeed - the large s.d. was an error. I know 7 Bayesian priors is a lot - I hope it does not hurt as long as I increase the no of draws until the d-errors become stable, and wait longer
I am unable to reproduce different d-errors when using different evaluation methods. Most probably I was making some mistake, sorry for bothering you about this.

Thanks again,

Mikolaj

[Michiel Bliemer]

You are never bothering us, we value the feedback of power users like yourself