Worse design properties with lower d-error?

This forum is for posts covering broader stated choice experimental design issues.

Moderators: Andrew Collins, Michiel Bliemer, johnr

Worse design properties with lower d-error?

Postby rich_imr » Wed Apr 11, 2018 11:52 pm

Greetings,

I created a simple design with 13 exercises/rows using the Federov algorithm that had a d-error of 0.22669. I then tried to create a design with the same number of rows from the same candidate search space using a genetic algorithm (in R). The resulting design had a d-error of 0.22433 with the same number of exercises.

The Federov optimized design has slightly better level co-occurrence balance than the genetic algorithm optimized design and one slightly lower correlation between two variables.

Why would a design with lower d-error have worse design properties (even though just barely) than a design with a higher d-error?

Just in case anyone is interested, I've included my code to create a d-optimal design using a genetic algorithm in R below:
Code: Select all
### Load GA
install.packages("genalg")
library(genalg)

### Defind the evaluation function
evalFunc <- function(x) {
  search.space = expand.grid(F1 = c("A", "B", "C", "D"),
                             F2 = c("I", "II", "III"),
                             F3 = c("Low", "High"),
                             F4 = c("Yes", "No"))
 
  current_design <- search.space[x==1,]
 
  ### Calc D-error
  current_design.EC <- data.frame(model.matrix(~., current_design)) #include intercept
  current_design.NU <- as.matrix(sapply(current_design.EC, as.numeric)) #numeric matrix
  derror <- det(t(current_design.NU)%*%current_design.NU)^(1/ncol(current_design.NU))/(nrow(current_design.NU)) #D-error
 
  if (sum(x) != 13)
    return(0) else return(-derror)
 
}

### Next, we choose the number of iterations, design and run the model
GAmodel <- rbga.bin(size =48, popSize = 200, iters = 100, mutationChance = 0.01, elitism = T, evalFunc = evalFunc)
# Notice that within the settings of genalg, the type is a binary chromosome by default
plot(GAmodel)
summary(GAmodel, echo=T)

### Best solution from GA;  d-error 0.2243314
chromosome = c(0,   0,   1,   1,   0,   0,   0,   0,   0,   1,   0,   0,
               0,   1,   0,   0,   1,   0,   0,   1,   0,   0,   1,   0,
               1,   0,   0,   0,   0,   1,   0,   0,   1,   0,   0,   0,
               0,   1,   0,   0,   0,   0,   1,   0,   0,   0,   0,   1)
dopt <- search.space[chromosome==1,]

### Calc D-error by hand
dopt.EC <- data.frame(model.matrix(~., dopt)) #includes intercept
dopt.EC <-as.matrix(sapply(dopt.EC, as.numeric))
det(t(dopt.EC)%*%dopt.EC)^(1/ncol(dopt.EC))/(nrow(dopt.EC))
rich_imr
 
Posts: 12
Joined: Wed Oct 21, 2015 2:52 am

Re: Worse design properties with lower d-error?

Postby Michiel Bliemer » Thu Apr 12, 2018 9:38 am

Great work on the genetic algorithm, we have also used genetic algorithms in the past to generate designs (and is implemented in one of our research versions) and it usually performs pretty good.

To answer your question, the D-error is not related to attribute level balance, and the D-error is only indirectly related to correlations between variables. Note that attribute level balance is a restriction on the design, designs with the lowest D-errors are generally not balanced. Further, correlations can be good for capturing information. Orthogonal designs with zero correlation do not necessarily capture most information (which is what D-efficient designs aim to do).

Efficiency, orthogonality, balance, they are all different properties of a design and they can all move in different direction. The question is what you care about most, and how you weight all these different properties.

Michiel
Michiel Bliemer
 
Posts: 1705
Joined: Tue Mar 31, 2009 4:13 pm

Re: Worse design properties with lower d-error?

Postby rich_imr » Fri Apr 13, 2018 12:28 am

Greetings Michiel,

Thank you very much for the reply.

Regards,

Richard
rich_imr
 
Posts: 12
Joined: Wed Oct 21, 2015 2:52 am


Return to Choice experiments - general

Who is online

Users browsing this forum: No registered users and 3 guests

cron