We apologize for an experimental methodology flaw of the paper "Gene selection for Cancer Classification". The problem concerns mostly Figure 4 (colon cancer) where several methods are being compared with the leave-one-out cross-validation method (LOO). Here, LOO was performed to assess the performance of the classifiers, using a fixed set of features previously selected with the whole training data set. The result tables of both Leukemia and colon cancer quote both these bad LOO results and test set results.
The proper way to conduct LOO for feature selection is to avoid using a fixed set of features selected with the whole training data set, because this induces a bias in the results. Instead, one should withhold a pattern, select features, and assess the performance of the classifier with the selected features using the left out example. One then rotates over all the examples, recomputing the feature set and the classifier parameters each time. Note that in this way, the performance of a classifier using a given number of features can be assessed, not the predictive power of a given feature subset. This later problem is better addressed using an independent test set.
Several papers have since then benchmarked RFE properly against other methods. See for instance:
Sridhar Ramaswamy et al.
Multiclass cancer diagnosis using tumor gene expression signatures.
PNAS, vol.98, No26, pp. 15149-15154, December, 2001.
B. Scholkopf, I. Guyon, and J. Weston.
Statistical learning and kernel methods in bioinformatics.
In proceedings NATO Advanced Studies Inst. on Artificial Intelligence and Heuristics Methods for Bioinformatics, San Miniato, Italy October 1-11, to appear. 2001
J. Weston, A. Elisseeff, M. Tipping and B. Schölkopf. "Use of the zero norm with linear models and
kernel methods" JMLR special Issue on Variable and Feature selection, to appear.
A. Rakotomamonjy. Variable selection using SVM-based criteria. JMLR special Issue on Variable and Feature selection, to appear.