Publications

Our recent paper answers precisely this question. The paper is

B. Zadrozny and C. Elkan. "Learning and Making Decisions When Costs and Probabilities are Both Unknown". Technical Report No. CS2001-0664, January 2001, UCSD.

available at www-cse.ucsd.edu/users/elkan/

Abstract: In many machine learning domains, misclassification costs are different for different examples, in the same way that class membership probabilities are example-dependent. In these domains, both costs and probabilities are unknown for test examples, so estimators for both must be learned. This paper first reviews how to make optimal decisions given cost and probability estimates. We then present new methods that learn decision trees that give well-calibrated probability estimates from highly unbalanced datasets where one class is rare. Next we explain how to obtain unbiased estimators for example-dependent costs, taking into account the difficulty that in general, probabilities and costs are not independent random variables, and the training examples for which costs are known are not representative of all examples. The latter problem is called sample selection bias in econometrics. Our solution to it is based on Nobel prize-winning work due to the economist James Heckman. We show that the methods we propose are successful in a comprehensive comparison with MetaCost that uses the well-known large and difficult dataset from the KDD'98 data mining contest.