The Costs of Misclassifications

By Charles Harrison, Salford Systems.

Did you know some hospitals use robots to clean hospital rooms to reduce the risk of infection? I read an article online about this the other day and I started to think about the consequences of medical misdiagnoses. What happens if someone has an infection and is told they do not by their doctor? What if someone does not have an infection but is treated as if they do? Which mistake would be worse?

To answer these questions it helps to think in terms of the cost of a mistake and to do this let’s look at two hypothetical classification models and their confusion matrices:

A confusion matrix has a simple interpretation. The one for Model A says 630 people were correctly predicted to not have an infection (True Negatives), 150 people were correctly predicted to have an infection (True Positives), 170 people were incorrectly predicted to not have an infection (False Negatives), and 50 people were incorrectly predicted to have an infection (False Positives).

Model A and B behave differently. Model A favors false negatives (170; lower left corner) whereas Model B favors false positives (200; upper right corner), and Model A’s misclassification rate is lower (22% vs. 27%). Is Model A superior to Model B? In terms of the misclassification rate it is, but misclassification rate does not take the costs of a mistake (false positive or false negative) into account.

Considering Costs

One obvious way to views costs is in a monetary sense. Patients predicted to have an infection are tested by the hospital for infections including the type. Those that test positive for infections are moved to a new room, treated for their infection, and their old room is sanitized with the cost being:

1. Tests for an infection: $2,000
2. Sanitizing a room and moving a patient to a new one: $5,000
3. Treating an infection early: $20,000
4. Treating an infection later: $30,000

This information defines a cost matrix so that accounts for the severity of false positives and false negatives.

My rationale for this matrix was

1. If a patient is predicted to not have an infection and truly does not then there is no cost (True Negative)
2. If a patient is predicted to have an infection and does not then the cost is the $2,000 for the tests (False Positive). Also, for simplicity I assumed the tests run by the hospital were always correct.
3. If a patient is predicted to not have an infection but does then the hospital spends $2,000 for the tests to determine the type of infection, $30,000 to treat the infection due to the later discovery, and another $5,000 to clean the room and move the patient for a total of $37,000 (False Negative)
4. If a patient is predicted to have an infection and does, then the hospital spends $2,000 to run the tests to determine the type of infection, $20,000 to treat the infection due to the early discovery, and $5,000 to clean the room and move the patient for a total of $27,000 (True Positive)

Another way to view the costs of an incorrect prediction is in terms of the costs that the hospital will always incur:

1. If a patient has an infection (i.e. Actual = “Infection”; second row in the cost matrix) then the hospital has to treat them (regardless of what the model predicts) and always spends at least $27,000 ($27,000 if the model correctly predicts a disease and $37,000 if it does not)
2. Similarly, if a patient does not have an infection (i.e. Actual = “No infection”; first row in the cost matrix) then the hospital does not need to treat them and thus will always spend at least $0 ($0 if the model correctly predicts no infection and $2,000 if incorrectly predicts an infection)

The minimum cost for patients with an infection is $27,000 and $0 for those without, and I subtracted the minimum cost of $27,000 from the values in the bottom row (Actual = “Infection”) and $0 from the values in the top row (Actual= “No Infection; the top row does not change). My new cost matrix, which is equivalent to my first, is:

This matrix has a valuable interpretation. If the model predicts a false negative (lower left corner) it costs $10,000 more than it would otherwise cost to treat a patient whereas if the model predicts a false positive (upper right corner) then it costs $2,000 more than it would otherwise.

Using the Cost Matrix

A cost matrix can be used to evaluate the cost of a model (or with the model’s loss function to build entirely new models that minimize cost instead of purely model error). Model cost is computed by multiplying the number of false positives by $2,000 and the number of false negatives by $10,000. The final cost values for each model represent the cost incurred beyond what it normally costs to treat patients for an infection:

Model A: 170 * $10,000 + 50 * $2,000 = $1,800,000

Model B:   70 * $10,000 + 200*$2,000 = $1,100,000

Model B is superior because its predictions result in a lower total cost even though Model A has a superior misclassification rate. Model A classifies more patients correctly overall, but predicts more costly false negatives.

What about the patients?

Model B results in a lower overall cost, but what about the patient? In this example a false negative is the most expensive option and is also very damaging to patients because allowing infections to go untreated can be fatal or have severe side effects. If a situation arises where minimizing costs have detrimental effects on patient health then the cost matrix should be adjusted to reflect this. For instance, if the treatment for an infection is dangerous, then false positives would be more serious (perhaps even more serious than false negatives) and the cost matrix would have to be updated to reflect this. Numerical values may not be available to account for potential treatment risk to patients, so experimenting with different values in the cost matrix may be necessary.

Different mistakes have different costs and recognizing that a model should incorporate asymmetric costs results in improved performance that incorporates practical considerations. Costs are an important concept in classification models and should always be considered in any analysis.

Original. Reposted with permission.

Bio: Charles Harrison is a Marketing Statistician at Salford Systems. In this role he uses his statistical modelling and computer programming background to illustrate the practical techniques and uses of Salford Software in real-world applications in tandem with our in-house marketing team.

Related:

• The Best Metric to Measure Accuracy of Classification Models
• Artificial Intelligence Classification Matrix
• A Primer on Logistic Regression – Part I