Uncertainty in Multiclass Classification

1. What is Uncertainty in Classification?

• Uncertainty refers to the model’s confidence or doubt in its predictions.
• Quantifying uncertainty is important to understand how reliable each prediction is.
• In multiclass classification, uncertainty estimates provide probabilities over multiple classes, reflecting how sure the model is about each possible class.

2. Methods to Estimate Uncertainty in Multiclass Classification

Most multiclass classifiers provide methods such as:

• predict_proba: Returns a probability distribution across all classes.
• decision_function: Returns scores or margins for each class (sometimes called raw or uncalibrated confidence scores).
• The probability distribution from predict_proba captures the uncertainty by assigning a probability to each class.

3. Shape and Interpretation of predict_proba in Multiclass

• Output shape: (n_samples, n_classes)
• Each row corresponds to the probabilities of all classes for a single data sample.
• Probabilities for each sample sum up to 1.
• Example:

For a 3-class problem, the output might look like:

[[0.1 0.7 0.2],

[0.8 0.1 0.1],

[0.2 0.5 0.3]]

This means the model predicts the second class with the highest certainty for the first sample, the first class for the second sample, and the second class again (but with less confidence) for the third sample.

4. Using predict_proba in Multiclass — Example on the Iris Dataset

• The Iris dataset has 3 classes.
• Using a model (e.g., logistic regression or gradient boosting), one obtains:

predicted_probabilities = model.predict_proba(X_test)

print(predicted_probabilities.shape)  # (n_samples, 3)

print(predicted_probabilities[:5])

• This tells us how confident the model is about each class for every test point.
• The highest probability in a row is usually the predicted class (via argmax).

5. Visualization of Uncertainty

• Decision boundaries around different classes can be visualized.
• Probabilities reveal “soft boundaries” and small areas of uncertainty where probabilities are similar across classes.
• Figure 2-56 demonstrates how uncertainty is visible in certain regions near the decision boundary.

6. Calibration of Multiclass Probability Estimates

• Similar to binary classification, calibration indicates how well predicted probabilities reflect actual outcomes.
• A perfectly calibrated model predicts class probabilities such that when it says “class 1 with 70% probability”, that class is indeed correct 70% of the time.
• Poor calibration may result in overconfident or underconfident probability estimates in multiclass settings.
• Calibration techniques can be applied for multiclass as well.

7. Practical Uses of Uncertainty in Multiclass

• Thresholding: In some applications, you might only classify a sample if the predicted probability for the predicted class exceeds a certain threshold.
• Reject option: Skip or ask for human review when uncertainty is high (all probabilities close to uniform).
• Active learning: Prioritize samples with high uncertainty for labeling.
• Ranking: Use probabilities to rank samples by certainty or risk.

8. Model Specific Notes

• Different models have varying quality of uncertainty estimates:
• Gradient boosting, random forests, and logistic regression often produce reasonable probability estimates.
• Fully-grown decision trees are less reliable for uncertainty due to extreme (0 or 1) predicted probabilities.
• Consider model calibration and complexity to get realistic uncertainty estimates.