Predicting Probabilities

1. What is Predicting Probabilities?

The predict_proba method estimates the probability that a given input belongs to each class.
It returns values in the range [0, 1], representing the model's confidence as probabilities.
The sum of predicted probabilities across all classes for a sample is always 1 (i.e., they form a valid probability distribution).

For binary classification, the shape of the output is (n_samples, 2):
Column 0: Probability of the sample belonging to the negative class.
Column 1: Probability of the sample belonging to the positive class.
For multiclass classification, the shape is (n_samples, n_classes), with each column corresponding to the probability of the sample belonging to that class.

The probability reflects how confidently the model believes a data point belongs to each class.
For example, in binary classification:

Input Sample	Predicted Probability (Negative Class)	Predicted Probability (Positive Class)
1	0.2	0.8
2	0.9	0.1

The model predicts positive class if the positive class probability is greater than a threshold (default 0.5).

The default threshold for making classification decisions is 0.5:
If predict_proba for positive class > 0.5, sample is classified as positive.
Otherwise, it is classified as negative.
You can adjust this threshold depending on the problem, which affects false positive and false negative rates.
Adjusting thresholds can optimize metrics like precision, recall, F-score, especially on imbalanced datasets.

Not all models produce well-calibrated probabilities.
A calibrated model outputs probabilities that closely match true likelihoods.
Example of a poor calibration: a decision tree grown to full depth might assign probability 1 or 0, but be often wrong.
Calibration can be improved using methods like:
Platt scaling
Isotonic regression
Reference: Paper by Niculescu-Mizil and Caruana, “Predicting Good Probabilities with Supervised Learning”.

# Suppose gbrt is a trained GradientBoostingClassifier

print("Shape of probabilities:", gbrt.predict_proba(X_test).shape)

# Output:

# Shape of probabilities: (n_samples, 2)

print("Predicted probabilities:\n", gbrt.predict_proba(X_test[:6]))

[[0.1 0.9]

[0.8 0.2]

[0.7 0.3]

...

The first column corresponds to the first class probability, the second column to the second class.

Provides interpretable uncertainty estimates in terms of probabilities.
Useful for decision making where probabilistic thresholds are preferable to hard decisions.
Can be integrated into pipelines that weigh risks (e.g., medical diagnosis, fraud detection).
Helps in ranking samples by probability to prioritize further analysis.

Some classifiers implement both decision_function and predict_proba:
decision_function returns raw scores or margins.
predict_proba converts these scores to probabilities.
Probabilities are usually obtained by applying a logistic function or softmax on the decision function scores.
Calibrated models provide better probability estimates compared to raw scores alone,.

When probabilities are needed (e.g., for risk assessment), prefer models supporting predict_proba.
Be cautious that probabilities are only as good as model calibration.
Always validate probabilities with calibration plots or metrics like Brier score.

Aspect	Details
Purpose	Provides class membership probabilities
Output Shape	Binary: (n_samples, 2), Multiclass: (n_samples, n_classes)
Values	Probabilities between 0 and 1, sum to 1 per sample
Default threshold	0.5 for binary classification
Calibration	Models may need calibration for accurate probabilities
Applications	Threshold tuning, risk assessment, ranking predictions
Relation	Derived from decision_function scores via logistic or softmax
Example Models	GradientBoostingClassifier, Logistic Regression, Random Forest