Skip to main content

Uncertainty in Multiclass Classification

Dr. Rishabh Pathak
1. What is Uncertainty in Classification? Uncertainty refers to the model’s confidence or doubt in its predictions. Quantifying uncertainty is important to understand how reliable each prediction is. In multiclass classification , uncertainty estimates provide probabilities over multiple classes, reflecting how sure the model is about each possible class. 2. Methods to Estimate Uncertainty in Multiclass Classification Most multiclass classifiers provide methods such as: predict_proba: Returns a probability distribution across all classes. decision_function: Returns scores or margins for each class (sometimes called raw or uncalibrated confidence scores). The probability distribution from predict_proba captures the uncertainty by assigning a probability to each class. 3. Shape and Interpretation of predict_proba in Multiclass Output shape: (n_samples, n_classes) Each row corresponds to the probabilities of ...
Post a Comment

Decision Trees

Image
Dr. Rishabh Pathak

1. What are Decision Trees?

Decision trees are supervised learning models used for classification and regression tasks.

  • They model decisions as a tree structure, where each internal node corresponds to a decision (usually a test on a feature), and each leaf node corresponds to an output label or value.
  • Essentially, the tree learns a hierarchy of if/else questions that partition the input space into regions associated with specific outputs.

2. How Decision Trees Work

  • The model splits the dataset based on feature values in a way that increases the purity of the partitions (i.e., groups that are more homogeneous with respect to the target).
  • At each node, the algorithm evaluates possible splits on features and selects the one that best separates the data, according to a criterion such as Gini impurity, entropy (information gain), or mean squared error (for regression).
  • The process recursively continues splitting subsets until a stopping criterion is met (e.g., maximum depth, minimum samples per leaf).

Example analogy from the book:

·         To distinguish animals like bears, hawks, penguins, and dolphins, decision trees ask questions like “Does the animal have feathers?” to split the dataset into smaller groups, continuing with further specific questions.

·         Such questions form a tree structure where navigating from the root to a leaf corresponds to a series of questions and answers, leading to a classification decision,.

3. Advantages of Decision Trees

  • Easy to understand and visualize: The flow of decisions can be depicted as a tree, which is interpretable even for non-experts (especially for small trees).
  • No need for feature scaling: Decision trees are invariant to scaling or normalization since splits are based on thresholds on feature values and not on distances.
  • Handles both numerical and categorical data: Trees can work with a mix of continuous, ordinal, and categorical features without special preprocessing.
  • Automatic feature selection: Only relevant features are used for splits, providing a form of feature selection.

4. Weaknesses of Decision Trees

  • Tendency to overfit: Decision trees can create very complex trees fitting the noise in training data, leading to poor generalization performance.
  • Unstable: Small variations in data can lead to very different trees.
  • Greedy splits: Recursive partitioning is greedy and locally optimal but not guaranteed to find the best overall tree.

Due to these issues, single decision trees are often outperformed by ensemble methods like random forests and gradient-boosted trees,.

5. Parameters and Tuning

Key parameters controlling decision tree construction:

  • max_depth: Maximum depth of the tree. Limiting depth controls overfitting.
  • min_samples_split: Minimum number of samples required to split a node.
  • min_samples_leaf: Minimum number of samples required to be at a leaf node.
  • max_features: The number of features to consider when looking for the best split.
  • criterion: The function to measure split quality, e.g. "gini" or "entropy" for classification, "mse" for regression.

Proper tuning of these parameters helps optimize the balance between underfitting and overfitting.

6. Extensions: Ensembles of Decision Trees

To overcome the limitations of single trees, ensemble methods combine multiple trees for better performance and stability:

  • Random Forests: Build many decision trees on bootstrap samples of data and average the results, injecting randomness by limiting features for splits to reduce overfitting.
  • Gradient Boosted Decision Trees: Sequentially build trees that correct errors of previous ones, resulting in often more accurate but slower-to-train models.

Both approaches maintain some advantages of trees (e.g., no need for scaling, interpretability of base learners) while significantly enhancing performance.

7. Visualization of Decision Trees

  • Because the model structure corresponds directly to human-understandable decisions, decision trees can be visualized as flowcharts.
  • Visualization aids in understanding model decisions and debugging.

8. Summary

Aspect

Description

Model Type

Hierarchical if/else decision rules forming a tree

Tasks

Classification and regression

Strengths

Interpretable, no scaling needed, handles mixed data

Weaknesses

Prone to overfitting, unstable with small changes

Key Parameters

max_depth, min_samples_split, criterion, max_features

Use in Ensembles

Building block for robust models like Random Forests and Gradient Boosted Trees
Categories:

Comments

Post a Comment

Popular posts from this blog

Relation of Model Complexity to Dataset Size

Dr. Rishabh Pathak
Core Concept The relationship between model complexity and dataset size is fundamental in supervised learning, affecting how well a model can learn and generalize. Model complexity refers to the capacity or flexibility of the model to fit a wide variety of functions. Dataset size refers to the number and diversity of training samples available for learning. Key Points 1. Larger Datasets Allow for More Complex Models When your dataset contains more varied data points , you can afford to use more complex models without overfitting. More data points mean more information and variety, enabling the model to learn detailed patterns without fitting noise. Quote from the book: "Relation of Model Complexity to Dataset Size. It’s important to note that model complexity is intimately tied to the variation of inputs contained in your training dataset: the larger variety of data points your dataset contains, the more complex a model you can use without overfitting....
Post a Comment
Read more

Linear Models

Dr. Rishabh Pathak
1. What are Linear Models? Linear models are a class of models that make predictions using a linear function of the input features. The prediction is computed as a weighted sum of the input features plus a bias term. They have been extensively studied over more than a century and remain widely used due to their simplicity, interpretability, and effectiveness in many scenarios. 2. Mathematical Formulation For regression , the general form of a linear model's prediction is: y^ ​ = w0 ​ x0 ​ + w1 ​ x1 ​ + … + wp ​ xp ​ + b where; y^ ​ is the predicted output, xi ​ is the i-th input feature, wi ​ is the learned weight coefficient for feature xi ​ , b is the intercept (bias term), p is the number of features. In vector form: y^ ​ = wTx + b where w = ( w0 ​ , w1 ​ , ... , wp ​ ) and x = ( x0 ​ , x1 ​ , ... , xp ​ ) . 3. Interpretation and Intuition The prediction is a linear combination of features — each feature contributes prop...
Post a Comment
Read more

Predicting Probabilities

Dr. Rishabh Pathak
1. What is Predicting Probabilities? The predict_proba method estimates the probability that a given input belongs to each class. It returns values in the range [0, 1] , representing the model's confidence as probabilities. The sum of predicted probabilities across all classes for a sample is always 1 (i.e., they form a valid probability distribution). 2. Output Shape of predict_proba For binary classification , the shape of the output is (n_samples, 2) : Column 0: Probability of the sample belonging to the negative class. Column 1: Probability of the sample belonging to the positive class. For multiclass classification , the shape is (n_samples, n_classes) , with each column corresponding to the probability of the sample belonging to that class. 3. Interpretation of predict_proba Output The probability reflects how confidently the model believes a data point belongs to each class. For example, in ...
Post a Comment
Read more

Kernelized Support Vector Machines

Dr. Rishabh Pathak
1. Introduction to SVMs Support Vector Machines (SVMs) are supervised learning algorithms primarily used for classification (and regression with SVR). They aim to find the optimal separating hyperplane that maximizes the margin between classes for linearly separable data. Basic (linear) SVMs operate in the original feature space, producing linear decision boundaries. 2. Limitations of Linear SVMs Linear SVMs have limited flexibility as their decision boundaries are hyperplanes. Many real-world problems require more complex, non-linear decision boundaries that linear SVM cannot provide. 3. Kernel Trick: Overcoming Non-linearity To allow non-linear decision boundaries, SVMs exploit the kernel trick . The kernel trick implicitly maps input data into a higher-dimensional feature space where linear separation might be possible, without explicitly performing the costly mapping . How the Kernel Trick Works: Instead of computing ...
Post a Comment
Read more

Supervised Learning

Dr. Rishabh Pathak
What is Supervised Learning? ·     Definition: Supervised learning involves training a model on a labeled dataset, where the input data (features) are paired with the correct output (labels). The model learns to map inputs to outputs and can predict labels for unseen input data. ·     Goal: To learn a function that generalizes well from training data to accurately predict labels for new data. ·          Types: ·          Classification: Predicting categorical labels (e.g., classifying iris flowers into species). ·          Regression: Predicting continuous values (e.g., predicting house prices). Key Concepts: ·     Generalization: The ability of a model to perform well on previously unseen data, not just the training data. ·         Overfitting and Underfitting: ·    ...
Post a Comment
Read more