# Metrics & Overfitting

Our machine learning workflow incorporates a robust set of metrics to evaluate model performance for both regression and classification tasks. This document provides an overview of these metrics and how they are calculated and used in our system.

<figure><img src="https://4269815422-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGc9mnST31tkU3h2NNgSL%2Fuploads%2F8j8elqs5EZsQS1KzNgDw%2Fimage.png?alt=media&#x26;token=4560cd3d-5cbc-4195-b4bb-5bdcdf617605" alt=""><figcaption></figcaption></figure>

The `_calculate_metrics` method is the core function responsible for computing various performance metrics. It handles both regression and classification tasks, as well as multi-output scenarios.

***

## <mark style="color:blue;">Metrics Calculation Process</mark>

1. **Data Preparation**:
   * Ensures all inputs (y\_true, y\_pred, y\_train, y\_train\_pred) are 2D numpy arrays.
   * Adjusts prediction arrays to match the shape of true values.
2. **Target-specific Metrics**:
   * Iterates through each target, calculating metrics based on the target type (categorical or numeric).
3. **Overall Metrics**:
   * Computes average metrics across all targets.
4. **Overfitting Detection**:
   * Calculates metrics to detect and quantify potential overfitting.

***

## <mark style="color:blue;">Classification Metrics</mark>

For categorical targets, the following metrics are calculated:

1. **Accuracy**:
   * Ratio of correct predictions to total predictions.
   * Calculated using `accuracy_score` from scikit-learn.
2. **Precision**:
   * Ratio of true positive predictions to total positive predictions.
   * Calculated using `precision_score` with weighted average.
3. **Recall**:
   * Ratio of true positive predictions to total actual positives.
   * Calculated using `recall_score` with weighted average.
4. **F1 Score**:
   * Harmonic mean of precision and recall.
   * Calculated using `f1_score` with weighted average.

***

## <mark style="color:blue;">Regression Metrics</mark>

For numeric targets, the following metrics are calculated:

1. **Mean Absolute Error (MAE)**:
   * Average absolute difference between predicted and actual values.
   * Calculated using `mean_absolute_error` from scikit-learn.
2. **Mean Squared Error (MSE)**:
   * Average squared difference between predicted and actual values.
   * Calculated using `mean_squared_error` from scikit-learn.
3. **R-squared (R2) Score**:
   * Proportion of variance in the dependent variable predictable from the independent variable(s).
   * Calculated using `r2_score` from scikit-learn.

***

## <mark style="color:blue;">Overfitting Detection</mark>

To detect and quantify overfitting, we calculate:

1. **Performance Difference**:
   * Difference between average train performance and average validation performance.
2. **Performance Ratio**:
   * Ratio of average train performance to average validation performance.
3. **Overfitting Flag**:
   * Set to `True` if performance difference > 0.15 or performance ratio > 1.3.
4. **Overfitting Score**:
   * Maximum of performance difference and (performance ratio - 1).