Model Retraining: Why & How to Retrain ML Models? [2023]
Hello friend! Machine learning models have become deeply ingrained in our lives, powering everything from product recommendations to financial fraud detection. However, their predictive capabilities decay over time without proper maintenance. In this guide, we will explore proven techniques for retraining models to keep them accurate for the long haul.
Why Retrain? Causes of Model Drift
Once deployed, models make predictions on new, live data. However, this production data often gradually diverges from the original training data used to develop the model. There are two common root causes for this divergence, known as model drift:
Concept drift occurs when the statistical relationship between input features and the target variable changes over time. For example, economic factors that influence loan default rates can shift due to a recession.
Data drift occurs when the distribution of input features changes but the underlying concept remains stable. For example, customer purchasing habits can change as new products are introduced.
These changes in data patterns lead to accuracy degradation – the model makes increasingly worse predictions on the new data compared to its performance on the original training data.
Without retraining, model accuracy decays as data drifts farther from original training data
In one study, machine learning models saw a 5-20% drop in accuracy within the first 6 months after deployment due to various drift types. For mission-critical applications, even small amounts of drift can be highly problematic.
When Should You Retrain?
To counteract accuracy decay from drift, models need to be periodically retrained on new data representative of current realities. There are two main approaches for determining optimal retraining frequency:
Periodic retraining involves refreshing the model on a fixed schedule – e.g. every 1-2 weeks or 1-3 months. This methodology is straightforward to implement but can result in unnecessary retraining if the data is stable.
Trigger-based retraining sets specific accuracy or performance thresholds, automatically retraining the model only when metrics violate the defined threshold. This adaptive approach prevents wasteful retraining but requires accurate monitoring capabilities.
The right retraining frequency depends on several factors:
- Data velocity – How quickly new training data becomes available. Higher velocity necessitates more frequent retraining.
- Model criticality – For mission-critical models, even small drift may warrant immediate retraining.
- Resource constraints – Retraining large models on new data can require significant compute resources.
| Model Criticality | Data Velocity | Recommended Frequency |
|---|---|---|
| Low | Low | Every 6-12 months |
| Moderate | Moderate | Monthly |
| High | High | Weekly |
For many real-world systems, monthly or quarterly retraining is sufficient. For mission-critical models on rapidly changing data, weekly or even daily retraining may be warranted.
How Should You Retrain Models?
When retraining, you need to decide how much new data to use and how to combine it with previously seen data. Common retraining techniques include:
Batch retraining – Train the model from scratch exclusively on all new data collected since the prior retraining run. This approach maximizes the influence of latest data but can disregard potentially useful past data.
Online learning – Incrementally train model on new data only, typically weighting it higher than past observations. Computationally cheaper but may overfit to peculiarities of recent data.
Hybrid – Blend old and new data, with influence of older data decaying gradually over time via exponential weighting or Windowing. Balances old knowledge with new trends.
In hybrid retraining, recent data has higher influence without fully discarding old data
Technical considerations for retraining process include:
- Retrain full model or only later layers?
- How to track and manage model versions?
- When to replace with entirely new model?
- Monitoring data quality and hidden feedback loops
Operationalizing Retraining Pipelines
To scale retraining and make it sustainable, it is critical to embed it within a MLOps framework for continuous development and monitoring of production models. Key requirements include:
Automated regression testing to detect accuracy drops or data mismatches. Tests can run on schedule or trigger on dataset changes.
CI/CD pipelines that rebuild models on a fixed schedule or other triggering events based on DevOps best practices.
Central model management with tools like MLflow to version control and track retraining operations.
Scalable compute resources such as GPU clusters to run retraining workloads without compromising production systems.
Monitoring and alerts to track model performance KPIs and data quality metrics, notifying teams of significant deviations.
With the right MLOps architecture powered by tools like Kubernetes and Tensorflow Extended, retraining can happen reliably and efficiently at scale.
Case Study: Retraining a Recommender System
A major e-commerce site noticed declining performance of their personalized product recommendation model. Research showed user preferences had gradually drifted from the initial training data. They operationalized a retraining pipeline, rebuilding the model weekly using the latest 60 days of user interaction data. Retraining improved overall clickthrough rate (CTR) by 12% and conversion rate by 8%, with significant ROI gains.
Key Takeaways
Regular model retraining is essential to maintaining accuracy as data evolves. By identifying causes of drift, optimizing retraining frequency, leveraging MLOps pipelines, and continuously integrating new data, you can keep models relevant for the long term. Retraining transforms models into continuously adaptive systems capable of responding to today‘s business needs.
What questions do you have about productionizing retraining processes? I‘m happy to provide additional tips and advice – please feel free to ask!
