Engineering ML Model Lifecycle
Given an ML initiative, work through four phases in order:
- Define & prep: SMART objective + data acquisition + cleaning/EDA
- Experiment: feature engineering + model selection + iterative training
- Productionize: containerize + build pipelines + deploy with autoscaling
- Monitor: track drift, latency, and performance post-deployment
Example objective statement:
"Reduce customer churn prediction latency to <200ms while maintaining >85% recall, deployed to production within 6 weeks, serving 10K requests/day."
Progress:
- Step 1: Define SMART objective
- Step 2: Acquire and inventory raw data sources
- Step 3: Clean, normalize, handle missing values, run EDA
- Step 4: Engineer and select features
- Step 5: Select model algorithm(s)
- Step 6: Split data (train/val/test), train iteratively, tune hyperparameters
- Step 7: Containerize the trained model
- Step 8: Build training/inference pipelines, store artifacts
- Step 9: Deploy with autoscaling (e.g., Vertex AI, SageMaker, KServe)
- Step 10: Set up continuous monitoring (drift, performance, cost)
Step 1: Problem Definition
Write a SMART objective — Specific, Measurable, Achievable, Relevant, Time-bound. Reject vague goals like "improve predictions with ML." Tie every objective to a business metric (revenue, cost, latency, risk).
Step 2-3: Data Readiness
- Inventory sources: databases, event streams, IoT, third-party APIs, logs.
- Assess volume, freshness, and access permissions before committing to a data strategy.
- Cleaning checklist: deduplicate, handle nulls (impute/drop/flag), fix schema drift, detect outliers.
- EDA: distribution plots, correlation matrix, target leakage check, class imbalance check.
- Gate: do not proceed to feature engineering until data quality issues are documented and either fixed or explicitly accepted as risk.
Step 4-6: Experimentation
- Feature engineering: encode categoricals, scale numerics, create interaction/time-window features, drop leaky features.
- Model selection: start with a simple baseline (logistic regression, gradient boosted trees) before jumping to deep learning. Justify complexity with measured lift.
- Split data with stratification if classes are imbalanced; hold out test set untouched until final evaluation.
- Tune hyperparameters via grid/random search or Bayesian optimization; track every run (params, metrics, data version) in an experiment tracker (MLflow, Vertex Experiments, W&B).
Step 7-9: Productionization
- Containerize: package model + dependencies + inference code into a Docker image. Pin versions. Test the container locally with a sample payload before pushing.
- Pipeline: automate training as a repeatable pipeline (Vertex AI Pipelines, Kubeflow, Airflow) — not a one-off notebook run. Store model artifacts + metadata in versioned cloud storage (GCS, S3).
- Serve: deploy to a managed endpoint (Vertex AI Endpoints, SageMaker Endpoints) with autoscaling configured on request volume/latency thresholds. Define min/max replica counts to control cost vs. availability.
Step 10: Continuous Monitoring
Track in production:
- Data drift: input feature distributions vs. training baseline.
- Concept drift: prediction/label distribution shift over time.
- Performance decay: accuracy/precision/recall against ground truth as it arrives.
- Operational health: latency, error rate, throughput, cost per prediction.
- Set alert thresholds and a retraining trigger (scheduled, or drift-triggered).
Example 1: Input: "We want to use ML to detect fraudulent transactions." Output: SMART objective — "Reduce fraud losses by 20% within Q3 by deploying a real-time transaction classifier with <100ms inference latency and false-positive rate under 2%, using the last 18 months of transaction data." Followed by data acquisition plan (transaction DB, chargeback logs), EDA focus on class imbalance (fraud is rare — plan for SMOTE or class weighting), and baseline model = gradient boosted trees before considering neural nets.
Example 2: Input: "Model works great in the notebook but deployment keeps breaking." Output: Root cause is skipping containerization/pipeline steps. Fix: wrap the trained model + preprocessing code in a Docker image with pinned dependency versions, test it with sample requests locally, then push through an automated training pipeline that version-stamps artifacts to cloud storage before deploying to a managed endpoint.
- Treat data quality gates as blocking — a great model on bad data is worse than a mediocre model on clean data.
- Always start with a simple baseline model to establish a performance floor.
- Version everything: data, features, code, model artifacts, and hyperparameters.
- Automate training as a pipeline early — don't wait until "we need to retrain" to build it.
- Configure autoscaling with both floor (avoid cold starts) and ceiling (cost control).
- Monitoring is not optional — deploy dashboards/alerts on day one of production, not after the first incident.
- Skipping SMART objective definition and jumping straight to modeling — leads to unmeasurable "success."
- Doing EDA after model training instead of before — hides data quality issues until they cause silent failures.
- Evaluating only on validation set and never touching a truly held-out test set.
- Deploying a model directly from a notebook without containerization — "works on my machine" failures in production.
- No drift monitoring — model performance silently degrades until a business metric alarms fire.
- Overengineering model choice (jumping to deep learning) before proving a simple baseline can't meet the objective.