Integrating and Scaling AI Features
Given a request like "add image recognition to our app," the workflow is:
- Clarify the task: classification, detection, segmentation, OCR, speech-to-text, embeddings/search, etc.
- Decide where inference runs (see decision framework below).
- Pick or build the model (pretrained first, fine-tune only if needed).
- Convert/optimize for target runtime (TFLite, Core ML, ONNX Runtime, or cloud serving stack).
- Integrate with app code and benchmark (latency, memory, battery, accuracy).
- Add fallback/monitoring for edge cases and drift.
Ask these questions in order — first "yes" usually wins:
| Question | If Yes → |
|---|---|
| Must it work offline? | On-device |
| Is latency budget < 100ms and network is unreliable? | On-device |
| Is the model large (>200MB) or compute-heavy (LLMs, video)? | Cloud |
| Does it need frequent updates without app releases? | Cloud |
| Is user privacy/data residency critical (e.g., biometric, health)? | On-device |
| Will usage scale to millions of concurrent requests? | Cloud (autoscaled) |
| Is it a hybrid case (quick on-device pre-filter, cloud for heavy lifting)? | Hybrid pipeline |
Default recommendation when unsure: start on-device with a small pretrained model; add cloud fallback for cases the on-device model can't handle confidently (confidence threshold routing).
Progress checklist for a typical AI integration task:
- Define the task and success metric (accuracy, latency, cost per request)
- Choose model source: pretrained (TF Hub, Hugging Face, Core ML models), vendor API (Vision Kit, ML Kit, Cloud Vision), or custom-trained
- Decide on-device vs cloud vs hybrid (use table above)
- Convert model to target format:
- Mobile: TensorFlow Lite (
.tflite), Core ML (.mlmodel/.mlpackage), ONNX Runtime Mobile - Cloud: TorchServe, TensorFlow Serving, Triton Inference Server, or managed (SageMaker, Vertex AI)
- Mobile: TensorFlow Lite (
- Apply optimization techniques (quantization, pruning, distillation — see below)
- Integrate into app: wrap inference in an abstraction layer (e.g.,
Classifier.predict(input) -> Result) so backend can be swapped without touching UI code - Benchmark on real target devices/hardware tiers (low-end and high-end)
- Add graceful degradation: cached results, lower-fidelity model, or "feature unavailable" state
- Instrument telemetry: inference time, failure rate, model version, confidence scores
- For cloud: set up autoscaling, load testing, and cost monitoring
- Plan for model updates (versioning, A/B testing, rollback)
Apply in this order of effort-to-benefit:
- Quantization (biggest win, least effort): convert FP32 → INT8 or FP16. Typically 2-4x smaller, 2-3x faster, small accuracy drop (~1-2%). Use post-training quantization first; use quantization-aware training if accuracy drop is unacceptable.
- Pruning: remove low-impact weights/channels. Useful when quantization alone isn't enough.
- Knowledge distillation: train a small "student" model to mimic a large "teacher" model. Use when you control training and need a much smaller footprint.
- Architecture selection: prefer mobile-first architectures (MobileNet, EfficientNet-Lite, DistilBERT) over general-purpose large models.
- Hardware delegation: use NNAPI (Android), Core ML Neural Engine (iOS), GPU delegates — offload to specialized silicon instead of CPU.
- Batch/throttle inference: don't run every frame; sample frames, debounce triggers, cache repeated inputs.
Always measure: model size (MB), cold-start load time, per-inference latency (p50/p95), memory footprint, and battery drain (mAh/hour of active use) — not just accuracy.
- Serve via dedicated inference servers (Triton, TorchServe, TF Serving) rather than embedding model loading in a general web app — enables batching, GPU sharing, and independent scaling.
- Dynamic batching: group concurrent requests to maximize GPU utilization; tune max batch size/delay against latency SLA.
- Autoscaling: scale on queue depth or GPU utilization, not just CPU — GPU-bound inference doesn't correlate with CPU metrics.
- Model warm pools: keep a minimum number of warm instances to avoid cold-start latency spikes.
- Multi-tier serving: route cheap/fast requests to smaller models or cached embeddings; route complex requests to larger models.
- Cost control: use spot/preemptible GPU instances for batch/offline inference; reserved/on-demand for real-time serving.
Example 1: Input: "Add a feature that recognizes plants from a photo, must work in remote areas with no signal." Output: On-device recommendation — use a MobileNetV3-based image classifier fine-tuned on a plant dataset, convert to TFLite with INT8 post-training quantization, deploy via ML Kit custom model or TFLite Interpreter, target <150ms inference on mid-tier phones, bundle model in app (~10-15MB) with over-the-air update capability for retraining.
Example 2: Input: "We need real-time translation for a chat app used by millions of concurrent users globally." Output: Cloud-based — deploy a transformer translation model behind Triton Inference Server with dynamic batching, autoscale on GPU utilization across multiple regions, use a CDN/edge cache for frequent phrase pairs, fall back to a smaller on-device model for offline drafts that sync once connectivity resumes (hybrid).
Example 3: Input: "Our on-device face detection model works but drains battery fast." Output: Reduce inference frequency (process every 3rd-5th frame, not every frame), switch to a quantized lighter model (e.g., BlazeFace-style), enable hardware delegate (Core ML Neural Engine / NNAPI) instead of CPU fallback, and add a motion-detection pre-filter so the model only runs when the frame changes significantly.
- Always benchmark on the lowest-spec device you intend to support, not just flagship devices.
- Abstract inference behind an interface so you can swap on-device ↔ cloud without refactoring UI/business logic.
- Version your models explicitly and log which version produced each prediction — essential for debugging regressions.
- Set confidence thresholds and define an explicit "low confidence" UX path rather than silently showing bad predictions.
- Prefer pretrained/vendor models (ML Kit, Vision framework, Cloud Vision) before building custom models — only custom-train when off-the-shelf accuracy is insufficient.
- Treat model updates like software releases: test, canary/A-B, and have a rollback plan.
- Shipping an unquantized FP32 model to mobile "because it was easier" — bloats app size and drains battery.
- Running inference on every camera frame instead of throttling/sampling.
- Ignoring cold-start latency for cloud models — first request after scale-up can be seconds slower.
- Scaling cloud inference on CPU metrics when the bottleneck is GPU.
- No offline/degraded-mode plan when the on-device model can't handle a case that needed cloud.
- Hardcoding model file paths/versions in app code instead of using a swappable abstraction — makes future migration painful.
- Skipping real hardware testing and relying only on simulator/emulator performance numbers, which don't reflect real thermal/battery behavior.