Integrating and Scaling AI Features
Decide where inference should run before picking a model or SDK:
| Signal | Choose |
|---|---|
| Needs offline support, low latency (<100ms), privacy-sensitive data | On-device |
| Model is large (>500MB), needs frequent updates, or requires heavy compute (LLMs, video) | Cloud |
| High request volume from many users, bursty traffic | Cloud with autoscaling |
| Simple classification/detection on limited input | On-device |
Minimal on-device integration (mobile, e.g. TFLite/Core ML):
1. Convert model to device format (.tflite, .mlmodel, ONNX)
2. Quantize (float32 → int8/float16)
3. Bundle model in app or download-on-first-launch
4. Run inference on a background thread; never block UI thread
5. Benchmark: latency, memory, battery drain on low-end device
Minimal cloud serving setup:
1. Containerize model behind an inference server (TorchServe, Triton, or managed endpoint)
2. Put behind autoscaling load balancer (min instances > 0 to avoid cold starts)
3. Add request batching for throughput
4. Set p99 latency SLO and alert on breach
5. Version the model endpoint (v1, v2) for safe rollout
Progress checklist for adding an AI feature end-to-end:
- Define the task precisely (classification, detection, generation, transcription, etc.) and success metric (accuracy, latency, cost)
- Choose pre-trained model vs. custom-trained vs. fine-tuned
- Decide on-device vs. cloud (use table above)
- If on-device: convert, quantize, benchmark on target hardware tiers (low/mid/high-end)
- If cloud: containerize, deploy behind autoscaler, load test to expected peak
- Build the app-side integration layer (abstract the model behind an interface so backend can swap without app changes)
- Add graceful degradation (fallback to cloud if on-device fails, or cached/default behavior if offline and cloud-only)
- Instrument telemetry: inference latency, error rate, model version, device type
- Roll out behind a feature flag; monitor before full release
- Set up a retraining/update pipeline if model needs to improve over time
Example 1: On-device image recognition Input: App needs to identify plant species from camera photo, must work offline in remote areas. Output: MobileNetV3 fine-tuned on plant dataset → converted to TFLite → quantized to int8 (75% size reduction) → bundled in app (18MB) → inference on background thread using NNAPI/Core ML delegate for hardware acceleration → 40ms average inference on mid-range device → fallback UI shows "low confidence" below 70% threshold instead of guessing.
Example 2: Cloud-scale speech transcription Input: App feature transcribes voice memos for millions of users, spiky usage during commute hours. Output: Whisper-based model served via Triton Inference Server on GPU instances → Kubernetes HPA (Horizontal Pod Autoscaler) scaling on GPU utilization → request batching (batch size 8) to maximize GPU throughput → results cached by audio hash to avoid re-processing duplicates → p99 latency SLO of 3s, alerting via Prometheus.
Example 3: Hybrid approach Input: Real-time object detection for AR app, but also needs a "search similar items" cloud feature. Output: Lightweight YOLO-tiny variant on-device for real-time bounding boxes (30fps target) → cropped region sent to cloud embedding model only when user taps "search" → cloud does vector similarity search against product catalog → keeps UI responsive while offloading heavy retrieval work.
- Always quantize before shipping on-device. Int8 or float16 quantization usually costs <2% accuracy for 2-4x speed/size improvement — measure the actual delta, don't assume.
- Benchmark on the lowest-supported device, not the dev's flagship phone. Battery and thermal throttling behavior differs wildly.
- Abstract the inference call behind an interface (
predict(input) -> output) so you can swap on-device/cloud/model-version without touching UI code. - Version models explicitly and log which version produced each prediction — essential for debugging regressions.
- Warm up cloud autoscalers with a minimum instance count if cold-start latency matters; pure scale-to-zero often fails real-time UX requirements.
- Batch requests server-side when possible — GPU throughput gains from batching often outweigh the added few-ms latency.
- Monitor model drift, not just uptime — accuracy silently degrading on new data distributions is the most common production failure.
- Prefer pre-trained/foundation models with light fine-tuning over training from scratch unless you have a large, unique dataset — faster to ship and usually more robust.
- Running inference on the UI/main thread, causing jank or ANRs.
- Shipping an unquantized full-precision model "to be safe" — bloats app size and drains battery unnecessarily.
- Assuming cloud latency in dev/testing (same datacenter) reflects real-world mobile network conditions.
- No fallback path when the network is unavailable for a cloud-dependent feature — always define what happens offline.
- Ignoring cold-start latency when using scale-to-zero cloud endpoints for latency-sensitive features.
- Hardcoding model versions in app code, forcing an app store release just to update a model.
- Not testing on low-end/older devices — a model that runs fine on a high-end phone may be unusable on the median user's device.
- Treating model accuracy as static — skipping ongoing monitoring for drift once deployed.