Designing Instance As Token Sequence Compression

When a recommendation model's sequence features are hand-crafted (item ID, category, price, interaction type, timestamp, etc.) and the modeling architecture has outgrown what those sparse features can express, replace the feature engineering bottleneck with instance-level compression:

1. Take an existing "source" ranking model (any base CTR/CVR model) that already ingests the full instance (thousands of raw features describing one historical interaction).
2. Attach a bottleneck compression layer to its final representation, train it end-to-end on the original task labels, and persist the bottleneck activation as a dense InsEmb (instance embedding) per training instance, keyed by (user, timestamp).
3. In the downstream ranking model, at serving/training time, fetch the InsEmbs for a user's past interactions strictly before the current request timestamp, concatenate with a few optional lightweight side signals to build InsTokens, and feed this token sequence into a standard sequence architecture (Transformer, LONGER, etc.) in place of hand-crafted sequence features.

This turns "sequence of hand-crafted feature vectors" into "sequence of learned dense instance tokens" — increasing information density without exploding feature engineering cost.

Progress:

• Step 1: Define the source (compression) model and its training objective
• Step 2: Choose compression ordering scheme (temporal-order vs. user-order)
• Step 3: Design the bottleneck compression/decompression module
• Step 4: Build the streaming storage pipeline for InsEmb
• Step 5: Design downstream InsToken construction with leakage-safe retrieval
• Step 6: Integrate InsToken sequence into downstream sequence architecture
• Step 7: Validate offline (in-domain + cross-domain transfer) and online (A/B test)

Step 1 — Source model. Use an existing (or a lightweight) ranking model that consumes the entire raw instance (all sequential + non-sequential features of that one interaction) and produces a final representation h. This model is trained on the same or a closely related label space (e.g., multi-task CTR/CVR) as the downstream task — the compression quality depends on this supervision being meaningful.

Step 2 — Compression ordering scheme. Two options, pick based on constraints:

• Temporal-order: instances are compressed independently per-instance in arrival order; simpler, no cross-instance dependency, but the resulting InsEmb has no built-in sequence-awareness.
• User-order: instances belonging to the same user are grouped and passed through a lightweight in-source sequence module (e.g., a "Source Instance Transformer") during compression, so InsEmb already encodes short-range sequential context. Prefer this scheme — it aligns better with the downstream sequence-modeling consumer and empirically transfers better.

Step 3 — Bottleneck module. Insert a compression layer (down-projection, e.g. MLP or linear layer to low dimension) after the source model's final representation, followed by a decompression layer (up-projection back to original task-head input size) so the original task loss (BCE on click/conversion labels) still supervises the bottleneck. Store the compressed intermediate activation (not the decompressed one) as InsEmb. Optionally attach auxiliary heads (multi-task labels) whose outputs or labels are also stored as complementary key features.

Step 4 — Storage pipeline. Persist InsEmb (+ optional key features) keyed by (user_id, timestamp, instance_id) in a low-latency centralized store, produced via a streaming pipeline so downstream training/serving can retrieve recent instances with minimal delay. Treat this as a new "sequential feature" produced automatically, replacing manual feature engineering cycles.

Step 5 — Downstream retrieval. At training/serving time for a request at time t, fetch a fixed-length window of the user's past InsEmbs with timestamp < t (strict truncation to prevent label/future leakage). Concatenate each InsEmb with any retained lightweight side info (position/time-delta, optional multi-task labels) to form an InsToken.

Step 6 — Downstream architecture. Feed the InsToken sequence into a standard or state-of-the-art sequence encoder (Transformer, target-attention DIN-style, or long-sequence architectures like LONGER) exactly as you would hand-crafted sequence features — no need to redesign ℱ_interaction. This isolates the improvement to feature representation quality rather than architecture complexity.

Step 7 — Validation. Evaluate offline with (a) in-domain AUC/GAUC lift vs. hand-crafted-feature baseline, (b) cross-domain transfer (train source model in one scenario, deploy InsEmb in another) to test representation generality, and (c) online A/B test on business metrics (CTR, GMV, etc.) before full rollout.

Example 1: Input: An e-commerce ads ranking model uses a 50-length sequence of hand-crafted features (item_id, category_id, price_bucket, action_type, dwell_time) per historical click/purchase, and model quality has plateaued despite architecture upgrades. Output: Train a user-order source model on the same click/purchase labels using the full instance (thousands of raw features per interaction) with a Source Instance Transformer for in-source sequence context; bottleneck to a 64-dim InsEmb; stream-store per (user, timestamp); downstream model retrieves the last 50 InsEmb tokens (timestamp-truncated) and feeds them into the existing Transformer sequence encoder, replacing the 5-field hand-crafted vectors — yielding denser per-step representation without adding new hand-crafted fields.

Example 2: Input: A live-streaming e-commerce platform wants to reuse a mature "gifting propensity" model's rich instance representations to bootstrap sequence modeling in a new "shopping mall marketing" scenario with sparse labels. Output: Use the gifting-propensity source model (already trained with user-order compression) to generate InsEmb for shopping-mall interaction logs (cross-domain transfer); downstream shopping-mall ranking model consumes these InsEmb as InsTokens directly, benefiting from transferable dense representations despite limited native training data in the new domain.