Replace the fixed RoPE angle p_i * θ_j with a learned, signal-conditioned angle:

Python
Θ_j(T_i, p_i) = f_φ(T_i)_j * ω_j^s   +   p_i * θ_j * λ
                └── temporal (SIREN) ┘   └── ordinal (scaled) ┘

• f_φ: dual-branch network (periodic SIREN + aperiodic ReLU DNN) mapping timestamp features → per-dimension angles
• ω_j^s: learnable per-dimension frequency scale (init to π)
• θ_j = base^(-2j/d_k): standard RoPE inverse frequencies (kept fixed)
• λ: learnable scalar gate (init to 1.0) balancing ordinal vs. temporal contribution

Apply the resulting Θ_j exactly like standard RoPE's rotation angle (real-valued 2D rotation, no complex ops needed—torch.compile compatible).

Progress:

• Step 1: Identify the "hidden dimension" — what non-ordinal signal is being discarded (timestamps, categorical metadata, cyclical patterns)?
• Step 2: Decompose the raw signal into model-friendly features (see below)
• Step 3: Design the dual-branch mapping network (periodic + aperiodic)
• Step 4: Fuse with the ordinal term via a learnable gate, don't replace it outright
• Step 5: Initialize near the original RoPE behavior for training stability
• Step 6: Build controlled baselines to isolate rotation-space vs. embedding-space signal
• Step 7: Validate on calibration + ranking metrics, inspect learned gate value as evidence

Step 2: Feature decomposition for continuous/cyclical signals

For any periodic raw signal (e.g., Unix timestamp), always encode each cycle as a (cos, sin) pair to avoid phase discontinuities at period boundaries:

t(T) = [cos(2πT/τ_d), sin(2πT/τ_d),   # daily cycle
        cos(2πT/τ_w), sin(2πT/τ_w),   # weekly cycle
        T_normalized]                  # long-range aperiodic trend

Generalize: one (cos, sin) pair per known periodicity, plus one normalized scalar for aperiodic/monotone trend (e.g., recency decay). This lets the downstream network recover cycles (e.g., time-of-day patterns) without needing side info like timezone.

Step 3: Dual-branch mapping network

f_φ(T) = f_sin(T) + f_DNN(T)

• Periodic branch (f_sin): SIREN layers — sin(ω_0 · Wx + b) — to autonomously discover periodicities beyond the manually specified ones.
• Aperiodic branch (f_DNN): standard ReLU MLP for monotone trends (decay, drift). Still capable of periodic modeling if fed cyclical features, but primarily there for trend capture.
• Combine additively; keep the network small (target <1% parameter overhead).

Step 4: Fusion and gating

Never fully discard the ordinal term — fuse it with a learnable scalar gate λ:

Θ_j = f_φ(T)_j · ω_j^s  +  p_i · θ_j · λ

This preserves translational equivariance / recency-decay properties of vanilla RoPE while letting gradient descent decide the balance. Track λ's trajectory during training — a large drop from init (e.g., 1.0 → 0.04) is strong empirical evidence the new signal dimension is doing the work.

Step 5: Initialization for stability

Initialize ω_j^s near the value that recovers something close to standard RoPE's frequency scale (e.g., π), and λ = 1.0. This ensures training starts near a known-good configuration rather than from scratch.

Step 6: Controlled baseline design (critical for proving the idea)

To isolate where a new signal helps, hold everything else fixed and vary only how the signal is injected:

1. Ordinal-only baseline: no new signal at all (standard RoPE).
2. Feature-injection baseline: inject the new signal as an appended embedding/feature (semantic space), keep rotation purely ordinal. This tests whether the signal helps at all regardless of mechanism.
3. Fixed-schedule rotation baseline: inject the signal into rotation angle using the existing fixed inverse-frequency formula (no learned mapping).
4. Full method: learned signal-conditioned rotation (your contribution).

Comparing baselines 2 vs. 4 isolates "does routing through rotation space beat routing through embedding space." Comparing 3 vs. 4 isolates "does the learned mapping beat a fixed schedule."

Step 7: Evaluation

Use both calibration metrics (e.g., normalized entropy/NE) and ranking metrics (e.g., AUC) across multiple related tasks. Report parameter overhead explicitly (aim for negligible, e.g., ~0.2%). Use production-scale data if available — real irregular-interval behavior is the whole motivation, and synthetic/academic benchmarks with uniform sampling won't expose the gap.