Building Realtime Data Dashboards

JavaScript
// Minimal realtime dashboard pattern
const socket = new WebSocket('wss://api.example.com/stream');
const store = createReactiveStore({ metrics: [], connections: 0 });

socket.onmessage = (event) => {
  const update = JSON.parse(event.data);
  store.patch(update); // triggers only affected UI regions
};

// Render layer subscribes to slices, not the whole store
store.subscribe('metrics', (metrics) => renderChart(metrics));

Core principle: separate ingestion (socket/stream), state (reactive store), and rendering (subscribed components). Never let a render cycle block ingestion.

Progress:

• Step 1: Define data sources (WebSocket, SSE, polling, or hybrid)
• Step 2: Design the state shape (normalized, keyed by entity ID)
• Step 3: Build ingestion layer with backpressure handling
• Step 4: Implement reactive store with granular subscriptions
• Step 5: Build chart/widget components that subscribe to slices
• Step 6: Add connection resilience (reconnect, buffering, replay)
• Step 7: Optimize render performance (batching, virtualization)
• Step 8: Add controls (pause/resume, time-range, filters)

Step details

1. Data sources

• WebSocket for bidirectional, high-frequency updates.
• SSE for simpler one-way streams (metrics, logs).
• Polling only as fallback when neither is available.
• Hybrid: initial REST fetch for snapshot + stream for deltas.

2. State shape

• Normalize by ID: { byId: {}, allIds: [] } — avoids O(n) scans on update.
• Keep a rolling window for time-series (e.g., last 5 min) to bound memory.
• Separate "live" state from "historical" state (different retention policies).

3. Ingestion layer

• Queue incoming messages; process in batches per animation frame (requestAnimationFrame or debounced tick), not per-message.
• Apply backpressure: if queue exceeds threshold, drop or coalesce intermediate updates (keep latest value per key).

4. Reactive store

• Use fine-grained subscriptions (per key/slice) so unrelated components don't re-render.
• Prefer signals/observables over global re-render-on-any-change patterns.

5. Charts/widgets

• Charts should accept incremental updates (append/update point), not full re-renders on each tick.
• Use canvas/WebGL for high-frequency, high-density series; SVG/DOM for low-frequency widgets.

6. Resilience

• Auto-reconnect with exponential backoff.
• On reconnect, request a snapshot + resume from last known sequence/timestamp.
• Buffer outgoing user actions during disconnect if applicable.

7. Performance

• Batch DOM/canvas writes; never update on every single message synchronously.
• Virtualize long lists/tables of live rows.
• Throttle chart redraws to display refresh rate (~60fps max, often 10-30fps is enough for dashboards).

8. Controls

• Pause/resume freezes rendering but keeps buffering data (or drops it, explicitly chosen).
• Time-range selector switches between live tail and historical query.
• Filters should apply at the subscription level, not post-render.