Porting Bot Performance Architecture
Given two codebases (e.g., "vc bot" = fast, "newton bot" = slow):
- Map both bots' request lifecycle end-to-end (input → processing → output).
- Diff the two lifecycles stage-by-stage to find where newton bot spends extra time.
- Extract vc bot's core execution engine (streaming, concurrency, caching, model calls) as a swappable module.
- Replace ONLY newton bot's internal processing pipeline with that engine.
- Verify newton bot's UI, system prompts, feature set, and outputs are byte-for-byte unchanged in behavior — only speed changes.
Do not start rewriting code before completing the full comparison in the Workflow below. Premature patching causes partial fixes that don't address the real bottleneck.
Progress:
- [ ] Step 1: Inventory both codebases (files, entry points, dependencies)
- [ ] Step 2: Trace vc bot's full request→response pipeline
- [ ] Step 3: Trace newton bot's full request→response pipeline
- [ ] Step 4: Build a stage-by-stage latency comparison table
- [ ] Step 5: Identify root cause(s) of newton bot's slowness
- [ ] Step 6: Isolate vc bot's "engine" (the fast part) from its UI/prompt layer
- [ ] Step 7: Freeze newton bot's UI, prompts, and features as protected/do-not-touch
- [ ] Step 8: Swap newton bot's engine internals for vc bot's engine
- [ ] Step 9: Re-wire newton bot's UI/prompt layer to call the new engine
- [ ] Step 10: Test parity (features/output unchanged) and speed (latency improved)
Step 1–3: Trace both pipelines
For each bot, document every stage from user input to final output (text and audio):
- Input handling (webhook, socket, polling?)
- Preprocessing (validation, formatting, context building)
- Model/API call pattern (streaming vs. blocking, single call vs. chained calls)
- Concurrency model (async/await, threads, sequential blocking)
- Audio pipeline specifically: TTS invocation timing (upfront vs. streamed), chunking, buffering
- Caching (session/context caching, connection pooling, warm clients)
- Output delivery (streamed to client incrementally vs. sent whole after completion)
- Any middleware: logging, retries, rate-limit handling, unnecessary sleep/delay calls
Step 4: Latency comparison table
Build a table like:
| Stage | vc bot | newton bot | Delta |
|---|---|---|---|
| Input parse | 5ms | 5ms | none |
| Context build | 10ms (cached) | 300ms (rebuilt/fetched every call) | root cause |
| Model call | streamed, first token in 200ms | blocking, waits for full response (2s) | root cause |
| TTS | streams audio per sentence chunk | waits for full text then TTS's whole thing | root cause |
| Output delivery | streamed to client | sent as one blob after all processing | root cause |
This table is the deliverable that justifies every subsequent code change — always produce it before touching code.
Step 5: Common root causes (check these first in practice)
- Blocking vs. streaming model calls — slow bot waits for full LLM response instead of streaming tokens.
- No response streaming to the client — even if the model streams, the bot buffers the whole thing before sending.
- Serial audio pipeline — TTS starts only after all text is generated, instead of per-chunk/per-sentence streaming.
- Redundant work per request — rebuilding context, re-authenticating, re-initializing clients on every message instead of reusing warm connections.
- Unnecessary sequential awaits — independent I/O calls run one after another instead of concurrently (
asyncio.gatherequivalent). - Heavy synchronous logging/analytics in the hot path.
- No caching of static prompt/system-prompt tokens, embeddings, or repeated lookups.
Step 6: Isolate the fast engine
Extract vc bot's core loop into a standalone module with a clear interface, e.g.:
engine.process(user_input, context) -> stream of (text_chunk | audio_chunk)
This module should contain NO references to vc bot's UI, prompts, or bot-specific features — just the mechanics: streaming orchestration, concurrency, connection reuse, chunked TTS.
Step 7: Freeze newton bot's surface
Explicitly list what must NOT change:
- System prompt text/content and ordering
- UI components, message formatting, buttons/commands
- Feature flags and bot-specific logic (personas, filters, command handlers)
- Any user-facing strings
Treat this as a contract. Every subsequent diff should touch only internal plumbing.
Step 8–9: Swap and rewire
- Replace newton bot's blocking/serial call sequence with calls into the extracted engine.
- Keep newton bot's system prompt injection point identical — just pass the same prompt into the new streaming call instead of the old blocking call.
- Rewire newton bot's response handler to consume the engine's stream and forward chunks to text/audio output incrementally, matching how vc bot delivers output.
Step 10: Verify
- Feature parity test: run identical inputs through old and new newton bot; confirm identical (or equivalent) outputs, prompts, and UI behavior.
- Speed test: measure time-to-first-token and time-to-full-response for text; time-to-first-audio-chunk for audio, before/after.
- Confirm no regressions in bot-specific features (commands, persona quirks, guardrails).
Example 1: Input: vc bot streams LLM tokens directly into a sentence-chunked TTS pipeline; newton bot calls the LLM, waits for the full completion, then sends the entire text to TTS as one blob. Output: Refactor newton bot's response handler to consume the LLM stream and forward completed sentences to TTS incrementally, using vc bot's chunking logic verbatim. Newton bot's system prompt, persona text, and UI remain untouched — only the internal handler function changes.
Example 2: Input: vc bot reuses a single warm API client and session cache across requests; newton bot re-initializes the API client and rebuilds context from scratch on every message. Output: Introduce a shared client/session singleton in newton bot (matching vc bot's pattern), inject it into the existing request handler without altering the handler's public behavior or the prompts it sends.
- Always produce the latency comparison table before writing any fix — it prevents guessing.
- Change the engine, not the interface: newton bot's callers (UI, command handlers) should see no change in function signatures where possible.
- Port mechanics, not text: never copy vc bot's system prompts, personas, or UI code into newton bot.
- Prefer incremental streaming (text and audio) as the single highest-leverage change — it improves perceived latency even if total processing time is similar.
- Test with real audio/text transcripts side-by-side, not just "it feels faster."
- If working via upload-a-zip-and-review workflow (e.g., ChatGPT), request the full file tree and both entry-point files first, then request the specific modules handling model calls, streaming, and audio synthesis — don't try to review everything at once.
- Rewriting newton bot's prompts or UI "while you're in there" — this breaks the explicit requirement to keep them unchanged and makes regressions hard to isolate.
- Assuming the model API itself is the bottleneck without checking whether the bot is even using streaming mode.
- Copying vc bot's code wholesale instead of extracting just the engine — drags in unrelated features/config that don't belong in newton bot.
- Fixing symptoms (e.g., adding a loading spinner) instead of the root cause (blocking calls, no streaming).
- Skipping the parity test — a faster bot that silently drops a feature or alters the system prompt behavior is a regression, not a fix.
- Optimizing audio and text pipelines as if they're independent — audio latency is often dominated by waiting on the text pipeline to finish first; fix the upstream streaming first.