Building Multimodal Content Pipelines
A multi-modal content agent follows this pattern: structured outline (JSON) → per-section text generation → per-section image generation → PDF assembly → upload as tracked artifacts.
Pythonimport os, io, json from reportlab.platypus import SimpleDocTemplate, Paragraph, Image, Spacer from reportlab.lib.styles import getSampleStyleSheet from openai import OpenAI def generate_outline(llm, topic, num_sections): prompt = ( f"Create a {num_sections}-section outline on '{topic}'.\n" "Return JSON: [{ 'title': ..., 'description': ... }, ...]" ) return json.loads(llm.chat(prompt)) def generate_section_pdf(llm, openai_client, section, idx): text = llm.chat(f"Write Section {idx+1}: '{section['title']}'.\n" f"Description: {section['description']}\n\nContent:") img_resp = openai_client.images.create( prompt=f"Illustration for '{section['title']}'", size="512x512") img_data = io.BytesIO(img_resp.data[0].b64_json.encode()) buf = io.BytesIO() doc = SimpleDocTemplate(buf) styles = getSampleStyleSheet() story = [Paragraph(section['title'], styles['Title']), Spacer(1, 12), Paragraph(text, styles['BodyText']), Spacer(1, 12), Image(img_data)] doc.build(story) buf.seek(0) return buf
Progress checklist for building a multi-modal content pipeline:
- Define the input contract (topic, count, format params) as a single JSON entry point
- Generate a structured outline via LLM, forcing machine-readable JSON output
- Parse and validate the outline before proceeding (fail fast on malformed JSON)
- For each outline item, generate text content with a targeted prompt (title + description + explicit instruction)
- For each outline item, generate an accompanying image via an image-generation API
- Assemble text + image into a single formatted document (PDF/DOCX) per section, in-memory (BytesIO) when possible
- Upload/attach each generated artifact to the orchestration platform's job/run for traceability
- Verify environment configuration (API keys, orchestrator URL/token) is externalized via env vars/.env, never hardcoded
- Package the automation (CLI pack/publish equivalent) and confirm it appears in the target runtime environment
- Trigger a test run with minimal inputs and confirm artifacts land where expected
Step details:
-
Outline generation: Always instruct the LLM explicitly to return JSON with a defined schema (
title,descriptionfields). This makes downstream parsing deterministic instead of scraping free text. -
Content generation loop: Iterate the outline list; each iteration is independent (embarrassingly parallel) — design generation functions to be stateless and idempotent so they can later be parallelized or retried individually.
-
Image generation: Use a separate lightweight client dedicated to image APIs; keep prompt phrasing consistent (e.g., "Illustration for chapter titled '{title}'") so style stays cohesive across a document.
-
Document assembly: Use in-memory buffers (
io.BytesIO) rather than writing to disk mid-pipeline — reduces I/O overhead and simplifies cleanup, especially when the final step is an upload rather than local storage. -
Attachment/artifact upload: Treat every generated file as a trackable artifact tied to the orchestration job — this is what gives the pipeline auditability and lets consumers retrieve outputs without re-running the agent.
Example 1:
Input: {"topic": "Space Exploration", "num_chapters": 3}
Output: Outline JSON with 3 chapter objects (title + description), then 3 PDFs generated (chapter_1.pdf, chapter_2.pdf, chapter_3.pdf), each containing chapter title, LLM-written narrative text, and a DALL-E illustration — all uploaded as job attachments.
Example 2:
Input: A malformed LLM response missing the closing bracket in the outline JSON.
Output: json.loads raises JSONDecodeError — pipeline should catch this, log the raw LLM output, and retry the outline prompt once with a stricter "return ONLY valid JSON, no markdown fences" instruction before failing the job.
- Force structured (JSON) output from LLMs for any step whose result feeds into code logic — never parse free-form prose programmatically.
- Keep configuration (API keys, orchestrator URLs/tokens) in environment variables or
.envfiles, loaded at runtime, never committed to source. - Separate concerns cleanly: outline generation → content generation → asset generation → document assembly → upload. Each stage should be a standalone, testable function.
- Generate documents in-memory (buffers) rather than temp files when the end destination is an upload/API call.
- Name output artifacts predictably and consistently (e.g.,
chapter_{idx+1}.pdf) so downstream consumers can programmatically locate them. - Make the number of sections/chapters and the topic the only required manual inputs — everything else should be derivable or defaulted.
- Wrap external API calls (LLM chat, image generation) with retry logic since generation of many sections increases the chance of a transient failure somewhere in the loop.
- Don't ask the LLM for free-form text when you need structured data — always specify the exact JSON schema in the prompt.
- Don't hardcode credentials or environment-specific URLs in code; this breaks portability between dev/test/prod deployments.
- Don't write generated PDFs/images to disk unnecessarily if the final step is an upload — extra I/O adds latency and cleanup burden.
- Don't generate all chapter content in a single giant prompt — per-section generation gives better quality, easier debugging, and partial-failure recovery.
- Don't skip validation of LLM JSON output — always wrap
json.loadsin error handling since LLMs occasionally wrap JSON in markdown fences or add commentary. - Don't forget to track every generated artifact against the job/run — untracked outputs undermine auditability in enterprise settings.