Designing AI Semantic Capture Systems

Given a long chat screenshot (e.g., LINE capture), the pipeline is:

Image → Object Detection → OCR → Anchor Identification →
Range Computation → Object-Safe Pagination → Human Review → Export

Minimal object schema (every detected element must conform to this):

JSON
{
  "object_id": "obj_001",
  "object_type": "incoming_text_bubble",
  "bbox": {"x_min": 120, "y_min": 340, "x_max": 620, "y_max": 430},
  "text": "โอนแล้วครับ",
  "sender_side": "left",
  "timestamp": "14:32",
  "ocr_confidence": 0.94,
  "detection_confidence": 0.91,
  "belongs_to_group_id": "msg_group_001"
}

Core rule to enforce everywhere in the design: select semantically, layout object-safely, never cut evidence objects, always preserve context and metadata.

Use this checklist when designing or implementing each submodule. Treat each as independently testable.

Progress:
- [ ] 1. Object Detection Layer — detect + classify all chat objects, output bbox map
- [ ] 2. OCR & Text Understanding — extract text with per-object confidence
- [ ] 3. Anchor Decision Engine — find Start/End anchors via multi-signal scoring
- [ ] 4. Semantic Range Computation — apply buffer, snap boundaries to object edges
- [ ] 5. Object-Aware Block Layout — group related objects, assign to blocks
- [ ] 6. Object-Safe Pagination — enforce no-cut rule, handle oversized objects
- [ ] 7. Human Review Layer — compute confidence states, gate auto-crop
- [ ] 8. Audit Trail — log detection/OCR/anchor decisions with reasoning
- [ ] 9. UI Panel — wire controls + overlays + status cards
- [ ] 10. Export — produce clean pages for PDF generation

Step 1 — Object Detection

Classify against the fixed taxonomy (incoming/outgoing bubble, image/slip/file/voice attachment, sticker, profile icon, sender name, timestamp, read receipt, date divider, system message, reply quote, reaction, deleted placeholder). Group objects that co-occur into belongs_to_group_id clusters (a message group = icon + name + bubble + timestamp + receipt + attachment + reaction).

Step 2 — OCR & Text Understanding

Run OCR per readable object, not on the whole image. Support Thai/English/mixed/numeric patterns (account numbers, amounts, dates, times). Any object below the OCR confidence threshold gets needs_review: true — never silently discard it.

Step 3 — Anchor Identification

Never assume the first/last visible message is the anchor. Score each candidate object using combined signals, not single keyword matches:

• keyword match (domain dictionary)
• amount pattern / bank pattern regex
• presence of slip image nearby
• conversation order / sender direction
• timestamp continuity
• neighboring context window (±2 objects)

Produce for each anchor: object_id, confidence, reasoning summary. If no anchor clears the threshold → ANCHOR_NOT_FOUND.

Domain dictionaries are swappable config, never hardcoded:

JSON
{
  "domain": "financial_evidence",
  "start_anchors": ["ราคา", "ยอด", "บัญชี", "โอนเข้า", "เลขบัญชี", "ตกลง", "ชำระ"],
  "end_anchors": ["โอนแล้ว", "ส่งสลิป", "สลิป", "ได้รับแล้ว", "เรียบร้อย", "เงินเข้า"],
  "supporting_signals": ["bank_name", "amount_pattern", "slip_image", "timestamp", "account_number_pattern"]
}

Support additional domains (harassment_evidence, contract_evidence, custody_dispute_evidence, threat_evidence, custom_domain) by loading a different config, not by branching code.

Step 4 — Semantic Range Computation

selected_start_index = start_anchor_index - buffer_before   # default 1–2
selected_end_index   = end_anchor_index + buffer_after       # default 1–2

y_top    = bbox_top(first_selected_object)    - safe_margin  # 10–16px
y_bottom = bbox_bottom(last_selected_object)   + safe_margin

Crop boundaries always snap to object bbox edges — never to arbitrary pixel offsets.

Step 5 — Object-Aware Block Layout

Treat each message group as an atomic layout unit. Never split: bubble↔timestamp, icon↔message, slip↔referencing message, reply quote↔reply bubble.

Step 6 — Object-Safe Pagination

Default block: 645×890px.

for object in selected_objects (in order):
    if object.height <= remaining_space_in_current_block:
        place in current block
    elif object.height <= block.height:
        start new block, place object there
    else:  # object taller than one full block
        if scaling_allowed: scale proportionally, preserve aspect ratio
        else: place on dedicated overflow page
        if still unresolved: flag for human review

A block boundary must fall between two objects' bboxes, never inside one.

Step 7 — Human Review Gating

Compute a confidence state and gate behavior accordingly:

Step 8 — Audit Trail

Every export must emit source hash, object counts, OCR engine used, anchor reasoning, and human_confirmed boolean.

Example 1 — Financial evidence, clean case Input: Long LINE screenshot containing price negotiation → account number → "โอนแล้วครับ" → slip image, surrounded by unrelated chit-chat before/after. Output:

JSON
{
  "selected_range": {"start_object_id": "obj_012", "end_object_id": "obj_024", "buffer_before": 2, "buffer_after": 2},
  "start_anchor_reason": "Detected price agreement and account number request",
  "end_anchor_reason": "Detected slip image and received-confirmation phrase",
  "confidence_state": "HIGH_CONFIDENCE"
}

Example 2 — Bubble at page boundary Input: Selected object obj_020 (image attachment, 420px tall) would start at y=810 in a block with only 80px of space left (block height 890px). Output: obj_020 is moved entirely to the next block (page_002); page_001 ends at the bottom of obj_019. No cropping occurs.

Example 3 — Oversized slip image Input: Slip image object is 1100px tall, exceeding the 890px block height. Output: System either (a) scales the image proportionally to fit width=645px with preserved aspect ratio if scaling is permitted, or (b) places it alone on a dedicated overflow page at native size, and logs OVERSIZED_OBJECT in the audit trail.