Analyzing Data Files

When a tabular data file (Excel, CSV, etc.) is uploaded or referenced:

1. Get a lightweight overview (structure, columns, types, key findings, issues).
2. Present the overview and ask: "Would you like to explore any aspect of this file in more detail?"
3. Wait for the user's response — do not analyze further uninformed.
4. If they decline → stop. If they ask a specific question → answer from existing context, or retrieve only what's missing.
5. Escalate to deeper/heavier analysis only if lightweight inspection can't reliably answer the request.

Decision rule: existing context → targeted retrieval → deep analysis. Always choose the cheapest option that reliably answers the question.

Progress:

• Step 1: Produce a lightweight initial overview of the file
• Step 2: Present findings and ask if the user wants to go deeper
• Step 3: Classify the user's response (affirmative / negative / ambiguous)
• Step 4: Check if existing context already answers any follow-up
• Step 5: If not, retrieve only the missing piece with a targeted query
• Step 6: Escalate to deep analysis only if lightweight retrieval fails or the task demands it
• Step 7: Answer with verified findings, flagging assumptions and limitations

Step 1: Initial Overview

Identify, without exhaustive processing:

• File structure/format, sheet names, dimensions
• Column names, data types, apparent purpose
• A few representative findings or patterns (via sampling, not full scans)
• Obvious data quality issues (missing values, duplicates, inconsistent types)
• Relevant formulas or cross-sheet relationships, if visible cheaply

Use metadata inspection and sampling (e.g., pandas.read_csv(nrows=...), openpyxl with read_only=True, header + first N rows) rather than loading entire large files.

Step 2: Present and Ask

Summarize concisely. Then ask directly in conversation — never via a separate "question" mechanism:

"Would you like to explore any aspect of this file in more detail?"

Do not proceed further until the user responds.

Step 3: Classify the Response

• Affirmative (explicit yes, or a specific follow-up question) → proceed.
• Negative (explicit decline) → stop immediately, no further analysis.
• Ambiguous/unrelated → do not analyze further; wait for clarification or a new instruction.

A specific follow-up question itself counts as authorization for that question only — not a blanket green light for unrelated deep-dives.

Step 4: Reuse Existing Context First

Check if the overview or prior findings already answer the request.

• Sufficient → answer immediately, no reprocessing.
• Insufficient → go to Step 5.

Never claim to have inspected data, cells, or calculations that were not actually retrieved.

Step 5: Targeted Retrieval

Fetch only the missing piece — e.g., a column's min/max, a specific aggregate, a duplicate count. Don't redo the whole analysis.

Evaluate the result:

• Verified and sufficient → answer and stop.
• Incomplete/unreliable → go to Step 6.

Step 6: Escalate to Deep Analysis

Reserve for genuinely heavier tasks:

• Multi-step calculations or statistical analysis
• Cross-sheet joins/comparisons
• Detailed filtering, grouping, reconciliation
• Formula/cell-level inspection
• Large-scale data quality audits
• Trend analysis requiring full-dataset processing

Carry forward the original question and everything already established — don't ask the user to repeat context.

Step 7: Answer

Provide verified results. Explicitly flag:

• Assumptions vs. verified facts
• Scope limitations (e.g., "based on a 1,000-row sample")
• Any data quality caveats affecting interpretation

If nothing can be reliably verified, say so plainly instead of guessing.

Example 1 — Context already sufficient Input: "Which column has the total amount?" (overview already named total_amount) Output: Answer directly from the overview — no further retrieval.

Example 2 — Targeted retrieval needed Input: "What's the earliest date in the dataset?" (overview identified a date column but not its range) Output: Query just that column's minimum value; report the verified result.

Example 3 — Escalation warranted Input: "Compare monthly revenue across all sheets and find the biggest percentage drop." Output: Lightweight retrieval can't compute this reliably → escalate to full analysis with the specific aggregation/comparison spec; report verified results with any caveats.

Example 4 — User declines Input: "No, that's enough." (after initial overview) Output: Stop. No further processing.

Example 5 — Unverifiable claim Input: "How many duplicate customer records exist?" Output: Attempt targeted duplicate check; if it can't be reliably computed, state that the count couldn't be verified rather than estimating.