AI Skill Report Card
Building Personal Search Indexes
Quick Start13 / 15
Minimal setup flow for a local, privacy-first search engine:
Bash# 1. Install/run the indexer (example pattern) docker run -d --name hister \ -p 8080:8080 \ -v hister_data:/data \ hister/hister:latest # 2. Point it at sources to index hister crawl add --type browser-history --since 30d hister crawl add --type local-dir --path ~/Documents --recursive hister crawl add --type url --seed https://example.com --depth 2 # 3. Run the crawl/index job hister index run # 4. Query it hister search 'title:"project plan" AND modified>2024-01-01'
If no tool is specified, default to Hister's model: local daemon + crawler + full-text index + query language, with AI/semantic search as an optional add-on layer, not a requirement.
Recommendation▾
The skill is built almost entirely around a single, seemingly fictional/niche tool ('Hister') without clarifying whether it's real or hypothetical—this risks hallucinated command syntax being presented as fact; verify tool existence or generalize to real tools (e.g., Recoll, DocFetcher, Meilisearch).
Workflow13 / 15
Progress:
- Step 1: Define sources (browser history, local directories, seed URLs)
- Step 2: Configure privacy boundaries (what stays local, what's excluded)
- Step 3: Set crawler mode (headless browser vs. lightweight HTTP fetcher)
- Step 4: Run initial full index
- Step 5: Set up incremental/scheduled re-indexing
- Step 6: Design queries using the advanced query language
- Step 7 (optional): Enable AI/semantic layer for fuzzy or conceptual search
- Step 8 (optional): Expose as multi-user/team instance with access controls
Step details
1. Define sources
- Browser history: export or live-sync from browser profile dirs.
- Local files: pick directories with real content (docs, notes, PDFs), exclude caches/binaries.
- Web: seed URLs + crawl depth for sites you want fully indexed (wikis, docs sites).
2. Privacy boundaries
- Keep index and raw content on local disk or private network only.
- Exclude sensitive paths (password managers,
.env, private keys) explicitly before first crawl. - For multi-user deployments, separate per-user indexes unless a shared team knowledge base is intended.
3. Crawler mode
- Use a real browser engine (headless Chrome/Playwright-style) for JS-heavy sites or when capturing rendered page text.
- Use a traditional HTTP fetcher for speed on static sites/large local file sets.
4-5. Indexing
- Full index once; switch to incremental (diff-based on mtime/ETag) for ongoing updates.
- Schedule via cron or built-in scheduler (e.g., nightly for browsing history, weekly for static doc sets).
6. Query language
- Favor structured queries over keyword soup: field filters (
title:,url:,path:), boolean operators (AND/OR/NOT), date ranges, phrase matching with quotes. - Combine full-text terms with metadata filters to cut noise fast.
7. AI/semantic layer (optional)
- Layer embeddings-based search on top of, not instead of, full-text index — use it for "find similar/related" and conceptual recall, keep exact-match query language for precision tasks.
- If MCP integration is available, expose search as a tool so AI assistants can query the local index directly.
8. Multi-user/team
- Add auth and per-user index scoping before exposing beyond localhost.
- Decide shared vs. isolated indexes per team/project.
Recommendation▾
Add a troubleshooting section for common setup failures (crawler fails on JS sites, index corruption, permission errors) rather than just conceptual pitfalls.
Examples14 / 20
Example 1: Input: "I want to search everything I've read on arxiv.org in the last year, plus my local PDF notes." Output:
hister crawl add --type url --seed https://arxiv.org --filter "path:/abs/*" --since 365d
hister crawl add --type local-dir --path ~/notes --filetypes pdf,md
hister search 'domain:arxiv.org AND after:2023-01-01 AND "transformer"'
Example 2: Input: "Set up a private team search engine for our internal wiki and shared drive." Output:
hister crawl add --type url --seed https://wiki.internal --depth 5 --auth cookie.json
hister crawl add --type network-share --path //fileserver/team-docs
hister instance enable-multiuser --auth oidc
hister search 'path:"team-docs/Q3" AND title:roadmap'
Example 3: Input: "Find that article about rate limiting I read a few weeks ago but can't remember the exact title." Output:
hister search 'rate limiting AND visited:14d..30d'
# If exact terms fail, fall back to semantic search:
hister search --semantic "article about throttling API requests, read recently"
Recommendation▾
Examples all use the same invented CLI syntax—show a bad/failed query example (e.g., malformed query syntax or an overly broad crawl) alongside the corrected version to meet 'good and bad outcomes' criteria.
Best Practices
- Default to full-text + query language first; reach for semantic/AI search only when exact queries fail.
- Re-index incrementally, not from scratch, to keep crawl times low and avoid redundant storage.
- Exclude noisy/low-value sources (ad trackers, redirect pages, binary caches) from browsing history indexing.
- Keep raw crawled content and the index itself on encrypted local storage if the machine is shared or portable.
- Use depth limits on web crawls — unbounded crawls balloon storage and index junk.
- Version or snapshot the index before major re-crawls so you can roll back.
Common Pitfalls
- Indexing entire browser history unfiltered — it captures logins, banking, and other sensitive pages; always exclude/allowlist.
- Running a headless-browser crawler on thousands of static pages — wastes resources; reserve it for JS-rendered content only.
- Treating semantic search as a replacement for structured queries — it's fuzzier and slower; use field-based queries when you know what you're looking for.
- Exposing a local instance to the network without auth "just for a quick team test" — lock down access before binding beyond localhost.
- Letting local file indexing recurse into
node_modules,.git, or build artifacts — exclude these explicitly to avoid index bloat.