Processing ChatGPT or Claude chat exports to extract durable insights into the wiki. Full pipeline: index → tier → batch extract → promote to wiki.
Build a searchable inventory. Outputs: index.csv + index.md.
conversations-NNN.json files with mapping structure (message.author.role, message.content.parts).
Claude format: Single conversations.json with {uuid, name, summary, chat_messages: [{sender, text: "..."}]}. The text field is usually a flat string — just read msg["text"] directly. Some older exports may use content: [{type: "text", text: "..."}] block format instead; if text is missing or empty, check content and iterate blocks. The sender field uses values like "human" / "assistant" (not role).
Per conversation: id, title, date, model, msg counts, total_chars, has_code, has_image, preview, tags, tier, project.
Tiers: tier1 (20K+ chars), tier2 (5-20K), tier3 (<5K), skip (routine >50K).
Project mapping (ChatGPT): conversation_template_id → Custom GPTs via GPT_PROJECT_MAP.
Batches of 15-25 conversations. Per batch: manifest.json + per-conversation .md + summary.md.
Extraction per conversation: Summary, Key Ideas, Decisions Made, Themes, Notable Quotes, Wiki Concepts. Filenames: Usesanitized_title from manifest — NOT UUIDs.
⚠️ Extracted files are summaries only, not full transcripts. The markdown files produced in Phase 2 contain AI-generated Themes, Key Ideas, and Decisions — they do NOT contain the full back-and-forth conversation text. The full transcripts live only in original/conversations-*.json. If original/ is deleted (e.g., during cleanup), all detailed chat history is permanently lost.
File source mapping: The manifest.json does NOT include file_source. To find which conversations-NNN.json file a conversation lives in:
id with index.csv id → file_source. Group conversations by file_source to avoid loading the same 3-6MB JSON file multiple times.conversations-*.json files directly. Load each file, check if any conversation ID matches your manifest IDs. Group results by file_source. This is slower but works when index.csv wasn't generated or is stale. Use execute_code with a loop over sorted conversation files — load each JSON once, check for matching IDs, build the mapping.for node in mapping.values():
msg = node.get('message')
if not msg: continue
role = msg['author']['role']
if role not in ('user', 'assistant'): continue
parts = msg['content']['parts']
text = '\n'.join(p for p in parts if isinstance(p, str))
if text.strip():
messages.append({'role': role, 'text': text.strip()})
This visits every node exactly once, doesn't require finding root or building a children dict, and works regardless of tree structure. Messages may not be in chronological order, but that's fine for batch extraction where we need text content for analysis, not conversation flow. Use this as the default; fall back to tree-walking only when chronological order matters (e.g., continuing a conversation).
ChatGPT tree-walking extraction (for chronological order): ChatGPT JSON uses amapping structure with parent/child node references (not a flat message list). The children field is often None for every node — never rely on it. To extract text in order:
1. Build a children dict by iterating all nodes: for each node_id, node in mapping.items(), if node["parent"] exists, append node_id to children[node["parent"]]
2. Find root node (the one with parent=None) — there should be exactly one
3. Walk the tree depth-first from root via the children dict, extracting [role] text from each node's message.content.parts
4. Skip nodes with no message (root is often a placeholder with no message)
1. Write a Python script that does regex-based heuristic extraction (Bible verse refs, theme keywords, decision patterns, question detection)
2. Generate baseline markdown files for ALL conversations
3. Then dispatch LLM subagents ONLY for the top 5-10 most important/largest conversations
4. See references/heuristic-extraction-patterns.md for the regex patterns and theme keyword lists
conversations-*.json to find file_source, (4) loads only the needed JSON files, (5) extracts text via tree-walking (ChatGPT) or chat_messages iteration (Claude), (6) generates markdown with theme/key-idea detection, (7) writes files and updates manifest. This avoids subagent overhead, timeout risks, and staging complexity. Use grep -l for fast ID→file_source mapping instead of loading full JSON. For theme/key-idea detection in this pattern, use lightweight title-based regex matching (see references/lightweight-theme-detection.md) instead of LLM analysis — it's fast, deterministic, and sufficient for review-grade output.
Size-based batching (recommended over count-based): When conversations vary wildly in size (20K–350K chars), batch by SIZE tier, not count. This prevents the largest conversations from timing out while smaller ones finish quickly. Example split for 16 conversations:
Dispatch each batch's subagents, wait for completion, then dispatch the next batch. Within a batch, dispatch 3 parallel subagents (max concurrency), then the 4th.
/tmp/ staging pattern: Before dispatching subagents, pre-extract all raw conversation texts to/tmp/<project>_convos/ as individual .txt files. This avoids:
Use execute_code with a grouped-by-file_source loop: load each JSON once, extract all matching conversations, write to /tmp/<project>_convos/<sanitized>.txt. Then subagents just read_file the clean text.
delegate_task(tasks=[...]), then dispatch the 4th as a separate delegate_task(tasks=[single]). The 4th will queue if at capacity, or run synchronously if slots are full.
Produces a single synthesis.md in the ChatGPT extracted/ directory (or the primary corpus root). This is the capstone document — it should surface patterns the user wouldn't see from reading individual files.
1. Read ALL existing analysis files in Claude extracted/ (analysis.md, corpus-analysis.md, memories.md, reflections.md, decisions.md, self-portrait.md, stats.md, index.md) — these provide the Claude-side baseline
2. Read Claude deep-thread analyses (extracted/deep-threads/) and project files (extracted/projects/)
3. Extract metadata from ALL ChatGPT files: use terminal with a loop to pull title, date, tags, chars from frontmatter across all project directories. Group by subdirectory to understand volume per domain.
4. Read full content of representative files from EACH ChatGPT project directory — at minimum: the summary.md, the first 2-3 files alphabetically, and any file with 50K+ chars
5. Cross-reference: look for the same topics, frameworks, or decisions appearing in both ChatGPT and Claude corpora
Required synthesis sections:1. Topic Evolution Timelines — Trace 4-6 key topics across the full date range. For each topic, identify phases (curiosity → framework construction → crystallization → automation). Reference actual conversation titles and dates. Show how thinking changed, not just what was discussed.
2. Cross-Domain Connections — Map ideas from one domain that were applied to another. Examples: faith principles → consulting methodology, homeschooling insights → content creation, technical skills → financial analysis. The WINS framework often serves as a universal connector.
3. Unpublished Ideas — 10-15 concepts that could become blog posts, LinkedIn articles, or book chapters. For each: the idea, source conversation(s) with dates, why it's publishable, and potential title/angle. Mine the faith conversations, consulting frameworks, and writing series especially — these are richest for standalone publication.
4. Usage Patterns — Topic distribution by time period, platform specialization (what went to ChatGPT vs Claude), the shift from Q&A to collaboration to deep thinking, and which conversation types produced the most value. Include a "highest-value conversations" ranked list.
5. Meta-Observations — 4-6 cross-cutting insights about the person: identity tensions, operating philosophies, blind spots, and the corpus-as-mirror finding (deepest AI use is often reflective, not technical).
Output format: Markdown with YAML frontmatter (title, created, source, type). Use## N. for main sections, ### N.M for subsections. Include an Appendix with corpus statistics table.
Quality bar: Every observation must reference specific conversations (title + date). No generic statements. The synthesis should feel like it could only have been written by someone who read the entire corpus.
After extraction and synthesis, move content directly from chatgpt/extracted/ (and claude/extracted/) to their final WINS wiki locations. No intermediate consume/corpus/ staging — move straight to the destination.
| Source Folder | WINS Destination | Notes |
|---|---|---|
health/ |
self/health/ |
Merge into existing directory |
faith/ |
self/faith/ |
Create new subdirectory |
homeschooling/ |
self/homeschooling/ |
Create new subdirectory |
daily-logs/ |
self/daily-logs/ |
Create new subdirectory |
consulting/ |
wealth/consulting/ |
Merge into existing directory |
finance/ |
wealth/finance/ |
Create new subdirectory |
writing/ |
produce/hermes/writing/ |
Create new subdirectory |
app-dev/ |
produce/hermes/app-dev/ |
Create new subdirectory |
ai-ethics/ |
consume/concepts/ai-ethics/ |
Create new subdirectory |
common/ |
produce/common/ |
Staging area for later triage |
chatgpt/extracted/ after migration:
images/ — extracted images (review later)index.csv, index.md — conversation index metadatasynthesis.md — cross-corpus synthesis documentbatch-001-summary.md — extraction batch summarytask.md — migration task trackingclaude/extracted/ follows the same pattern — content moves to WINS locations, only metadata stays.
Execution steps:
1. Create target directories that don't exist yet
2. Move files with mv (not cp -n — see pitfall)
3. Remove empty source directories (including manifest.json artifacts)
4. Verify source is clean (only metadata files remain)
5. Verify destination file counts match
6. Update task.md with completion status
cp -n leaves orphan source files. When merging into existing directories, cp -n (copy no-overwrite) copies files but doesn't remove originals — leaving duplicate orphans. Use mv everywhere, or cp -n + explicit rm of source files. Always verify source is empty after move.common/ folder is a catch-all; after WINS migration, review for re-categorization (many "common" items may actually fit a WINS bucket)original/ directories intact — they're the raw source of truthindex.csv, index.md, synthesis.md) stay in chatgpt/extracted/ — they're reference artifacts, not contentAlways: pilot 1 batch → user reviews → tune → overnight run.
Progress in task.md (not JSON). 2 batches/night, ~5h apart. Prompt references task.md.
Each corpus self-contained: corpus/original/, corpus/extracted/, corpus/scripts/, corpus/task.md, corpus/readme.md.
After indexing, organize extracted output by project, not by batch. Batch folders are an intermediate format — project folders are the durable structure.
chatgpt/extracted/
├── consulting/ ← GPT: consulting (106 convos) = "[client]" in user's mental model
├── health/ ← GPT: health-medical (137 convos)
├── faith/ ← MERGED: wins-faith + bible-study + bible-genesis + sermons-faith + christian-life + reflections (~240 convos)
├── writing/ ← MERGED: business-content + book-club + field-notes + podcast-planning + career-linkedin (~134 convos)
├── homeschooling/ ← GPT: homeschooling (25 convos)
├── finance/ ← GPT: finance-investing (32 convos)
├── daily-logs/ ← GPT: daily-logs (28 convos)
├── app-dev/ ← GPT: app-dev (17 convos)
├── ai-ethics/ ← GPT: ai-ethics (10 convos)
├── common/ ← GPT: general (780 convos) + ALL unmapped g-p-* GPTs
├── index.csv
├── index.md
└── ...
g-p-* IDs) go to common/. User may delete thin ones later.
ChatGPT exports include assets as .dat files in original/ (1,001 files, ~464 MB total). The extraction script (batch_extract.py) skips all asset parts — it only processes string text parts.
.dat file breakdown (verified):
| Type | Count | Notes |
|---|---|---|
| JPEG/JPG | 901 | User screenshots, uploads, generated images |
| PNG | ~51 | Including ~1 unmapped file |
| WebP | 11 | |
| Markdown | 3 | Mapped to original names like go-before-you-know-1-samuel-16.md |
| Text | 2 | Pasted text.txt files |
| 1 | [client] Automotive Service Page.pdf |
|
| Unmapped | 33 | Mostly markdown/text content (titles like "How to...", "Why CTO...") |
conversation_asset_file_names.json maps 968 .dat filenames → original filenames. 33 .dat files are unmapped (check first bytes to identify type — text files start with # or plain text, images start with magic bytes \xff\xd8 for JPEG, \x89PNG for PNG, RIFF for WebP).
Strategy: Keep assets centralized in original/. Do NOT copy to project folders. Reference by asset pointer when needed.
Conversations with inline image refs: The conversations JSON files have zero inline image_asset_pointer content_type references — the mapping tree's content.parts arrays contain only string parts. Images are standalone files, not referenced inline in conversation text. The chat.html export also has minimal image references.
To pull an asset for a specific conversation:
1. Find asset_pointer in conversation JSON (file-service://file-xxx) — if present
2. Look up file-xxx.dat in conversation_asset_file_names.json
3. File is at original/file-xxx.dat
.dat → correct extension using the JSON mapping. Non-image files (markdown, text, PDF) should be extracted separately — they contain user-written content that may be valuable. After extraction, generate an HTML gallery for visual review:
python3 scripts/generate_gallery.py # → extracted/images/gallery.html
The gallery is a dark-themed grid with click-to-enlarge. Sync to Dropbox via rclone bisync, review on Mac/iPhone, delete unwanted images. See references/asset-review-workflow.md for the full review/discard workflow including bisync verification and non-image file handling.
Example extraction code:
with open('conversation_asset_file_names.json') as f:
mapping = json.load(f)
IMAGE_EXT = {'.jpeg', '.jpg', '.png', '.webp', '.gif'}
for dat_name, original_name in mapping.items():
if Path(original_name).suffix.lower() not in IMAGE_EXT:
continue # skip non-images; handle separately
src = Path('original') / dat_name
dst = Path('staging') / original_name
if src.exists() and not dst.exists():
shutil.copy2(src, dst)
When asked to "check the extraction status" or "continue the chat export work":
task.md may be stale. Verify by counting files on disk:
# Per-project extraction counts
for dir in extracted/*/:
count = find(dir -name "*.md" -not -name "index.md" -not -name "manifest.json" | wc -l)
Then compare with manifest.json already_extracted flags. The manifest is the source of truth for what's been processed; disk count confirms files actually exist.
Refresh the Phase 2 progress table with actual counts. The "Next:" line should point to the smallest pending project first (quick wins build momentum), then the large remaining chunk.
already_extracted: false)delegation.max_concurrent_children in config)python3 scripts/batch_extract.py --project <name> to refresh already_extracted flags from disk before dispatching the next batch. The subagent may not have updated the manifest reliably, and stale flags cause duplicate work or missed conversations.remaining = [c for c in manifest if not c['already_extracted']], batch N gets remaining[:20], batch N+1 gets remaining[20:40]. This avoids duplicate extraction without waiting for batch N to finish.file_source. Every subagent dispatch prompt must include: "Search ALL conversations-*.json files in original/ to find which file contains each conversation ID. For Claude, check claude/original/conversations.json." Don't assume the subagent knows this — it doesn't have your context. Without this instruction, the subagent will guess file paths and fail to find conversations.Phase 3 (cross-cutting synthesis) can start when Phase 2 is ~70%+ complete. Count: (named project extractions done + common done) / total tier1 convos. Don't wait for 100% — start synthesis on what's available while finishing remaining extraction.
ChatGPT conversations can't be resumed, but you can continue the thread in Hermes:
1. Look up the conversation in index.csv (by title or ID)
2. Pull full text from original/ JSON via batch_extract.py's load_conversation_text()
3. Summarize context: what was discussed, key conclusions, open questions, where it left off
4. Start a new Hermes session with that context loaded
5. User pastes the summary into the new thread to resume
The original conversation text is fully preserved in original/conversations-*.json — nothing is lost.
When extracting from pre-staged markdown files (not raw JSON), use the consulting-specific format documented in references/consulting-extraction-format.md. This covers the Themes/Key Ideas/Decisions/Questions/Novel Concepts template with metadata headers.
For projects with 30+ conversations, use parallel sub-agents:
1. Pre-load text: Write a Python script that reads the batch plan (manifest + index.csv for file_source), loads each conversation's raw text from original/conversations-NNN.json via load_conversation_text(), and saves to temp files (conv_<sanitized>.txt). Group by file_source to avoid re-reading the same multi-MB JSON.
2. Split into batches of 10-12 conversations (NOT 15-17 — see timeout pitfall below). Save each batch as /tmp/writing_batch_N.json with the plan entries including text_file paths.
3. Dispatch 3 parallel sub-agents via delegate_task. Each gets:
4. Monitor progress: Check ls extracted/<project>/*.md | wc -l periodically. If a sub-agent times out, identify remaining conversations and dispatch a follow-up sub-agent.
5. Post-processing: Update manifest.json to mark newly extracted files as already_extracted: true, update task.md, run batch_extract.py --status.
# <Title>
Source ID: <conversation UUID>
File Source: conversations-NNN.json
Date: YYYY-MM-DD
Tags: tag1, tag2
Themes
1. <Theme Title> — <Description>
Key Ideas (synthesized)
- <Key idea with explanation>
Questions Explored
- <Question>
Decisions Made
- <Decision>: <Details>
Novel Concepts
- <Concept>: <Description>
Field: syntax. YAML frontmatter requires exact formatting (indentation, colon placement) that subagents frequently botch. All existing extracted files (health, homeschooling, writing) use this inline format. Match the established convention.
Exception — task-specified format: When the extraction task explicitly specifies YAML frontmatter (e.g., a parent agent's instructions), use YAML. Some extraction pipelines parse frontmatter programmatically and need it. The tradeoff: YAML is machine-parseable but subagents occasionally produce malformed output; inline bold is human-readable and more reliable for subagents but harder to parse automatically. Choose based on whether the output needs programmatic ingestion. Direct Python scripts (not subagents) can produce YAML frontmatter reliably — the subagent unreliability warning applies only to LLM-generated output.
Post-extraction summary.md format:
[[concepts/name]] links)Some projects (notably common/) merge conversations from both ChatGPT and Claude into a single manifest. The raw data lives in separate directories:
chatgpt/original/conversations-NNN.json (ChatGPT format)claude/original/conversations.json (Claude format, single file)When extracting from a dual-corpus project:
1. Load the manifest to get metadata (title, date, tags, chars)
2. Check each conversation's corpus field to determine which raw directory to search
3. Load from the correct directory — ChatGPT files use mapping tree, Claude uses chat_messages array
4. You may need to load both ChatGPT and Claude raw files in the same script
Claude uuid lookup: When processing multiple Claude conversations, loadconversations.json once and build a uuid→conversation dict for O(1) lookup: claude_lookup = {c['uuid']: c for c in claude_data if 'uuid' in c}. This avoids scanning the full list for each conversation.
Manifest ID mismatch: The manifest IDs may not match raw file IDs (different export versions or UUID regeneration). When ID-based lookup fails, fall back to title-based search: iterate through raw conversations and match by title (ChatGPT) or name (Claude). This is slower but reliable.
Dual-corpus corpus field: Each conversation in the manifest has a corpus field ("chatgpt" or "claude"). Use this to determine which raw directory to search:
corpus: "chatgpt" → search chatgpt/original/conversations-NNN.jsoncorpus: "claude" → search claude/original/conversations.jsonThis avoids searching both directories when you know which corpus the conversation belongs to.
ChatGPT current_node traversal (alternative to children dict): Instead of building a children index and walking from root, you can trace fromcurrent_node back to root via parent references, then reverse the chain for chronological order. This is simpler for single-path conversations:
chain = []
node_id = conv.get("current_node")
while node_id and node_id in mapping:
chain.append(mapping[node_id])
node_id = mapping[node_id].get("parent")
chain.reverse() # Now chronological
delegate_task(tasks=[...]) with 4+ tasks will error. Dispatch in groups of 3, then handle the remainder separately. When dispatching a single task while 3 are running, it may run synchronously (blocking) — this is fine, just means you get the result immediately./tmp/<project>_convos/ before dispatching subagents. This avoids duplicate I/O (multiple subagents reading the same 3-6MB JSON) and eliminates tree-walking logic from subagent prompts. Exception: For small batches (≤10 convos across ≤2 JSON files), letting subagents read directly is fine — the /tmp/ staging adds setup overhead that exceeds the I/O savings.conversation_template_id = project key. Map g-p-* to names via GPT_PROJECT_MAP.07fcfbfa.md.sender not role. Values: "human"/"assistant".summary as fallback.conversation_template_id empty = 'general'. Don't leave raw IDs.sanitized_title. Subagent uses for filenames.sanitized_title directly from manifest.json. Do NOT sort the manifest, pick titles, and pass them to a subagent — the subagent will fail to find conversations because your title strings won't match the manifest exactly. Read the manifest, filter for already_extracted: false, and pass the actual id + sanitized_title values.file_source and index.csv cross-reference fails, use execute_code to iterate through all conversations-*.json files, load each, check for matching IDs, and build an id→file_source mapping. Group conversations by file_source to avoid loading the same multi-MB JSON file repeatedly. Fast ID search optimization: Instead of parsing full JSON for each file, read the file as a string and check if the target ID appears in the content. This is much faster for large files (3-6MB) when you only need to verify presence: if conv_id in open(f).read(). Build the id→file_source mapping in one pass, then load only the files you actually need.corpus/scripts/ outputs to corpus/extracted/. NOT shared folder.batch_extract.py skips all non-string content parts. 1,001 .dat files (464 MB) stay in original/. 968 are mapped in conversation_asset_file_names.json (962 images + 6 text/md/pdf). 33 unmapped .dat files exist — mostly markdown/text content. Pull on-demand via the JSON mapping. See "Image and Asset Handling" section above.faith/, writing GPTs → writing/. Don't create per-GPT folders for these.g-p-* GPTs (1-9 convos) don't warrant their own folders.file_source. To find which conversations-NNN.json file a conversation lives in, join manifest id with index.csv id → file_source. Group extractions by file_source to avoid loading the same multi-MB JSON file repeatedly.children field on nodes is often None for ALL nodes — do NOT rely on it. Always build a reverse map from node.parent references: children_of[parent_id].append(node_id). Find root (parent=None), walk depth-first via the reverse map. Don't assume messages are flat, and don't assume children arrays exist. The only reliable traversal is parent→children built from node.parent.python3 scripts/batch_extract.py --project <name> to regenerate the manifest (it updates already_extracted flags from disk), then read the fresh manifest for the next batch. Do not try to reconstruct IDs from partial output.rclone bisync --resync does a full comparison and can take 10-30 minutes for large wikis. If one is already running, wait for it to complete — starting a second bisync on the same paths causes conflicts. Verify with ps aux | grep "rclone bisync" | grep -v grep.enrich_source_metadata.py silently fails for some conversations. The script looks up file_source from index.csv using the conversation UUID. If the UUID isn't in index.csv (e.g., conversations extracted outside the main pipeline, or tier3 conversations not in the index), it defaults to conversations.json (wrong) instead of the correct conversations-XXX.json format. This produces 51+ files with incorrect file_source and unreliable source_id metadata. After enrichment, audit: count files with file_source: conversations.json (wrong) vs file_source: conversations-XXX.json (correct). Files with wrong metadata have source_ids that won't resolve to real ChatGPT URLs. Investigation and correction: see references/source-id-investigation.md for the diagnostic steps (check JSON files, title matching, content matching) and the decision framework for whether to correct or strip the metadata.source_id in extracted files maps to the ChatGPT conversation UUID, and the URL format is https://chatgpt.com/c/{uuid}. However: (1) the URL only works if you're logged into the same ChatGPT account OR the conversation was explicitly shared, (2) shared_conversations.json has TWO different ID fields — id (internal UUID, same as index.csv) and conversation_id (the shareable UUID used in URLs) — these are NOT the same, (3) conversations with wrong source_id metadata (see above) won't load at all. Before constructing URLs, verify the source_id exists in index.csv.original/ is irreversible. Phase 4 of the task.md plan says "Delete original/ directory" — this removes all raw JSON (full transcripts) and .dat files (images). After deletion, only the extracted summaries remain. Always confirm with the user before executing this step, and consider archiving (compress) instead of deleting.