Writing Corpus Analysis

Analyze a large corpus of your own blog posts, articles, or essays to extract concepts,

discover themes, and identify product opportunities (ebooks, courses). Designed for

corpora of 100-2,000+ pieces with YAML frontmatter.

When This Skill Activates

Use when the user asks to:

• Process/analyze their blog posts, articles, or writing corpus
• Figure out "what themes and concepts I write about"
• Identify ebook or course ideas from their existing content
• Improve existing blog posts systematically
• "Map" their intellectual territory from their writing
• Organize and extract insights from exported chat histories (ChatGPT, Claude, etc.)
• Analyze any large structured text corpus for patterns, decisions, and frameworks

Architecture: Five Phases

Phase 1: INVENTORY → Phase 2: CONCEPT EXTRACTION → Phase 3: QA → Phase 4: THEMES → Phase 5: PRODUCTS

Each phase builds on the previous. Pause for user review between Phase 2 and 3.

Phase 1: Inventory & Clustering

Goal: Map the corpus — count, date range, tag distribution, cluster assignment.

Steps

① Run scripts/corpus-inventory.py (or write equivalent) to read all posts:

# For each post, extract: title, date, tags, slug from YAML frontmatter
Count totals, date range, posts per year, tag frequency

② Define clusters based on tags + title keyword analysis. Typical clusters for a generalist writer:

• AI & Technology
• Leadership & Management
• Career Growth
• Personal Development (split into Mindset, Goals/Habits, Life Lessons)
• Startups & Business
• Books & Reading
• Podcast Interviews (special handling — guest content, not original ideas)
• Faith & Worldview
• Parenting
• Finance & Wealth
• Writing & Content Creation
• Tools & Productivity

③ Assign each post to one or more clusters using tag overlap + title keyword matching.

Posts can belong to multiple clusters (e.g., a post about AI leadership goes in both AI and Leadership).

④ Save cluster inventory as JSON: _meta/cluster-inventory.json

{
  "clusters": {
    "ai-tech": {
      "name": "AI & Technology",
      "count": 133,
      "posts": [{"file": "...", "title": "...", "date": "...", "tags": [...]}]
    }
  }
}

⑤ Present the map to the user. Get buy-in on clusters before proceeding.

Scaling rule

Clusters over 150 posts should be split further. Use sub-clustering based on tag/title

patterns (e.g., Personal Development → Mindset, Goals/Habits, Life Lessons).

Phase 2: Parallel Concept Extraction

Goal: Extract recurring concepts, frameworks, and principles from each cluster using

parallel subagents. Create concept pages in the user's wiki.

Strategy

Use delegate_task(tasks=[...]) to run 3 subagents in parallel per batch. Each subagent

processes one cluster. Smaller clusters complete; larger ones may timeout but produce

partial results — integrate those per the wiki-ingest-qa skill.

Subagent Prompt Template

See references/subagent-prompt-template.md for the full template. Key elements:

• Context: Cluster description, number of posts, date range, what to look for
• Task: Read all posts, identify concepts, create pages in wiki/writing/concepts/
• Threshold: Create a concept page when an idea appears in 3+ posts OR is the central thesis of one post. Don't create pages for passing mentions.
• Format: YAML frontmatter with title, created, updated, type, tags, sources, first_published, revisited_in, confidence, status
• Style: Write in the author's voice — distill what THEY said, don't add your own ideas
• Critical: DO NOT update index.md or log.md (parent handles that). Use [[wikilinks]]. Check for existing pages before creating duplicates.
• Signal assessment: Each extraction file should include a signal level indicator (🟢 High = novel frameworks/decisions, ⚪ Low = utility/code-only). This lets the reviewer quickly filter what matters. The batch summary should report signal density (% high-signal conversations).

Batch Planning

Process clusters in batches of 3 (max concurrent subagents):

Batch	Example Clusters	Est. Time
1	Largest 3 clusters (100-200 posts each)	10-15 min
2	Mid-size (70-120 posts)	8-12 min
3+	Smaller (14-54 posts)	3-8 min

Special Cluster: Podcast Interviews

Podcast interview posts are guest content, not the author's original ideas. For this cluster:

• Extract the author's editorial lens — what themes recur across guests, what they chose to highlight
• Extract the author's own frameworks articulated in intros/outros
• Do NOT create concept pages for individual guests or their ideas
• Create a podcast-landscape.md overview page capturing patterns across the body of work

Timeout Handling

Subagents processing 100+ posts typically timeout at 600s. This is partial completion,

not failure. Per wiki-ingest-qa:

1. Audit what pages were created on disk

2. Integrate partial results into index.md

3. For significantly incomplete clusters, re-dispatch with explicit "ALREADY CREATED" lists

Phase 3: QA & Consolidation

After all batches complete, run QA per the wiki-ingest-qa skill:

Steps

① Count all concept pages — verify against expectations from each cluster

② Check for duplicates — some subagents may create overlapping pages (e.g., "structured-communication" from both Leadership and Mindset clusters). Check by filename slug.

③ Verify frontmatter — spot-check 5-10 pages for required fields

④ Wikilink verification — check that [[wikilinks]] resolve to actual files

⑤ Update index.md — one central pass adding all concept pages alphabetically under ## Concepts. Avoids the subagent collision problem where 3 agents all update index.md independently.

⑥ Update log.md — prepend one consolidated entry listing all pages created per cluster

Index Consolidation

Use this pattern (subagents are told NOT to touch index.md):

# 1. List all .md files in concepts/ directory
2. Extract title from each file's YAML frontmatter
3. Build alphabetically-sorted wikilink entries: [[concepts/slug]] — Title
4. Insert into index.md under ## Concepts section
5. Update "Total pages: N" header

Phase 4: Theme Discovery

Goal: Find cross-cutting patterns that span multiple clusters — ideas the author returns

to regardless of topic. These become wiki/writing/themes/.

What Makes a Theme

A theme spans 5+ posts across at least 2 different clusters. Example: "systems thinking"

appears in tech posts, leadership posts, AND parenting posts.

Process

After all concept pages exist, read the full concept list and look for:

• Concepts that share the same underlying pattern (e.g., flywheel, snowballs-not-slot-machines, compound-effects all point to "compounding")
• Same mental model applied to different domains
• Recurring frameworks that show up across categories

Create one theme page per cross-cutting pattern, linking to all concept pages and posts it touches.

Phase 4b: Self-Portrait (Chat Exports Only)

Goal: Build a behavioral profile from the person's actual messages — what they do, not what the AI says about them.

Independent Analysis First

⚠️ Do NOT read Claude's memories, reflections, or pre-existing analysis before this phase. Those are the AI's interpretation. You need the raw signal.

1. Extract human messages from conversations.json (see references/behavioral-signal-extraction.md)

2. Run behavioral signal analysis: punctuation habits, abbreviation patterns, message structure, conversation arcs, topic frequency

3. Identify patterns invisible in AI summaries: how they correct, how they provide context, how they end conversations, what roles they assign

Then Compare With AI Analysis

After your independent analysis:

1. Read Claude's memories/ and reflections/

2. Read any extracted/ analysis files

3. Identify gaps: what did the AI miss or smooth over?

4. The delta between your raw analysis and the AI's interpretation is often the most valuable insight

Output

Write extracted/self-portrait-independent.md with:

• Behavioral patterns from raw data (with evidence counts)
• Linguistic signatures (punctuation, abbreviations, opening patterns)
• What the person actually talks about vs. what the AI said they talk about
• Central tension visible in the raw data
• What the AI analysis got right vs. what it missed

Phase 5: Product Identification

Goal: Identify ebooks, courses, and other products for platforms like Gumroad.

From Themes → Books

A theme graduates to a book when there's enough density (15+ posts across 3+ years). Create a book outline in wiki/writing/books/:

books/systematic-leadership/
├── README.md    — outline, chapter map, target audience, pitch
├── 01-foundations.md
├── 02-frameworks.md
...

From Clusters → Courses

A cluster with 20+ posts and a clear "how-to" angle can become a course. Evaluate:

• Does the author have a unique framework or methodology?
• Is there a step-by-step progression across the posts?
• Is there a clear target audience with a problem to solve?

Blog Improvements

Flag posts for:

• Updating: Old AI posts (pre-2023), dated tool recommendations
• Combining: Multiple short posts on the same topic → one definitive guide
• Expanding: Posts that hint at a bigger idea but are too brief
• Series: Posts that should be linked into a series with a landing page

Focused Subset Extraction (No Inventory)

When the user provides a pre-curated list of posts (e.g., _meta/career-files.txt) or asks to extract concepts from a named tag/category, skip Phase 1 entirely and go directly to single-agent concept extraction. See references/focused-subset-extraction.md for the full workflow: audit existing concepts → batch-read posts → cross-reference → create → validate wikilinks.

Drift Prevention for Multi-Phase Extraction

When extracting from any large corpus (chat exports, blog archives, transcript collections),

create a task.md BEFORE starting extraction. This prevents drift, lost threads, and

scope creep. Joseph explicitly values this for complex multi-step work.

task.md Structure

# <Corpus> Extraction — Task Tracker

Started: <date>
Status: In Progress
Scope: <what's in scope, what's not>

Scope
IN SCOPE: ...
OUT OF SCOPE: ...

Drift Guards
1. Read before writing — no insights without source citations
2. Don't summarize what you haven't read
3. Stop at "good enough" — signal, not completeness
4. Cite sources — every output page has sources: in frontmatter
5. Check before creating — grep for duplicates
6. Time-box — stop after N items or M minutes, report progress

Phase 1: <metadata / quick wins>
- [ ] Task 1
- [ ] Task 2

Phase 2: <deep extraction>


  
Item
  
Date
  
Status
  
Notes



  
...
  
...
  
[ ]
  
...




Progress Log


  
Date
  
Action
  
Output





Acceptance Criteria
- [ ] Criterion 1
- [ ] Criterion 2

Key Principles

• Scope explicitly — list what's IN and OUT before starting
• Time-box deep phases — after 15 items or 30 min, stop and report
• Progress table for batch work — user can watch completion
• Acceptance criteria define "done" before you start

Independent vs. Guided Analysis

When the user asks to "analyze this corpus" or "what profile emerges from this data," they usually want independent analysis of the raw data — not a rehash of pre-existing summaries that came with the export. Claude exports include extracted/ folders with AI-generated analyses (corpus-analysis.md, memories.md, etc.). These are Claude's interpretation, not ground truth.

Rule: Raw Data First, AI Analysis Second

1. Identify what's raw vs. derived. Check for extracted/, analysis.md, *-analysis.md files. These are AI outputs, not source data.

2. Analyze raw data independently. Load conversations.json (or equivalent), extract human messages, run behavioral/textual analysis from scratch.

3. Compare with AI's analysis LAST. After your independent analysis, read the AI's pre-existing analysis to find what it missed or smoothed over. The delta between your analysis and theirs is often the most valuable insight.

4. Tell the user which you did. If you relied on pre-digested outputs, say so. Joseph corrected this directly: "Did you rely on Claude memory and profile or did you do this on your own from the corpus."

Behavioral Signal Extraction from Raw Messages

Techniques that revealed patterns invisible in AI-summarized analysis:

• Punctuation habits: Count messages with/without ending punctuation. Stream-of-consciousness writing (no punctuation) signals intimacy with the tool, not laziness.
• Abbreviation patterns: Track deliberate abbreviations ("abt", "shud") — these signal casual working relationships with AI.
• Message opening patterns: What words start messages most often? ("I want to", "if we were to", "ok", "what is the") reveals how the person frames requests.
• Conversation ending patterns: Does the person end with gratitude, a new task, a question, or a problem? Reveals whether conversations are consultations or work sessions.
• Correction language: Separate substance corrections ("that's wrong") from style corrections ("too verbose", "not chatty"). The ratio reveals trust level.
• Topic frequency in raw text vs. AI summary: Count keyword occurrences in human messages, compare with what the AI's analysis highlighted. Gaps reveal blind spots.
• "Act as" role assignments: What roles does the person assign to AI? Maps to capabilities they don't have themselves.
• Context-providing behavior: Do they state goals, paste code, provide URLs, describe current state? Reveals how they manage AI context.

See references/behavioral-signal-extraction.md for code patterns.

Directory Convention for Chat Exports

When setting up a chat export project (ChatGPT, Claude, or other), use this structure:

chatgpt/  (or claude/)
├── original/       ← Raw export files (DO NOT MODIFY — source of truth)
├── extracted/      ← Processed outputs only (index.csv, index.md, batch results)
├── scripts/        ← Python scripts (NOT in extracted/ — keeps code separate from data)
├── task.md         ← Phased plan with review gates
└── readme.md       ← Corpus stats, structure docs, how-to re-run

Why scripts/ not alongside data: Joseph prefers clean separation — extracted/ holds outputs only, scripts/ holds code. This scales as more phase scripts are added. Why readme.md: Durable context for future sessions. Capture: corpus stats (conversations, chars, date range), tier breakdown, tag distribution, directory map, how to re-run scripts. Prevents "what was this?" in later sessions. Extraction output filenames: Use sanitized conversation titles, not UUID prefixes. AI-Phases-of-Adaptation.md is browsable; 6fbfc9b5.md is not. The manifest JSON should include a sanitized_title field:

sanitized = re.sub(r'[^\w\s-]', '', title)
sanitized = re.sub(r'\s+', '-', sanitized.strip())[:60].rstrip('-') or 'unnamed'

When the title is empty (common in Claude exports — ~22% of conversations), use the summary field as fallback, then 'unnamed'.

ChatGPT projects = Custom GPTs. ChatGPT's conversation_template_id field links each conversation to the Custom GPT it was created in. Map these IDs to meaningful project names via GPT_PROJECT_MAP in the indexer script. 780 conversations had no custom GPT (mapped to 'general'); 19 Custom GPTs had 10+ conversations each. See references/chatgpt-export-format.md for the full mapping. Claude projects are standalone. Claude exports include projects/<uuid>.json files with project briefs and docs, but conversations in conversations.json have NO project_uuid field — there is no link between them. Projects must be extracted separately; cross-referencing conversations to projects requires content matching, not a foreign key.

Review Gate Pattern

Phase 1 (index/inventory) is always detailed. Phases 2+ are outlined at high level.

Each phase gets detailed planning only when reached. Review gate between phases:

user reviews Phase 1 output, adjusts tier assignments, agrees on Phase 2 batch sizes.

This prevents over-planning and ensures Phase 2 design reflects what Phase 1 revealed.

Pitfalls

• String iteration bug in auto-tagging: When checking if tags contain a prefix (e.g., any(t.startswith('routine/') for t in tags)), ensure tags is a list, not a pipe-delimited string. Iterating over a string yields individual characters, so the check silently fails. Always .split('|') first. Caught in production: 0 skip-tier conversations when there should have been 5.
• Subagent timeout is normal for clusters over 100 posts. Don't restart from scratch — integrate partial results.
• Post overlap between clusters means some posts get processed by multiple subagents. This is fine — different aspects get extracted. But it also means duplicate concept pages can result. Check for same-slug files.
• Don't let subagents touch index.md. Having 3+ agents update the same file causes collision, duplicate entries, wrong alphabetical ordering, and incorrect page counts. Parent handles index.md centrally.
• Podcast interviews need different treatment. Guest ideas aren't the author's IP. Extract only the author's lens.
• Check for existing concept pages before creating. The 2nd and 3rd batch must scan what Batch 1 created.
• Cluster assignment is fuzzy. Posts can legitimately belong to multiple clusters. The inventory should be treated as a guide, not ground truth.
• For focused subset extraction (pre-curated lists): audit existing concepts BEFORE reading posts. Batch-read with execute_code loops. Validate all wikilinks before reporting completion.
• Path resolution for nested scripts: Path(__file__).parent.parent resolves relative to the script's actual location. A script at chatgpt/scripts/batch_extract.py has parent.parent = chatgpt/, not agentic-chats/. Use parent.parent.parent when the script is 3 levels deep. Always test path resolution with a print statement before running the full script.
• Batch manifest handoff pattern: For overnight batch extraction, split into two steps: (1) a deterministic Python script reads index.csv, groups conversations into batches, loads raw text, and writes batch-NN/manifest.json; (2) subagents consume manifests via delegate_task. This keeps data prep deterministic and LLM extraction delegatable. The manifest JSON includes full conversation text plus a sanitized_title field for human-readable output filenames.
• Chat exports have structure beyond conversations — Claude exports include memories (AI's accumulated context), reflections (monthly meta-insights), and projects (structured work briefs). Extract these metadata files FIRST as quick wins before mining conversation threads. See references/claude-export-format.md for field-level details.
• conversations.json is single-line 32MB JSON — Cannot use head, cat, read_file, or line-based tools. Always use python3 -c "import json; ..." or execute_code with json.load(). Check size with ls -lh first.
• Read metadata before conversations — memories and reflections are highest value per byte. They give you the lens to interpret threads. Don't start with the 32MB conversations.json.
• "Good enough" stopping rule — for chat exports, many conversations are code-only with no strategic thinking. Skip them. The goal is durable insights (decisions, frameworks, evolution), not exhaustive coverage.
• Pre-existing analysis files are NOT the source. If the export includes extracted/, analysis.md, corpus-analysis.md, or similar AI-generated summaries, these are derived outputs — not ground truth. When asked for independent analysis, you must load and analyze the raw data (conversations.json) yourself. Relying on the AI's own analysis produces the AI's interpretation, not yours. Joseph explicitly corrected this: "I told you to do an independent analysis."
• Reading AI metadata first biases your analysis. The existing advice to "read memories/reflections before conversations" is correct for guided extraction (Phase 1 quick wins) but WRONG for independent analysis. When the user wants to know what they are like based on their data, reading the AI's pre-existing profile first makes you see what the AI saw, not what the data actually shows. Do raw analysis first, then compare.

Related Skills

• llm-wiki: Core wiki operations — SCHEMA.md, frontmatter conventions, page creation
• wiki-ingest-qa: Post-subagent QA — wikilink verification, index.md collision detection, timeout recovery, log consolidation
• plan: For writing a plan.md before executing if the corpus is very large (1,000+ posts)

Scripts & References

• scripts/corpus-inventory.py — Read all posts, extract frontmatter, tag frequency, date range
• scripts/cluster-posts.py — Assign posts to clusters based on tags + title keywords
• references/subagent-prompt-template.md — Template for concept extraction subagent prompts
• references/focused-subset-extraction.md — Workflow for single-agent extraction from pre-curated post lists (no inventory phase)
• references/claude-export-format.md — Claude chat export JSON structure, field reference, extraction patterns