Compartmentalising research project memory

The problem: CLAUDE.md works well for simple projects. A research project with real data has five interlocking memory needs that a single file cannot serve without becoming unmanageably long — and length causes Claude to quietly ignore whatever appears later in the file.

The five needs:

The solution: Keep CLAUDE.md as a routing document. Give each type of knowledge its own file. Claude reads the right file for the right session — not everything, every time.

Three layers of project memory

Most researchers fill Layer 1 (CLAUDE.md) and Layer 3 (meta-documents). The domain-knowledge layer in the middle is the gap.

Layer What it contains When Claude reads it
Instructions — CLAUDE.md Project identity, folder map, session routing, conventions, what not to do Always, automatically
Domain knowledge — _logic/ or named spec files Data definitions, analysis plans, transformation logic, variant rules Explicitly: via @ import or session instruction
Meta-documents — _history.md, _ownership.md, etc. Development record, attribution, concept index On demand; not every session

A real example: the _logic/ directory

The trialinter project — a computational history study of interrogation dynamics in medieval heresy trial records — provides a working implementation of this pattern. Its _logic/ directory contains six modular specification files, each addressing a distinct analytical dimension:

File What it specifies
clause.segmentation.logic.md What constitutes a clause; the question-envelope structure; the governed-by hierarchy
narrative.agency.logic.md Two agency dimensions (discourse frame vs. grammatical subject); how they propagate through clause chains
dependent.variable.logic.md DV filter; empirical distinction between two question types; what the DV does and does not capture
content.type.scope.logic.md Content-type assignment; why clause-level cross-tabs reverse at deposition level (Simpson's paradox)
register.structure.logic.md The three corpus registers; the T77b genre anomaly; what registers proxy and confound
sequential.analysis.logic.md Valid vs. confounded sequential predictors; how to construct envelope-level alternatives

Each file follows the same structure: problem → evidence → recommendation. Each file identifies not just what to do, but what not to do. The sequential.analysis.logic.md file, for example, marks specific predictors as CONFOUNDED with empirical evidence (OR values, ICC, confounding mechanism) and provides VALID alternatives. Without this file, Claude would confidently build models using the confounded predictors — and the output would look plausible.

A README.md inside _logic/ indexes the files, defines the reading order, and points to the extraction prompt that is the authoritative source of truth for the coding scheme.

The naming is not critical. This can be _logic/, _specs/, _domain/, or simply named files at the project root (data-description.md, analysis-plan.md). What matters is the separation from CLAUDE.md.

What each file does

Data description (data-description.md)

One section per dataset: name, source, date range, unit of observation, key variables with types and meaning, known issues, file path. The trialinter version includes per-register tables: N depositions, mean clauses per deposition, question rate, proportion zero-Q depositions.

Variable-level description is essential. Claude hallucinating variable names is one of the most common failure modes in data work — a plausible-looking column name that does not exist in the actual dataset, causing silent errors downstream.

This file is the single source of truth for "what does this field mean?" Update it when new data arrives. Never duplicate variable definitions elsewhere.

Analysis plan (analysis-plan.md)

The first line links to the proposal: "This plan implements the research design in _proposal.md."

Structure: research questions → hypotheses → planned analyses → operationalisation decisions → what counts as confirmation vs. disconfirmation.

Include a ## Plan changes section. This records deliberate changes to the analytical strategy with reasons — different from _history.md, which records everything done. This is the provisionality record: when the plan shifts (a new analysis is added, a hypothesis is reformulated), the change and its rationale appear here. This is how the analysis plan stays honest across months of iterative work.

This is not a results file. Results go in _history.md or a separate findings.md.

Transformation log (transformation-log.md)

The most underrated file in data-heavy research.

The top of the file states the current canonical dataset: "Current canonical: outputs/corpus_v3_anon.csv as of 2026-03-17."

Then each transformation step:

## Step 3: Geocoding
Input:  outputs/corpus_v2_deduplicated.csv (12,441 clauses)
Operation: matched location strings against GeoNames gazetteer;
           unmatched entries flagged for manual review (n=34)
Output: outputs/corpus_v2_geocoded.csv
Rationale: preserve raw coordinates for sensitivity analysis
Date: 2026-02-14

Research projects routinely accumulate 4–6 transformation layers (raw → deduplicated → geocoded → merged with census → aggregated → anonymised). Without a log, Claude cannot help debug the pipeline when something breaks in step 4, and cannot reproduce step 3 six months later. Skipping this file is the most common regret in long-running data projects.

Data variants (data-variants.md)

Named variants with purpose, path, creation date, and decision rule:

## Variants

| Name | Path | Purpose | Use when |
|------|------|---------|----------|
| raw | data/raw/corpus.xml | Original source | Debugging only; contains PII |
| deduplicated | data/interim/corpus_dedup.csv | Dedup applied | Data cleaning sessions |
| analytical | data/final/corpus_v3_anon.csv | **Current canonical** | All analyses |
| sensitivity_full | data/sensitivity/corpus_v3_full.csv | Includes T77b register | Sensitivity analyses only |

## What not to do

- Do not use `raw` for any analysis — it contains witness names and locations

- Do not treat `sensitivity_full` as the default — T77b is a different genre

The explicit "valid vs. confounded" pattern from trialinter applies here: name each variant, state its purpose, state when not to use it. A file named data_final_v2_REAL_use_this_one.csv is a symptom of this file not existing.

Project proposal / protocol (_proposal.md)

The anchor for the analysis plan. The analysis plan should begin by citing the proposal; they cross-reference each other.

Convert to markdown with markitdown if the proposal is a PDF or DOCX. Keep it at the project root. Reference it in CLAUDE.md: "When scope questions arise, consult _proposal.md."

The proposal is not updated as the project evolves — that is what the analysis plan's ## Plan changes section is for. The proposal is the original commitment. Drift from it is worth noticing explicitly.

CLAUDE.md as a routing document

At this scale, CLAUDE.md's job is to tell Claude which files to read for which task. Claude should not need to infer this from context.

## Session routing

**Data cleaning session:**

- Read `@transformation-log.md`

- Read `@data-variants.md`

- Do not introduce new hypotheses in this session

**Analysis session:**

- Read `@analysis-plan.md`

- Read `@data-description.md`

- Confirm which dataset variant before proceeding

**Writing session:**

- Consult `_proposal.md` when scope questions arise

- Use `_history.md` for results so far

## What not to do

- Do not rename output files without updating `data-variants.md`

- Do not add predictors without checking `_logic/sequential.analysis.logic.md`

- Do not interpret clause-level cross-tabs as deposition-level evidence

The @ import syntax pulls files into the session context automatically. Without explicit routing, Claude will work from whatever context it has — which for a data project is never enough.

Practical principles

Infrastructure follows discovery. Do not create all files upfront. Create each file the moment you notice Claude working without it and making mistakes. The moment you catch Claude using the wrong variable name is the moment you write data-description.md. The moment you catch Claude running an analysis on the wrong dataset variant is the moment you write data-variants.md.

One source of truth per fact. Variable definitions live in one file. If they appear in CLAUDE.md, in the analysis plan, and in a comment in a notebook, they will drift apart. Duplication is the beginning of inconsistency.

The ## What not to do section. Every domain knowledge file benefits from explicit anti-patterns. Domain knowledge files in trialinter systematically flag: confounded predictors (with OR values), Simpson's paradox pitfalls (with the clause-vs.-deposition reversal), and DV scope errors (what the DV does not capture). Without this, Claude produces plausible-looking but subtly wrong outputs — and the outputs look correct precisely because Claude does not know what it does not know.

Session routing is explicit. CLAUDE.md tells Claude what to read per session type. If this is left implicit, Claude reads whatever was loaded last session, which may be wrong for the current task.

The transformation log saves the project. Researchers who skip it consistently encounter the same situation: six months later, a collaborator asks how a particular dataset was produced, and the answer requires reconstructing the pipeline from commit messages and memory. Write the log as you go.

What this looks like in practice

A DISSINET project analysing network data from medieval sources might have:

project/
├── CLAUDE.md                    ← routing document
├── _proposal.md                 ← funding proposal converted to markdown
├── data-description.md          ← source inventory + variable glossary
├── analysis-plan.md             ← hypotheses + operationalisation + plan changes
├── transformation-log.md        ← current canonical: network_v4_anon.csv
├── data-variants.md             ← raw / cleaned / anonymised / teaching subset
├── _history.md                  ← session-by-session development record
├── data/
│   ├── raw/                     ← original sources (gitignored if PII)
│   ├── interim/                 ← intermediate transformations
│   └── final/                   ← canonical analytical dataset
└── notebooks/                   ← analysis notebooks

Or, following the trialinter pattern, a _logic/ directory for a project with complex domain specifications:

project/
├── CLAUDE.md
├── _logic/
│   ├── README.md                ← index + reading order
│   ├── data.model.logic.md      ← relational structure, unit of analysis
│   ├── coding.scheme.logic.md   ← annotation decisions, category definitions
│   ├── dependent.variable.logic.md
│   └── confounders.logic.md     ← valid vs. problematic predictors
├── data-variants.md
├── transformation-log.md
└── _proposal.md

The exact structure does not matter. The principle does: each type of knowledge has one home, CLAUDE.md knows where each home is, and every session begins with the right files loaded.

Related