B. Data Analysis: Execution, Interpretation, and the Feedback Signals

Purpose: Analysis is where research produces its findings — but it is also where the distinction between what AI can do and what you must do is most consequential. Processing data and interpreting data are categorically different acts. Conflating them is one of the most common and most damaging errors in AI-assisted research. This document draws that line clearly, covers the specific cautions around qualitative analysis, and describes how analysis generates the feedback signals that return the project to earlier phases.

Part of the B.lifecycle series.

The central distinction: execution vs. interpretation

The clearest framing of AI's role in analysis comes from political scientist Tom Pepinsky: AI can execute analysis; the interpretation is your intellectual work and your responsibility.

Execution — running a statistical test, generating a visualisation, applying a coding scheme to a text, calculating frequencies, producing a summary table — has verifiable outputs. You can check whether the code ran correctly, whether the counts are right, whether the output matches the input. AI is highly productive here.

Interpretation — what does this pattern mean for your research question, what do the outliers suggest, does this result confirm or complicate your hypothesis, what is the relationship between this finding and the existing literature — cannot be verified against the data alone. It requires your field knowledge, your judgment about what matters, and your accountability to a scholarly community. AI is fragile here.

The danger is not that AI interpretation is always wrong. It is that AI interpretation is fluent and confident, which makes it hard to scrutinise. A well-phrased interpretation of a statistical result, generated by Claude, reads like a finished conclusion. It may be correct. It may be subtly wrong in ways that only become visible to a reader who knows the field. The researcher who treats it as finished, because it sounds finished, has delegated an act of judgment that was theirs to make.

The discipline: treat every AI-generated interpretation as a hypothesis. Read it, evaluate it against what you know, decide whether you agree and can defend it, then own the conclusion — or replace it.

Quantitative and structured analysis

Running quantitative analyses, transforming structured data, generating visualisations, and producing summary statistics — these sit clearly on the engineering-like end of the gradient. AI assistance is productive, and the outputs are verifiable:

Errors in execution are detectable in principle — which makes this a manageable domain for AI assistance. Check the outputs against your data and your expectations; flag anything that looks wrong before interpreting.

In Claude Code specifically, this verifiability goes further than a one-off check. Producing a visualisation or running an analysis usually means Claude writes an actual script — Python, R, whatever fits the task — that performs the transformation, rather than generating an answer directly. That script is a real research artifact: you can read it, rerun it once the underlying data changes, hand it to a collaborator, or include it in a methods appendix. A researcher with no GIS background asking Claude Code to turn a spreadsheet of place names and dates into an interactive map ends up with both the map and the code that produced it — a reproducible pipeline, not a one-off picture. This is a different trust model from Claude Desktop simply describing a result: the analysis is not just verifiable in principle, it is inspectable and repeatable in fact, the same way any other piece of research code is.

The interpretive acts — what the regression tells you about your hypothesis, what the pattern in the frequencies means for your argument, what the visualisation reveals that you did not expect — remain yours.

Qualitative coding: specifically complex

Qualitative coding — applying analytical categories to interview transcripts, field notes, historical documents, or other text — is one of the most common analytical tasks in social science and humanities research. AI assistance here is possible but involves specific complications that deserve explicit attention.

Productivity: If your coding scheme is explicit and operational — categories are clearly defined, distinguishable from each other, and grounded in the material — Claude can apply them at scale. First-pass coding across a large corpus, which would take weeks manually, can be done faster. You verify and correct the output rather than doing the initial coding from scratch.

The reliability problem: Inter-rater reliability — the standard method for establishing that qualitative coding is consistent and not idiosyncratic — assumes two human coders working independently. Claude is not a rater in the traditional sense. It does not have consistent independent judgment; its coding can vary across sessions, and it cannot be assessed for reliability using standard metrics. This matters especially if your method section will need to report reliability.

The validity problem: Does Claude interpret your categories the way you intend? Categories in qualitative research often carry theoretical weight that is not fully captured in a written definition. A human coder trained in your framework brings that theoretical understanding; Claude works from the text of your definition, which may not fully convey the underlying concept. Spot-checking Claude's coding against your own judgment — not just for errors but for the interpretive logic it is applying — is essential.

The disclosure requirement: Using AI for qualitative coding is a methodological choice that belongs in your methods section. Journal and funder policies increasingly require description of AI tools, prompts, and parameters used in analysis (see E.journal-funder-policies). Document your scheme, your prompts, and your verification process as you go, not retrospectively.

The productive workflow: Use Claude for first-pass coding; reserve final coding judgments for yourself; document the scheme and prompts; report the verification process in the methods section.

Pattern-finding at scale

One of the most compelling uses of AI in analysis is surfacing patterns across a corpus too large to read comprehensively. Claude can identify recurring themes, flag anomalies, note which sources cluster around similar concerns, and draw attention to what appears rarely when you might expect it frequently.

These are starting points for your analysis, not conclusions. A pattern Claude identifies is a prompt for your investigation — why does this cluster here? what explains the absence there? The answer to those questions requires your interpretive judgment and your domain knowledge. But the prompt itself, which might have taken months of reading to arrive at, can come quickly.

The anomaly is specifically worth noting: AI trained to find patterns tends to highlight what fits a pattern and underweigh what does not. If your research question depends on taking exceptions and outliers seriously — and in historical and interpretive research it often does — be deliberate about asking Claude what does not fit, not just what does.

Analysis generating feedback signals

Analysis is not the end of data collection — it is a continuous generator of signals that can redirect the whole project. These signals are among the most important information your research produces:

The category that keeps not fitting. You designed a coding scheme, you apply it, and one category consistently produces edge cases or requires forced assignments. This is not a coding problem — it is a conceptual problem. The category may not carve the phenomenon at its joints. The feedback signal points back to the theoretical framework, back to the research design, sometimes back to the research question.

The finding that contradicts the hypothesis. The expected pattern does not appear, or appears reversed, or appears only in a subset of cases that makes the generalization untenable. This is the most productive kind of failure in research. It redirects: do you need more data to be confident, different data to understand the mechanism, or a reconceptualization of what you were actually looking for?

The result that cannot be interpreted. You have a statistically significant result, or a pattern across a corpus, and you do not know what it means. This is not a failure of analysis — it is often a signal that the connection between your data and your research question needs to be rethought. The question may need to be reformulated in terms of what the data can actually speak to.

The gap that becomes visible. Analysis often reveals what you do not have. A comparison you want to make requires a variable you did not collect. An interpretation you want to defend requires a source you have not found. These are return signals to data capture.

AI-assisted analysis makes these signals legible faster — not because AI identifies them (that is your interpretive work) but because faster processing means you encounter the pattern, the contradiction, or the gap earlier in the project, when redirection is less costly.

The provisionality principle

In an iterative research process, analysis produces provisional findings — not conclusions. A provisional finding is something that looks significant enough to build on, but that you hold open to revision as further material comes in or as the manuscript-writing process reveals what it actually implies for your argument.

This is methodologically honest and practically important. Claude can help you reason about provisional findings: what would confirm them, what would disconfirm them, what they require you to find in the remaining data, what they imply for the shape of the final argument. Treat Claude as a thinking partner for this — not as an authority on what the finding means, but as a way to think through its implications systematically before committing to a direction.

Slide: Corpus workflow — first-pass categorisation through anomaly detection to interpretation

Related