A quick primer before the walkthrough:

In plain English: An illustrative workflow for using Claude with a structured JSON schema to extract candidate genes, drugs, doses, and adverse events from PDF documents for non-clinical exploratory research.

Key terms:

• NER (named entity recognition) — identifying and pulling targeted categories (e.g., drug names, doses, gene symbols) out of unstructured text.
• Schema — a predefined template specifying required field names, data types, and formatting for structured output.
• SciSpaCy — an open-source Python library designed for biomedical named entity recognition; this guide illustrates an alternative prompt-based approach.

What you'll need: Access to a Claude account and a small set of sample PDFs (e.g., 5–20 open-access publications, drug labels, or case reports).

Comfort level: No coding background is required for basic prompt testing. Light Python knowledge is helpful if you choose to automate downstream file handling.

Data responsibility: Do not upload confidential, sensitive, or protected health information (PHI) unless an appropriate agreement or organizational data governance framework is in place. Refer to your organization's active vendor terms and internal data governance policies regarding document processing.

New to this? "Entity extraction" just means pulling specific facts out of free text — drug names, doses, cell lines, adverse events — into tidy columns you can sort in a spreadsheet. The hard part is not the AI; it is deciding precisely what counts as a valid entry before you start.

Biomedical entity extraction with Claude

Traditional NLP libraries (such as SciSpaCy or BERN2) are effective tools but require a Python execution environment and ongoing software maintenance. For non-clinical exploratory research synthesis, using Claude with a strict JSON schema offers an accessible alternative for generating structured candidate datasets.

Note: This tutorial is for informational and exploratory research purposes only and does not constitute medical, clinical, diagnostic, dosing, legal, or regulatory advice. All extracted information must be verified by qualified specialists against official primary sources.

When Claude may be suitable

• Extracting candidate entities from small-to-medium document sets for exploratory synthesis.
• Supporting human-in-the-loop workflows where expert oversight is applied to all outputs.
• Internal exploratory research where outputs are subject to expert evaluation rather than formal regulatory submissions.

When to use specialized NER tools or dedicated pipelines

• Automated batch processing of high-volume document collections.
• Direct integration into production pipelines without human-in-the-loop validation.
• Validated-records requirements. Generative LLMs do not natively satisfy GxP, HIPAA, or regulatory validation standards.

Step 1 — Define the schema

Below is an illustrative schema prompt designed to request a structured JSON array from attached documents:

For each attached paper, return a JSON array with objects:
{
  "source": "filename.pdf",
  "genes":   [{"symbol": "...", "context_quote": "..."}],
  "drugs":   [{"name": "...", "dose": "...", "route": "...", "context_quote": "..."}],
  "adverse_events": [{"term": "...", "grade": "...", "context_quote": "..."}],
  "populations": [{"description": "...", "n": ...}]
}
Only include entities explicitly stated in the source text. context_quote must be a verbatim sentence.

Step 2 — Batching strategy

As an illustrative starting point, consider testing small batches (e.g., 5–10 PDFs per prompt turn). Optimal batch size depends on document length and token limits, as larger uploads may lead to context degradation depending on document length.

Step 3 — Normalize terminology

You can prompt the model to align raw extracted terms with controlled vocabularies for internal review:

Normalize every "drugs.name" to its INN. If the paper uses a code name (e.g. AZD1234), 
keep both fields: "name_as_written" and "name_inn".
Normalize every "adverse_events.term" to CTCAE v5 terminology where possible.

Disclaimer: Standardized term mappings generated by language models are illustrative and must be verified against official terminology registries (such as WHO INN or NCI CTCAE) prior to downstream use.

Step 4 — Quality assurance and sampling

Implement a human-in-the-loop quality check based on your organization's internal standard operating procedures (SOPs) and project goals.

Illustrative QA approach: As an example workflow, a research team might select a random sample (e.g., 5% of extracted entities) to manually verify against source PDFs. Precision targets depend on project requirements and internal SOPs. If accuracy falls below your project threshold, consider refining the prompt schema, adding negative constraints, or splitting extraction into separate passes by entity type. All extracted dosage or adverse event data must be verified by a qualified researcher before downstream decision-making.

Step 5 — Export structured output

Return the combined dataset as CSV, one row per entity, with a source_page column.

Potential failure modes

• Unextracted tabular data: Language models may skip content inside embedded PDF tables. Include explicit prompt instructions if table parsing is required.
• Contextual confusion: Terms like "Grade 3" can refer to adverse event severity or tumor grade. Requesting surrounding sentences helps human reviewers clarify context.
• Biological vs. pharmacological entities: Biological targets (e.g., PARP) may be categorized solely as drug targets. Prompting for both biological classifications and target roles helps capture full context.

Cost estimation and resource planning

API costs depend on document page count, average token length, model selection, and current vendor pricing schedules. To estimate job costs, multiply expected input and output token counts by the rates listed in your organization's current vendor agreement or service portal. For large document collections, consider running a small test batch to measure token consumption, splitting long PDFs into target sections, or executing entity extraction in staged passes.

If something goes wrong

• The output is not valid JSON. Give one complete worked example in the prompt, ask for JSON only with no commentary, and reject anything that fails to parse rather than hand-fixing it.
• The same drug appears three different ways. That is a normalization problem, not a model problem — map to a controlled vocabulary after extraction, not during it.
• Fields come back empty for records that clearly contain the value. Your schema wording is probably ambiguous. Rewrite the field description as an instruction a new hire could follow, and say explicitly what to do when a value is absent.
• Accuracy drops on longer batches. Shrink the batch. Ten documents extracted well beats fifty extracted loosely.
• You cannot tell how good the extraction is. Hand-label a random sample (25–50 records is a reasonable start) and score against it before trusting any of it.

Plain-English glossary

• Entity — a single fact you want pulled out, such as a drug or a dose.
• Schema — the exact list of fields and allowed values you asked for.
• Normalization — mapping different spellings of the same thing to one standard term.
• Gold set — a small hand-checked sample used as the yardstick.
• Precision / recall — how much of what it found was right, and how much of what was there it found.