Builder SouryaneelPal built InsightRX multi-agent chest radiograph triage system that runs entirely on local models, verifies its own reports before a radiologist ever sees them, and reaches prior imaging through a NitroStack MCP server, without sending a single DICOM file to an external API.
What this enables
- Classifies and localizes chest X-ray findings without any data leaving the machine running Ollama.
- Blocks a case from proceeding until DICOM metadata, burned-in pixel text, and free-text PHI are all confirmed scrubbed.
- Rejects any report citation that does not trace back to a real, signed guideline passage.
- Queries a patient's prior imaging through a single MCP tool instead of a custom PACS integration.
- Requires a role-checked radiologist sign-off before any diagnosis is finalized.
The workflow this replaces
Chest radiograph triage sits in an uncomfortable gap. Hospitals have more scans than radiologists have hours, and a backlog on a chest film means a slower read on a case that could be pneumonia. The obvious fix, routing images through a hosted LLM API, runs straight into HIPAA and PHI handling rules. The other common fallback, an on-prem CAD tool, tends to hand back a probability score with no reasoning a clinician can check against.
The project, built as a research prototype and documented as such in its own README, takes a different route: keep every model call local, and make the system show its reasoning and its evidence at each step rather than a single opaque number.
Why a single model call was not enough
A chest X-ray finding is not just a label. A useful triage report needs a structured finding, a confidence score that is actually calibrated rather than raw softmax output, a citation back to a real clinical guideline, and a check that the citation and the finding do not contradict each other. Asking one model to generate a report and then grade its own report is a weak safety design, because the same model that produced an error is also the one being trusted to catch it.
The system separates these jobs into distinct agents wired together as a LangGraph state machine: a Diagnosis Agent, an Evidence Agent, a Report Agent, and a Verifier Agent that runs deterministic code checks before any LLM ever reviews its own output. Every case also has to clear a PHI de-identification gate first. DICOM tags are scrubbed with pydicom, burned-in pixel text is caught with EasyOCR, and free-text fields are redacted with Presidio and spaCy. If de-identification cannot be verified, the case does not move forward.
How the pipeline is built
A submitted radiograph moves through a fixed sequence. The perception layer, built on a torchxrayvision DenseNet-121 backbone pretrained on real chest X-ray data rather than generic ImageNet weights, classifies the image and generates a Grad-CAM heatmap. A Faster R-CNN detector attempts localization, though the repository is explicit that the detector head has no trained checkpoint yet; rather than quietly returning a meaningless score, run_perception() returns None and a provenance flag surfaces that to the console.
From there, the Diagnosis Agent turns the vision output into a structured finding with a calibrated confidence score. The Evidence Agent retrieves matching passages from an HMAC-signed FAISS index built over ATS and IDSA guideline PDFs, so the eventual report cites retrievable source text instead of relying on the model's memory. A PACS retrieval step then checks for prior imaging on the same patient. The Report Agent drafts a structured report with inline citations, and the Verifier Agent runs a three-stage check: an abstention gate for low-confidence cases, deterministic schema and citation-grounding checks, and only then an LLM semantic review. A defect that can be fixed sends the case back to the Report Agent, capped at three attempts.
Every case still ends at a human. graph.interrupt() pauses the LangGraph run and persists state to SQLite, so a case can sit for hours without losing progress. Only a session with the radiologist role can approve or revise a diagnosis; an admin session can reject a case for manual workup but cannot approve one, which mirrors the real distinction between system administration and clinical authority.
Where NitroStack fits
Prior imaging context matters for a triage read, but hospitals do not want a bespoke PACS integration for every AI tool that touches their imaging archive. The project handles this with a NitroStack-based MCP server that exposes exactly one tool, query_prior_studies, over stdio. The Python side spawns it as a child process per call rather than keeping a persistent connection open, a deliberate trade of a small amount of latency for process isolation on a lookup that is normally sub-second anyway.
The current PACS server is a fixture, not a live hospital connection, and the project marks that fact end to end. Every simulated study carries a simulated:true flag inside the MCP payload itself, not a constant sitting in the UI code, so the marker goes away on its own once a real PACS is wired in instead of depending on someone remembering to delete it. The report-drafting prompt is told about the same flag and instructed not to make comparative claims, such as "unchanged from prior," off fixture data.
What the design choice is actually doing
The interesting part is not that an MCP tool returns prior studies. It is where the honesty boundary sits. An unreachable PACS server degrades to prior_studies=None and the case still proceeds, but the system keeps None and an empty list as separate facts, because "the archive was unreachable" and "this patient has no prior studies" are different clinical statements. A tool boundary like this is a natural place to encode that distinction, because the MCP call is the one point where an external, unreliable system enters an otherwise local, deterministic pipeline.
What the prototype demonstrates, and what it does not
The repository is upfront that this is a research prototype, not a validated clinical product. Every quantitative figure in the project, including the model card, is generated from a synthetic RSNA-style dataset and watermarked "SYNTHETIC, NOT CLINICAL EVIDENCE." No component has FDA clearance or CE marking, and the detector head is untrained. What the build does demonstrate is a working pipeline: PHI de-identification as a hard gate, a signed retrieval index that fails closed on tampering, a citation-grounding check that treats an unsupported citation as the most severe defect class, and several hundred passing tests covering the privacy, security, and RBAC layers.
The reusable pattern
The takeaway is not "AI can read chest X-rays." It is that a safety-critical pipeline benefits from putting an unreliable external dependency, like a PACS lookup, behind a narrow MCP tool with an explicit degrade path, instead of letting that dependency's uptime determine whether the rest of the system runs. The same pattern applies to any agent workflow that needs an external system's data but cannot let that system's availability decide whether the core logic proceeds.
Read the source on GitHub, or explore the NitroStack SDK to build an MCP server around your own external dependency.