Context & Engineering Objective
This individual ME 4853 project addressed a core limitation of traditional retrieval-augmented generation (RAG): a long technical document is usually split into independent text chunks and ranked by similarity. That approach can retrieve a relevant HVAC clause while omitting the parent scope, exception, numbered condition, or referenced table that determines how the clause should be interpreted.
The system preserves those relationships through a hierarchy of chapters, sections, subsections, and requirements in Neo4j Aura, paired with lexical and vector retrieval. Canonical local records retain cross-references, tables, equations, and source passages. The engineering objective was to improve section-level retrieval over a 306-page source while tracing each released code claim to its supporting text, section number, and PDF page.
What I Built
Built the complete GraphRAG and evidence-grounded RAG workflow from PDF ingestion through Neo4j data modeling, hybrid retrieval, answer validation, evaluation, and the Streamlit review interface.
- Parsed and audited 897 numbered sections while preserving nested headings, lists, exceptions, tables, equations, cross-references, and original page locations
- Modeled the document in Neo4j Aura as connected
Document,Chapter,Section, andRequirementrecords - Implemented graph lookup, semantic vector search, and a hybrid search that combines multiple ranked result lists
- Built an evidence ledger that assigns stable IDs to source blocks and checks quotations, numbers, units, and semantic support before release
- Created a Streamlit review workspace with section search, source-page viewing, tool history, citation dependencies, checkpoints, and bounded runtime budgets
System Architecture
The architecture separates finding evidence from trusting it. Neo4j Aura acts as a searchable map of the document: chapters contain sections, sections can contain subsections, and sections state requirements. A separately audited canonical corpus retains the complete source text. The graph locates candidates; the corpus supplies the evidence used in the answer.
GraphRAG combines those document relationships with semantic search. A ReAct controller observes each search result and decides whether to query the graph, run a vector search, combine both rankings, or ask for clarification.
- Graph search: bounded, read-only Cypher for document hierarchy, known section numbers, and structured requirement lookup
- Vector search: meaning-based retrieval through a 3,072-dimensional embedding index when the wording of the question differs from the source
- Hybrid search: reciprocal-rank fusion that balances lexical, full-text, and vector candidates before expanding parent sections and referenced tables
- General information: a deliberately limited path for basic HVAC concepts that does not present general knowledge as code evidence
The controller is allowed to change search strategy after seeing a result, but it cannot run indefinitely: each question has a six-tool-call ceiling plus query, time, and payload limits. Complete source blocks receive stable evidence IDs, and section/page citations are rendered only after validation.
Validated Ingestion & Provenance
Before graph loading, the parser output was compared with an independent inventory of the PDF. The audit found 897 numbered section headings across eight chapters, 30 table captions, 15 equation identifiers, 125 exception blocks, and 861 explicit references. The canonical output contains 34 tables, 543 table rows, 15 equations, and 1,568 normalized reference records.
The structural audit recovered 100% of the inventoried headings, table captions, equation identifiers, reference occurrences, and exception blocks. That is a coverage result—not a claim that every clause was interpreted perfectly. Unmatched or ambiguous content remains visible as a review item instead of disappearing silently, and flagged or unassigned requirements are kept out of authoritative retrieval.
Grounded Answer Pipeline & Inspector
Each checkable code statement in a draft answer must name an evidence ID and include the exact supporting quotation. Deterministic checks reject unknown evidence, altered quotes, missing numeric values, and mismatched numeric/unit pairs for recognized units. A separate semantic review checks meaning, unit interpretation, scope, and necessary qualifications. If a gate fails, the system abstains or asks for clarification. Recognized basic HVAC concepts use a separate set of authored explanations and do not establish code requirements.
The Streamlit workspace makes this process reviewable. A user can search every numbered section, open the exact PDF page behind a citation, inspect which retrieval tools ran, follow section-parent links, export the execution record, and remove a reference experimentally to see which claims lose support. The inspector shows evidence dependencies; it does not claim to reveal private model reasoning.
Engineering Decisions & Trade-Offs
The design uses explicit release gates and measurable trade-offs instead of treating retrieval and generation as a single black box.
1. Preserve structure before adding AI
Decision: Keep section hierarchy, lists, exceptions, tables, and page provenance as first-class data. Why: the exception several lines below a rule can reverse its meaning. Trade-off: ingestion is more complex, but the system can reconstruct the context a similarity-only search may omit.
Finding and fix: A source-position audit found ten equations attached to the wrong section. Ownership and reverse links were repaired using the physical PDF heading positions, and the parser now applies the same rule. Regression checks verify that each formula appears under its correct section and is excluded from the neighboring section.
2. Use the graph for discovery, not as the final source of truth
Decision: Treat graph and vector results as candidates, then expand them from the audited corpus before answering. Why: short graph passages are useful for search but may exclude a parent clause or referenced table. Trade-off: maintaining both representations adds work, but final claims remain tied to complete source text.
3. Allow adaptive search, but bound it
Decision: Let the controller switch among graph, vector, and hybrid retrieval after observing results, with a six-call ceiling and query limits. Why: section-number questions and conceptual questions need different search strategies. Trade-off: the workflow is more flexible than a fixed router, while the limits keep cost and failure behavior predictable.
4. Validate claims and investigate rejected answers
Observed trade-off: In the 120-case fixed-retrieval answer comparison, fused hybrid retrieval produced 40 answers that passed automated source checks and 80 abstentions or clarifications; vector-only retrieval produced 46 source-checked answers. The rejection count motivates investigation of both insufficient evidence and false rejections. It does not establish that all rejected drafts were incorrect.
Finding and fix: Local chatbot testing exposed conflicting citation instructions and altered source quotations. Structured claims now keep evidence IDs separate from displayed text and copy quotations verbatim. A separate Section 303.3 draft omitted the enclosure door's energy-code air-leakage condition; the answer instructions now explicitly preserve every exception condition in the displayed claim. Retests returned supported answers for Sections 303.7 and 303.3 without relaxing validation.
Decision: Retain exact evidence binding, recognized numeric/unit pair checks, and semantic qualification review. The revised code passes 106 offline regression tests. Remaining measurement: These local fixes have not been evaluated by rerunning the full 120-case comparison, so no new acceptance rate is claimed.
5. Measure retrieval separately from answer correctness
Observed gap: In a separate historical benchmark of 20 questions, retrieval found the expected evidence for all 20, while 18 answers were citation-valid and 17 were manually judged grounded-correct. Evidence coverage measures whether the needed source was found; grounded correctness measures the answer itself. Those historical counts have not been rerun after the latest fixes.
Decision: Measure retrieval and answer support separately, then validate each draft claim against complete source context. Reciprocal-rank fusion raised mean section recall@5 from 28.6% to 79.1% versus legacy concatenation and matched vector-only recall. Trade-off: Additional source checks can reject a draft even when retrieval finds the right section, making false-acceptance and false-rejection measurement part of the next evaluation.
6. Separate application guardrails from database security
Decision: Restrict generated Cypher to one bounded read query and reject writes, procedures, UNION, comments, variable-length traversal, and expensive operations. Why: model-generated queries require a narrow execution envelope. Boundary: these checks do not replace database permissions; a restricted Aura identity is still required before deployment.
Results & Validation
Retrieval impact Increased mean section recall@5 by 50.5 percentage points—from 28.6% to 79.1%—by replacing result concatenation with reciprocal-rank fusion. The frozen held-out set contains 120 questions; recall is averaged over its 110 source-backed cases, with 10 ambiguous or out-of-scope cases evaluated separately. Recall@5 measures whether expected sections appear within the top five results. Fused hybrid retrieval matched vector-only recall while reducing lexical crowd-out.
Source coverage Audited 897 numbered sections across eight chapters in the 306-page source, preserving 34 tables, 543 table rows, 15 equations, and source-page provenance. The independent structural audit recovered every inventoried heading, table caption, equation identifier, reference occurrence, and exception block.
Execution and regression checks All 360 fixed-retrieval answer checks across three retrieval methods completed without service errors. The latest code passes 106 offline tests, with local chatbot retests confirming cited answers and original PDF-page access.
Measurement scope Recall measures section retrieval, structural coverage measures source accounting, and regression tests exercise specific runtime behavior. Answer-quality findings and their fixes are described in the engineering trade-offs above; independent expert answer accuracy remains a separate evaluation.
Impact, Limitations & Next Steps
The project demonstrates an end-to-end GraphRAG architecture for converting a complex technical PDF into source-linked answers. Its engineering strength is the integration of audited document structure, multi-channel retrieval, reciprocal-rank fusion, deterministic evidence binding, conservative abstention, and a review interface.
The remaining limitations define the next engineering phase. The source edition and jurisdiction still need independent authority verification. The held-out questions need expert labels, the strict validator abstains frequently, preserved parser omissions are not automatically usable requirements, and the inspected Aura identity still had administrative privileges. The next steps are a read-only database identity, expert-reviewed evaluation questions, false-acceptance and false-rejection measurement, and a controlled ablation of hierarchy expansion.
