In the era of AI-driven decision making, organizations are tempted to shortcut their data strategy by feeding large language models (LLMs) with unstructured documents such as PDFs. While this approach may seem convenient, it often results in poor performance, hallucinated outputs, and a lack of meaningful insight. Throwing PDFs into an LLM is not a valid data strategy.

We propose the need for semantic enrichment and domain-specific context as essential components of a scalable strategy. A key enabler in this transformation is the use of knowledge graphs, which organize information into interconnected entities and relationships. By grounding LLMs in structured, contextualized knowledge, organizations can drastically reduce hallucinations and improve the relevance and accuracy of AI-generated responses. We explore how knowledge graphs bridge the gap between raw data and intelligent reasoning, offering a foundation for trustworthy, enterprise-grade AI systems.

“Throwing documents at an LLM is like swapping the idol for a bag of sand – looks clever, but you’re still triggering all the traps. Structure your knowledge if you want to escape with the prize.”

In the era of AI-driven decision-making, organizations are tempted to shortcut their data strategy by feeding large language models (LLMs) with unstructured documents such as PDFs. While this approach may seem convenient, it often results in poor performance, hallucinated outputs, and a lack of meaningful insight. Throwing PDFs into an LLM is not a valid data strategy.

“A little error in the beginning leads to a great one in the end.” – St. Thomas Aquinas

When we provide an LLM with information in the form of text (either the whole text attached in the prompt or matching chunks through a RAG approach), we do reduce the risk of hallucination. The model’s responses will be somewhat closer to the source material. But let’s be clear: the LLM is still not using the information – it is interpreting it through its world model (which is still just a language model), and generating output based on that, i.e., statistical patterns.

That means hallucination risk never truly disappears. Even when the system returns a reference to the chunk from the document that was used as input, the burden of validation remains with you. You must read the referenced passage, interpret it, and verify that the model made the same assessment as you. Ouch. Every answer is still a gamble. Not scalable, not possible to automate, and not agentic compliant.

The alternative is to remove the knowledge domain from the LLM entirely. Let the LLM do what it excels at: understanding natural language questions and translating them into structured queries. But do not let it invent the answers. Instead, answers should come from a deterministic, rule-based, yet semantically accessible data layer – a knowledge graph or equivalent structured representation.

We propose the need for semantic enrichment and domain-specific context as essential components of a scalable strategy. A key enabler in this transformation is the use of knowledge graphs, which organize information into interconnected entities and relationships. By grounding LLMs in structured, contextualized knowledge, organizations can drastically reduce hallucinations and improve the relevance and accuracy of AI-generated responses. We explore how knowledge graphs bridge the gap between raw data and intelligent reasoning, offering a foundation for trustworthy, enterprise-grade AI systems.

The reality check: Testing AI with structured knowledge

To make this concrete, we set out to test the hypothesis that using an LLM to generate a knowledge model (and manually validating that), and then using an LLM to query the model for data to answer a question, is superior to letting the LLM ingest and assess the text directly.

To take an example: If you feed the annual report of a major car manufacturer into a cloud LLM and ask about numbers or strategy, you’ll often get decent answers. Today’s large models can parse tables, charts, even images. But you’ll still feel the need to double-check every response – which is exactly the point of this article. To make it clearer (and avoid wading through 14 megabytes of corporate boilerplate), we used a short excerpt with a diminutive local model, where hallucinations are easier to spot.

We tested a local LLM (Gemma 3:1B) with a short excerpt from MegaCorp’s annual report .

The prompt: Show all subsidiaries with ESG contributions

The reply came back smoothly:

Convincing – but wrong. SubZ had no ESG contributions in the source. The model simply invented a plausible answer.

Next, we asked the LLM to build a graph from the same document: nodes, edges, nothing else. After validating that graph, we let the LLM query it. This time it issued a Cypher query and returned only the subsidiaries with HAS_VISION actual ESG contributions. No hallucinated SubZ. And as a bonus, instead of getting merely a reference to a text chunk, you can easily ask the application to also HAS_SUBSIDIARY return the query (Cypher/SPARQL) as well as the result.

The point is simple: Querying a document index means validating every answer. Querying a graph means validating the graph once.

This illustrates the key point: When you query a document index, you validate every answer. When you query a graph, you validate the graph once.

Conclusion

At its heart, this isn’t just about documents versus graphs. It is about decoupling the LLM from the knowledge domain, and where you place the burden of validation.

• Document-first RAG forces validation at the very end of the pipeline – every answer must be checked, every hallucination caught, every gap patched. It scales poorly and exhausts human oversight and is not agentic compliant.

• A separated data/knowledge model approach moves validation upstream. Once information is structured, linked, and governed in a graph, the downstream systems – LLMs, agents, dashboards – can operate reliably on a deterministic knowledge base.

This shift enables modularity. In a world of autonomous agents and automated decision-making, you want each layer of your architecture to do one thing well and pass on clean, validated outputs. Ontology evolution, knowledge ingestion, reasoning, and generation become separate modules – each improvable, replaceable, and automatable.

The principle is simple: validate early, scale later. By moving the validation point as far upstream as possible, you gain the ability to automate reasoning chains, orchestrate agentic frameworks, and build enterprise AI systems that are not just powerful, but trustworthy.

Throwing PDFs at an LLM may give you an answer. Building a modular knowledge model gives you a system.

Key take aways:

  • Stop dumping PDFs into your LLM
    Treat unstructured documents as raw input, not a ready-made knowledge layer.
  • Validate upstream
    Structure and enrich your data – supported by LLMs – into a knowledge graph; so hallucinations are caught once, not every time you query.
  • Let LLMs translate, not invent
    Use them for natural language understanding and query generation – but keep answers grounded in deterministic, governed data.

Read the full report Data-powered innovation review – wave 11