Grounded Answers From Your Own Documents

Almost every company wants the same thing right now. Point an AI assistant at the company documents and let anyone ask it questions. The first demo feels like magic. Then it gives a confident, fluent, completely wrong answer to a real customer and the magic turns into a problem.

The trouble is that a plain language model will always answer. Ask it anything and it produces something that sounds right, whether or not it has any basis for it. That is fine for brainstorming. It is dangerous for a support desk, a policy question or an internal knowledge base, where a wrong answer given with confidence is worse than no answer at all.

This article walks through the system that fixes it. The pattern is called retrieval-augmented generation and the idea behind it is simple to state. The assistant is only allowed to answer from passages it actually found in your documents, it shows you where each answer came from and when it cannot find good support it says so instead of guessing.

The problem: Why a chatbot on your docs is risky

A language model does not know what it does not know. When you ask it about your refund policy or a clause in a contract, it does not look anything up. It predicts the most likely sounding answer from its training and it delivers that answer in the same calm, authoritative voice whether the answer is right or invented. The industry word for the invented ones is hallucination and they are the single biggest reason document assistants fail in production.

In a low-stakes setting that is a nuisance. In support, compliance or anything a customer acts on, it is a real risk. The fix is not a smarter model or a longer prompt. The fix is to stop letting the model answer from memory at all and to make it answer only from your approved content, with a citation you can check.

The blueprint: The idea in one picture

The system has two phases. The first happens once, when you set it up: you turn your documents into a searchable index. The second happens on every question: the assistant finds the most relevant passages, decides whether they are good enough and answers only from them. Here is the shape of it.

Index your documents once, then answer each question from what you retrieve. The confidence gate decides whether to answer with sources, fall back to a general reply or admit the answer is not in the knowledge base.

The one rule that makes the whole thing trustworthy is this. The assistant may only answer from what it retrieved and it always shows the sources. If it cannot find good support, it does not fall back on its imagination. It tells you.

Phase 1: Build the index, once

Before the assistant can answer anything, it needs to be able to search your documents by meaning, not just by keyword. That is what the indexing phase sets up and you only run it when content is added or changed.

First the raw content is extracted and cleaned, whether it comes from PDFs, web pages, plain text or a list of questions and answers. Then each document is cut into small overlapping pieces called chunks, because a whole manual is too big to search or to feed into a model at once. Each chunk is then turned into an embedding and stored in a database built for fast similarity search.

What is an embedding? An embedding is a way of turning a piece of text into a list of numbers that captures its meaning. Passages about the same topic end up with similar numbers, even when they share no words in common. That is what lets the assistant find the passage about "getting your money back" when someone asks about "refunds," without either side using the same phrase.

What is a vector store? A vector store is a database designed to hold those number lists and find the closest ones in milliseconds, even across millions of chunks. Here it is Supabase with the pgvector extension and an index called HNSW, which is what makes the search fast enough to feel instant.

# One time: turn your documents into searchable, embedded chunks
text   = extract(source)                 # from PDF (PyPDF2), web (safe crawl + BS4) or plain text
chunks = smart_chunk(text, size=1024, overlap=128)   # Q&A-aware, then semantic

for chunk in chunks:
    vector = embed(chunk, model="text-embedding-3-small")   # 1536 numbers
    vector_store.add(text=chunk, embedding=vector)          # Supabase pgvector, HNSW

Phase 2: Find the right passages

Now a question arrives, by chat or by voice. The first thing the agent does is decide whether the question even needs a search. A greeting or a follow-up like "thanks" does not, so it skips straight to a normal reply and saves the effort. When a search is warranted, it looks for the most relevant passages in two complementary ways at once.

Why hybrid search? Vector search is excellent at meaning but it can miss an exact term, such as a product code, an order number or a specific clause name. Keyword search is the opposite. It nails the exact term and misses the meaning. Running both together and merging the results catches what either one alone would let slip through.

Hybrid search returns a generous shortlist, around twenty candidates. That list is good but not precise enough to answer from directly, so a second, sharper model reorders it.

What is reranking? The first search is fast and approximate. Reranking is slow and precise. It takes the question and each candidate passage and reads them together, then scores how well the passage actually answers the question. The best five rise to the top. Doing a cheap wide search first and an expensive precise ranking second is how you get both speed and accuracy.

# The agent first decides whether this message even needs a search
if not needs_search(question):        # a greeting or a "thanks" does not
    return general_reply(question)

hits = hybrid_search(question, top=20)                      # meaning (vector) + exact words (keyword)
top5 = rerank(question, hits, model="cohere-v3", keep=5)   # read query + passage together

The gate: Answer, hedge or admit it does not know

Now comes the step that separates a trustworthy assistant from a confident guesser. Before writing anything, the system looks at how relevant the best passages actually are. If they are strongly relevant, it writes an answer using only those passages and attaches the sources. If they are weak, it can offer a careful general reply and flag that it is unsure. If there is nothing relevant at all, it says the answer is not in the knowledge base.

# Only answer if the best passages are actually relevant
best = top5[0].relevance

if best >= 0.9:
    answer = generate(question, context=top5, cite=True)   # answer only from these passages
    return grounded(answer, sources=top5)
elif best >= 0.5:
    return general_reply(question, note="low confidence")   # optional soft fallback
else:
    return "I could not find this in the knowledge base."

When the gate does allow an answer, the instruction given to the model is strict. It may use only the passages provided, it must cite them and it must refuse if they do not contain the answer. That single instruction, enforced on every question, is what keeps the assistant honest.

# The instruction that keeps the model honest
SYSTEM = (
    "Answer the question using ONLY the passages provided below. "
    "Quote the source for each claim. "
    "If the passages do not contain the answer, say you do not know. "
    "Never use outside knowledge and never guess."
)

In practice: Two honest answers

Here is what the two paths look like in practice. The first question has a clear answer in the documents. The second does not and the assistant handles that just as cleanly.

Question · "What is our refund window?" · Grounded · relevance 0.94

Answer
Your refund window is 30 days from the delivery date. Refunds are issued to the original payment method within five to seven business days.

Sources

• Returns Policy, section 3
• Support FAQ, "Refunds and returns"

Answered from your documents, with citations the customer can open and check. No outside knowledge was used.

Question · "Do you ship to Antarctica?" · Not in KB · relevance 0.21

Answer
I could not find this in the knowledge base, so I do not want to guess. I have passed the question to the team so a person can follow up.

No invented shipping policy. The gap is logged, so someone can add the missing answer and the assistant will handle it next time.

The payoff: What makes it accurate

The first and most important gain is trust. Grounding the model in retrieved passages, rather than letting it answer from memory, is what pulls the rate of made-up answers down to something you can live with.

The accuracy of the answer depends on putting the right passage in front of the model and each retrieval stage improves the odds. Vector search alone is good, hybrid search is better and reranking on top is better still.

Put together, the assistant answers most questions straight from your documents, hedges on a few and refuses the ones it genuinely cannot support. That last group is small and every question in it is a pointer to content worth adding.

These numbers are illustrative and yours will depend on how complete and well-organised your documents are to begin with. The pattern holds regardless. The better your source content, the more questions land in the green.

Applied: Where this fits your business

The same system fits any place where people ask questions that your documents already answer. Customer support that deflects the repetitive tickets, an internal assistant that lets staff find a policy without pinging a colleague, a compliance helper that answers only from approved wording and an onboarding guide for new hires all follow the identical shape. Because the assistant works over chat or voice, it can sit on your website, in your help desk or on the phone.

Doing it well takes a few things done properly rather than anything exotic. Your content has to be organised and current, the chunking and retrieval have to be tuned to how your people actually phrase questions and the confidence gate has to be set where a refusal is preferable to a risky guess. That is the work we focus on at Neulaxy: building knowledge assistants that answer from your data, show their sources and are honest about their limits.

The takeaway: Wrapping it up

The value here is not a chatbot that can talk. Talking is easy and it is exactly what gets companies into trouble. The value is an assistant that will only tell you what it can prove from your own documents, that hands you the source so you can check it yourself and that is willing to say it does not know. That restraint is the whole product. Trust is the feature you are actually shipping.

If there is one idea to carry away, it is this. Do not ask whether the assistant can answer. Ask whether it can show you where the answer came from. If it cannot, it should stay quiet.