Recently, we had a request from a client for an AI assistant that would answer questions using only their approved documents.
The important part was that the AI should not just use its general knowledge. The answers needed to come from the client’s own content.
My first thought was that we would probably need to train an AI model specifically for this.
Then I started researching different ways to approach it and came across RAG (Retrieval-Augmented Generation).
I realized that we don’t always need to train the AI to know the documents. Instead, we can let it search the documents when someone asks a question and give the relevant information to the AI before it answers.
That sounded much simpler and more practical for this type of use case.
So how does it work?
The basic idea is quite simple:
Documents → Search → Relevant information → AI → Answer
For example, imagine a company has 500 pages of approved documents.
Instead of sending all 500 pages to the AI every time someone asks a question, RAG first finds the parts that are most relevant to the question.
The AI then uses those parts to write the answer.
I explained the basic RAG process in my previous note: What Is RAG? A Simple Explanation of Retrieval-Augmented Generation.
I wanted to see it working
To understand the idea better, I experimented with a small RAG application.
I used AI coding tools to help with the implementation, but my main focus was understanding how the different parts work together rather than writing all the code myself.
For the demo, I uploaded a PDF and asked questions about its content.
I used Harry Potter and the Deathly Hallows as the test document. I chose a book because I could not share the actual company documents or the original client use case.
What happens when the PDF is uploaded?
The PDF is first processed page by page.
The text is divided into smaller pieces called chunks. These chunks slightly overlap so that information is not lost between two pieces.
Each chunk is then converted into an embedding, which is a numerical representation of its meaning.
These embeddings are stored in a vector database.
In my test, the 610-page PDF produced around 1,748 passages.
The database also keeps information such as the page number for each passage.
What happens when I ask a question?
When I ask a question, the system also turns my question into an embedding.
It then searches the vector database and finds the passages that are most similar to the question.
In this demo, it retrieves the 6 most relevant passages.
Those passages are then sent to the language model together with the question.
The model is instructed to answer only using the provided passages.
It also includes the page number in the answer when possible.
So the flow is roughly:
My question → Find relevant passages → Send those passages to the AI → Generate an answer based on those passages
The important part is that the model does not receive the whole book. It only receives the relevant parts found by the search.
This is one of the things I found most interesting about RAG.
What if the answer is not in the document?
This is also important.
A good RAG system should not try to answer everything.
If the information is not in the provided documents, the system can be instructed to say that it could not find the answer instead of making something up.
This is one way RAG can help reduce hallucinations.
Of course, RAG does not automatically guarantee perfect answers. The quality of the search, the documents, the instructions, and the language model all matter.
Why I think RAG can be useful
After looking into it, I started seeing many situations where this approach could be useful.
For example:
- A company knowledge base
- An AI assistant for internal documents
- A chatbot that answers based only on a company’s website
- Product documentation
- Customer support
- Policies and guidelines
- Educational platforms
- Any system where the information can change over time
One thing I liked about RAG is that if the documents change, you can update the knowledge base instead of retraining the whole AI model.
For many use cases, this can be a faster and more practical approach than training a model specifically for the information.
Access and usage
There is also another practical part to these systems.
If the platform is going to be used by customers, you don’t want everyone to have unlimited access to the AI service.
One possible approach is to give each customer a unique token.
The backend can validate the token and connect it to that customer. The token can also be stored securely as a hash instead of storing the original value.
This makes it possible to control usage per customer, for example by setting limits on how many questions they can ask.
So the solution is not only about the AI part. There are also questions around access, security, and usage limits.
What I learned
The biggest thing I learned from this was that sometimes the first solution that comes to mind is not necessarily the best one.
I initially thought:
We need to train the AI on the documents.
After learning about RAG, I understood that another option is:
Let the AI search the documents when it needs them.
That difference sounds small, but it changes the whole approach.
I’m still learning about AI and RAG, but I found this especially interesting because it connects something I was learning with a real client requirement.
And that is probably the part I enjoy most: understanding what a technology can actually be used for, rather than just learning the technology itself.
The demo
For the demo, I uploaded the PDF and asked questions about the book to see whether the system could find the right information and answer based on the document.
I also tested questions where the answer should not be available in the book, to see how the system handles information it cannot find.
