Building an Enterprise Knowledge Assistant with RAG

Large language models are excellent at general knowledge up to their training cutoff, but they know nothing about your quality procedure, clause 7 of a customer contract, or the price list you updated last month. Ask the model directly and it will either admit it doesn't know or, worse, invent a plausible-sounding but wrong answer. In business settings this hallucination risk is the single biggest barrier to adopting AI.

Retrieval-Augmented Generation (RAG) solves the problem without retraining the model. When a user asks a question, the system first searches the company's document archive for relevant passages, then passes those passages to the model together with the question and instructs it to answer only from those sources. Answers stay current, every claim can be traced back to a specific document and page, and when a document changes you only update the search index, not the model.

Consider a logistics company in Mersin: customs notes, carriage contracts, bill-of-lading samples, and internal procedures are scattered across hundreds of PDF and Word files. When a new operations employee asks 'Which documents are required for a container carrying dangerous goods?', a RAG assistant can find the relevant procedure and return a short summary with a link to the source. Senior staff save time, and knowledge stays with the organization instead of with individuals.

The quality of a RAG system usually depends more on how the data is prepared than on the model. The first step is converting documents to text: digital PDFs can be extracted directly, while scanned documents need OCR (for example Tesseract) or layout-aware tools such as Docling or Unstructured. Tables, headings, and page numbers must be preserved at this stage; otherwise questions like 'What is the value in Table 3?' become impossible to answer correctly.

The second step is splitting text into meaningful chunks. Chunks that are too small lose context; chunks that are too large drown the model in irrelevant text and hurt retrieval precision. In practice, 300 to 800 tokens per chunk, respecting heading boundaries and with a small overlap between neighbors, is a good starting point. Each chunk is enriched with metadata such as document name, section title, page number, date, and access permissions.

Metadata is not only for citations; it is essential for access control. Payroll documents from accounting and maintenance manuals from the field team may live in the same index, but results must be filtered by the user's permissions at query time. Otherwise the assistant may 'summarize' information the user would never be allowed to open, which is also a data-protection concern under KVKK and GDPR.

Each chunk is turned into a numeric vector by an embedding model. Texts with similar meaning land close together in vector space, so a user searching for 'invoice dispute' can also find a paragraph about 'the complaint process for a disputed bill'. For Turkish and mixed-language content, multilingual embedding models should be preferred, and the choice should be validated on a small test set built from the company's own documents.

Vectors are stored in a vector database such as PostgreSQL with pgvector, Qdrant, Weaviate, or Milvus. For an SME already running PostgreSQL, pgvector is the most practical way to start without learning new infrastructure. When the corpus grows to millions of chunks or latency becomes critical, dedicated vector databases and approximate nearest neighbor (ANN) indexes such as HNSW come into play.

Vector search alone is not always enough: for exact-match queries such as product codes, order numbers, HS codes, or personal names, classic keyword search (BM25) performs better. Production systems therefore typically use hybrid search, combining vector and keyword results, followed by a reranker model that picks the 5 to 10 most relevant chunks. This extra step often improves answer quality more than switching to a larger model.

The selected chunks are passed to the model with a carefully written system instruction: 'Answer only from the provided sources; if the sources do not contain the answer, say so; cite the source number next to every claim.' This constraint greatly reduces fabrication from the model's general knowledge and lets users verify an answer with a single click.

Long context windows (128K tokens and beyond) make 'just put the whole document in the prompt' tempting, but that increases both cost and latency, and models are known to overlook information buried in the middle of very long contexts ('lost in the middle'). Giving the model only what it needs through good retrieval is faster and more accurate. We explain the technical background in the attention article of this series.

For numeric questions we recommend an additional safeguard: critical values such as stock levels, balances, or delivery dates should be fetched directly from the ERP or database through tool calling rather than from text. The model then acts not as something that 'remembers' data, but as an interface that retrieves it from the right source and explains it.

A RAG system should be measured before it goes live. Build an evaluation set of 50 to 200 real company questions with known correct answers, then measure retrieval accuracy (is the right chunk in the top results?), faithfulness to the sources, and answer correctness separately. Open-source tools such as Ragas make this easier. Rerun the same set whenever the chunking strategy, embedding model, or prompt changes to prevent regressions.

In production, track user feedback (thumbs up/down), unanswered questions, and the most frequent topics. Unanswered questions usually point to missing or outdated documents, so a RAG system also exposes gaps in the company's knowledge management. Set up a synchronization pipeline (for example, watching a shared folder or SharePoint) so the index is refreshed automatically when documents change.

At MeSoft we build RAG assistants for manufacturing, logistics, agricultural export, and service companies in and around Mersin, running entirely on in-house servers so that data never leaves the company. A pilot on a small document set can show results within a few weeks, which is the safest way to see concrete value before committing to a larger investment. Earlier parts of this series cover on-prem LLM infrastructure, PDF reporting, inference optimization, and the attention mechanism in depth.