How can companies make their knowledge accessible in a way that allows it to be retrieved efficiently and used easily? This question is of timeless importance for companies, especially as the complexity and diversity of corporate knowledge increase along with the company’s size. Since the advent of ChatGPT, artificial intelligence (AI) has established itself as a powerful tool for quickly and precisely accessing information.
However, tools like ChatGPT or Gemini generally have no access to the knowledge that makes a company unique—the valuable information hidden in company policies, internal wikis, or other documents. This is exactly where this article comes in: By intelligently linking a company’s own knowledge sources with AI, not only is accessibility improved, but the value and relevance of knowledge management within the company are also sustainably enhanced.
Retrieval-Augmented Generation (RAG) has established itself over the past two years as a key technology for integrating AI with a specific knowledge base. This approach makes it possible to generate precise answers based on company-specific sources such as Confluence or file archives containing PDFs. This article explains the basic principles of RAG, highlights its advantages over public AI chatbots, and presents a concrete example from IT Sonix: an internal chatbot that uses information from Confluence.
Isn't that what ChatGPT is for?
AI systems such as ChatGPT or Gemini are based on publicly available data and impress with their ability to provide a wide range of knowledge quickly and easily. However, when it comes to specific, internal business issues, these systems naturally reach their limits. They are unfamiliar with a company’s individual processes or specific challenges, as this data is generally not included in their training datasets. In a specific business context, such systems therefore offer only limited value, since users must research and provide the necessary information themselves. Even if AI models have been trained with specific information, internal company processes and data are subject to constant change. Retraining such models is technically challenging, resource-intensive, and often not even possible for public AI models.
In addition, we must ask ourselves what information should even be shared with a public provider of AI systems. If the data is not to leave the company, then the LLM would have to be operated in-house (e.g., by using an open-source model).
AI that can seamlessly access company-specific knowledge sources has the potential to fundamentally transform how information is handled within an organization. It provides precise and context-sensitive answers tailored to the company’s specific needs. Employees benefit from easy access to knowledge by being able to ask for it using natural language. This speeds up processes, boosts productivity, and simultaneously increases the value of knowledge management. Only through the efficient use of internal data sources does the sustainable maintenance of the knowledge base become viable. These advantages make AI with domain knowledge an essential tool for the future.
Retrieval-Augmented Generation (RAG)
Retrieval-Augmented Generation (RAG) is a method that combines the rapid retrieval of relevant information (information retrieval) from data sources with text generation by generative AI. The retrieved information is used to generate specific answers. Instead of merely presenting search results from knowledge sources or relying exclusively on the training data of AI models, RAG combines the advantages of both approaches to deliver context-aware responses.
Based on a query, relevant information is retrieved from internal data sources. To do this, the information is first transferred from the source systems to a vector database. This process involves two key steps: First, the information is broken down into smaller chunks (chunking); then, these chunks are converted into numerical vectors (embedding). These numerical representations make it possible to efficiently find information that semantically matches the query.
The information found is then forwarded, along with the original query, to the system’s „brain“—the Large Language Model (LLM). Unlike the information from training data, which is stored structurally within the AI model, the data from the knowledge base is processed dynamically as part of the user’s query. In the final step, the AI can generate the response based on this information.
The benefits of RAG are immediately apparent:
• Precision: Answers are based directly on domain-specific sources and are therefore highly relevant.
• Timeliness: Information from existing data sources can be used immediately without requiring any adjustments to the AI model.
• Flexibility: New data sources can be seamlessly integrated, allowing for a dynamic response to changing requirements and knowledge bases.
IT Sonix Chatbot
To take advantage of RAG at IT Sonix, an internal chatbot was developed. This chatbot draws on information from the internal Confluence system to generate context-aware responses. The data is automatically retrieved via the Confluence API, processed using techniques such as chunking and embedding, and stored in a vector database. Metadata, such as links to the original Confluence page, is added to ensure the source is traceable. This makes it possible to find semantically relevant information quickly and accurately.
Example: IT Sonix Chatbot:
When a query is submitted, the vector database is searched for relevant information to serve as the basis for the response. This response is then generated using open-source AI hosted by IT Sonix itself, ensuring that internal information does not leave the company. In addition to the generated response, the chatbot provides links to the relevant Confluence pages so that users can view the sources directly. In this way, the IT Sonix chatbot makes corporate knowledge specifically available, ensures traceability, and at the same time does not share any data with external systems.
Challenges and Outlook
Although RAG is a promising technology, there are challenges that must be addressed in order to realize its full potential:
Sensitivity: Finding the right information can depend heavily on the exact wording of the user's query. This can result in the relevant context not being found. Improvements such as re-ranking algorithms, context-based enhancements to individual chunks, or additional tagging can help solve this problem.
Safety considerations: Safety mechanisms for AI outputs (guardrails) and smart rights management must be established to ensure responses that are appropriate for the company. Users should only be able to access information that has been approved for them. This can be implemented by integrating Active Directory (AD) and including the corresponding metadata for access rights in the knowledge base. At the same time, processes must be in place to enable rapid updates, such as when user access is blocked or modified.
Continuous Improvement: The rapid advancements in generative AI and RAG offer numerous new possibilities, but also require ongoing evaluation. Approaches such as agent-based RAG, which integrates reflection, planning, and the use of additional tools such as internet searches or the creation of visualizations and graphics, significantly expand functionality but must be understood and used appropriately.
Despite these challenges, it is clear that RAG is a forward-looking technology that, with the right optimizations and security measures, offers tremendous value to businesses.
