If you are considering AI that finds internal documents and answers questions, the first question may not be which model to use. Once you have worked out who can see which materials, where the authoritative versions are kept, and what counts as a good answer, you can set a concrete scope for a pilot.
01. Start by narrowing to recurring questions
Rather than connecting all of your organization’s knowledge from the start, choose questions that come up again and again in a specific department. Areas with a clear owner and clear reference documents, such as business rules, product descriptions, and operating manuals, make a good starting point.
Look not only at how long it takes to find an answer but also at the impact of giving wrong guidance. For work that needs approval or judgment, first decide the points at which the person in charge checks the answer.
What to prepare
- A list of questions real users ask often
- The documents behind each correct answer and the department that owns them
- Questions to refer to staff instead of answering
02. Check document status and the authoritative version
If several versions of the same rule are mixed together, or a title alone doesn’t show what a document applies to, search results can be confusing as well. Document the authoritative version, effective date, document owner, and update cycle for each one.
Materials in different formats, such as tables, images, and scanned documents, may need different ways of extracting content and checking quality. Start by selecting a few representative document types and reviewing how they are processed.
What to prepare
- Separating official reference documents from archived copies
- Checking file formats and whether their content can be extracted
- Updating what search covers when documents are revised or deleted
03. Search must follow existing permissions
Even with a single search box, not every document can be shown to every user. Check how access is divided by department, role, and project, and design the permission rules to apply at the search stage.
Also agree on the scope of question and answer history, the information sent to external models, and the records operators can view. Decide where data is kept and how it is provided based on your actual operating policies.
04. Review answers together with their sources
Consider a setup that links each answer to its place in the original text and to the reference document, so users can check where it came from. Even when a source is shown, you still need to check separately that the answer matches the original.
For questions with no supporting source in the materials, design the system to hold back an answer or refer the user to staff. Include outdated documents, conflicting rules, and requests outside a user’s permissions in the evaluation.
05. Pilot with a small evaluation set
Prepare an evaluation set that groups representative questions with their expected answers and the sources to check. Include not only questions that search handles well but also vague wording, topics the materials don’t cover, and questions that require comparing several documents.
In the pilot, check whether sources are appropriate, whether search results respect permissions, when answers are held back, and what users say. Based on the results, widen the document scope and question types step by step.
That’s all you need before a consultation.
If you prepare the items below, we can align on scope more quickly in early discussions. Even if not everything is ready, we can start with your current situation.
- Departments and question types to start with
- Owner, version, and update method for each reference document
- Access permissions by user and data policy
- Evaluation questions and staff review criteria
- Who runs the service after the pilot
KPS project guide · The specific technical setup and conditions depend on your requirements.
