An internal knowledge assistant is only as useful as the documents behind it. If it answers smoothly while hiding stale or missing sources, it can make the knowledge base harder to trust.
Build the first version as a source-finding assistant. It should help someone locate the relevant policy or guide, quote the useful section, and say when the collection does not contain an answer.
Clean the library before adding a model
Remove duplicate documents, identify owners, add revision dates, and separate current guidance from archives. Set access permissions before indexing. A search layer should not make a document visible to someone who could not open it directly.
Define what a good answer contains
- A short answer in plain language.
- One or more links to the source documents.
- A quoted or summarized passage that supports the answer.
- The source's last-updated date when available.
- A clear statement when the evidence is missing or conflicting.
Keep retrieval and generation separate
First retrieve a small set of authorized passages. Then ask the model to answer only from those passages. Log the source identifiers used for the response. This makes it possible to inspect whether a poor answer came from finding the wrong document or from summarizing the right one badly.
Give documentation an operating rhythm
When a user marks an answer as wrong or unhelpful, route the feedback to the document owner. Review unanswered questions each month. Retire or update pages that are frequently retrieved but often corrected. The assistant is also a sensor for the health of the knowledge base.
- Search respects the source document's permissions.
- Answers link to the evidence they use.
- Stale and conflicting sources are visible.
- Users can report a wrong or missing answer.
- Document owners receive actionable feedback.