A document assistant should cope with questions its sources cannot answer. Include those questions in acceptance rather than evaluating only successful retrieval.
Define reference cases
Subject owners provide answers and supporting passages. Keep evaluation cases separate from preparation examples. Check that users can access the documents.
Classify failures
A missing source, incorrect passage and unsupported explanation need different changes. Record each separately. A polished response should not conceal absent evidence.
Scope the next improvement
Ask the partner to demonstrate corrected failures and their limits. Maintain examples when documents change. This process gives your business a useful evaluation set and avoids accepting an assistant merely because its best answers read naturally.
Related reading : Luxembourg business software: scope a multilingual AI pilot. Discuss your project scope with Sky Vaults.