Choose a specific question
An AI project is easier to evaluate when the task is concrete. For example, helping staff find answers in approved internal documents is clearer than asking AI to transform an entire company. Define the users, the source material and the kinds of questions the assistant should handle.
Design the review before the automation
Decide who checks an answer, what the system should do when it is uncertain and which actions require approval. Use realistic test examples and include cases where the right answer is to ask for more information. Keep access to source material aligned with existing permissions.
Evaluate before expanding
Compare the pilot against agreed examples, then review usability, operating cost and the effort still needed from staff. Expand only after the team understands the limitations and the process has a clear owner. A useful pilot creates evidence for the next decision.
Example: an assistant for approved company documents
Consider an internal assistant that helps employees find approved procedures. Begin with a small document collection, named owners and questions that employees really ask. The assistant should point to its supporting material and explain when an answer is outside the available sources. This is a proposed pilot pattern, not a claimed FMN client result.
Keep permissions aligned with the source documents. A useful answer must not reveal a confidential document to an employee who could not otherwise access it.
Build an evaluation set before the demo
Include questions with clear answers, ambiguous requests, outdated information and questions that should not be answered. Have a responsible reviewer assess correctness, source relevance and the need for human escalation.
- Separate factual retrieval from actions that change business records.
- Review handling of sensitive information and access restrictions.
- Record latency, operating cost and the effort needed to correct answers.
- Test requests that try to override the assistant’s instructions.
Decide what makes the pilot ready to expand
Agree on acceptance criteria before reviewing results, including which errors are unacceptable and which actions always require approval. Store evaluation examples so changes to prompts, documents or models can be checked against the same tasks.
Expand only when the team understands the remaining limitations and somebody owns document updates, monitoring and incident response. A polished conversation is not sufficient evidence of reliable performance across the intended workload.
