All notes / artificial intelligence
Human-in-the-loop: how to supervise AI answers in a Service Desk
What to review, who reviews it and when to trust a generated answer.
Generative AI already drafts incident summaries, suggests articles and answers questions in the self-service portal. Done well, it saves minutes on every ticket. Poorly supervised, it confidently repeats a procedure that stopped existing six months ago.
AI is only as good as the knowledge behind it
Tools like ServiceNow Now Assist or Microsoft Copilot answer from the information they find in your sources. If the article is outdated, the answer will be too, just better written. That’s why knowledge governance is the first quality control for AI.
Three points of supervision
- Before: decide which sources the AI may use. Only published, current articles with an owner. Leave out drafts, retired content and sensitive internal documents.
- During: the agent reviews and edits the suggestion before sending it to the user. The AI proposes; the person decides.
- After: review a sample of answers every period, together with user feedback and the cases where the agent had to correct the suggestion.
When to grant more autonomy
Start with high-volume, low-risk topics backed by stable, well-rated articles: setting up MFA on a phone, connecting a printer, installing a VPN client. Anything involving privileged access, security or personal data keeps mandatory human review.
What to measure
- Share of suggestions accepted without changes.
- Edits the agent had to make, and why.
- Answers reported as wrong, and the article behind them.
- Effect on resolution time and user satisfaction.
Human-in-the-loop isn’t distrust of AI: it’s designing the exact place where a person adds judgment.