In December 2024 Gartner predicted that "at least 30% of generative AI projects will be abandoned after proof of concept by end of 2025, due to poor data quality, inadequate risk controls, escalating costs or unclear business value." We see the same number in our practice — but not because of poor data quality. The real killers are organizational, not technical.
Here is the post-mortem of 14 pilots from Polish, German and US clients in 2023–2025. Three died. Two went into permanent zombie state ("we are still evaluating"). Nine shipped. The patterns are surprisingly consistent.
Mistake #1: pilot scope is too wide
A retailer asked us for "an AI agent that can handle all incoming questions." That's not a pilot, that's a 3-year roadmap. We narrowed it to "answer questions about order status and delivery times in Polish, escalate everything else." Pilot delivered in 5 weeks. Original brief would have taken 9 months and cost €120K.
Mistake #2: no baseline metrics
You can't prove an AI agent saved money if you don't know what was happening before. We had a logistics client who proudly reported "AI handles 60% of driver calls now." Great. What was the previous percent? Nobody had measured. The CFO killed the renewal.
Spend 1–2 weeks at the start measuring: call volume, average handle time, escalation rate, FCR, cost per ticket. Without these numbers, you have no story for the steering committee at month 6.
Mistake #3: bus factor of one
Three pilots stalled because the one engineer who understood the system left for another job. AI agents have implicit context: prompt engineering decisions, edge-case handlers, monitoring rules. Document everything in a runbook from week 1.
Mistake #4: synthetic data instead of production logs
Engineering team tested with 50 synthetic call transcripts they wrote themselves. Production calls had 4× more interruptions, accents the team didn't anticipate, background noise from open-plan offices, and 7% of callers using mixed languages (Polish + English). Accuracy dropped from 94% in tests to 71% in production.
Mistake #5: no fallback path
When the LLM is down, the agent silent-fails. We had a Friday evening incident where Claude API rate-limited us for 23 minutes. Without a fallback to human, the bot just hung up on customers. 240 lost calls.
Every production agent needs three fallbacks: a) retry with another model, b) graceful "let me transfer you" with hold music, c) post-incident SMS to follow up.
Mistake #6: paying for consulting, not for code
A vendor charged €40K for a "discovery phase" — 60-page document with personas, journey maps, and a roadmap. Zero code. Six months later the client had nothing in production. We rebuilt it in 5 weeks for €18K including the actual agent.
Discovery has value, but cap it at 10–15% of total budget. If a vendor wants more for "research," they're selling a deck, not a deployment.
Mistake #7: no human-in-the-loop
A pilot in healthcare auto-cancelled appointments based on AI interpretation of patient messages. After two weeks the patient relations team was buried in complaints. The fix was a 2-line code change: any cancellation triggers a confirmation message, then a 30-min hold before action. No more complaints.
Mistake #8: ignoring edge cases
Edge cases in voice agents we've hit in production:
- —Caller hands phone to a child mid-conversation
- —Background TV news triggers "transfer to a human" intent
- —Caller speaks 3 languages in one sentence (PL + UA + RU)
- —Phone rings on speaker in a car — engine noise breaks STT
- —Caller hangs up mid-sentence; agent keeps talking to dead line
- —Called number was reassigned — wrong customer answers
You won't catch these in testing. You catch them in the first 2 weeks of production with active monitoring. Allocate 20% of your week-1 budget for triage.
Mistake #9: no measurement post-deploy
Three of our clients shipped agents and then... never looked. Six months later they couldn't answer if it was working. Always set up: error rate alerting, weekly transcript sampling (at least 50 calls), and a monthly steering review with one person accountable.
Pre-kickoff checklist
| Item | Status | Owner |
|---|---|---|
| Single intent, single channel scope written | Required | Product |
| Baseline metrics measured for 1+ week | Required | Ops |
| Two engineers familiar with the system | Required | Eng |
| 200+ real production samples available | Required | Data |
| Fallback path defined and tested | Required | Eng |
| Discovery budget capped at ≤15% | Required | CFO |
| HITL approval flow for sensitive actions | Required | Compliance |
| Monitoring + weekly review scheduled | Required | Ops |
If you're already stuck
Most "stuck pilots" we adopt have one fixable cause — usually #2 (no baseline) or #5 (no fallback). Three weeks of focused work usually unstucks them. Send a 30-min screen recording of the agent failing — we'll tell you which of the 9 you hit.