The AI advisory market is full of people who know how to talk about AI without having implemented anything. This is partly the market’s fault — the demand for AI guidance outpaced the supply of people who have done real implementations, so the gap got filled.
Buying well means knowing the difference between someone who talks about AI and someone who has shipped it.
The five questions that reveal the difference
1. “Tell me about a specific implementation that didn’t go the way you expected.”
Anyone with real implementation experience has multiple examples. The details should be specific: what failed, what was done about it, what the client had to accept. Someone without real experience will give you a vague answer about “challenges” and pivot to a success story.
2. “What percentage of your clients run their systems 12 months after we deliver?”
This is the adoption question. A 12-month follow-up is easy to ignore if you’re a project-based firm. Firms that measure it care about outcomes, not deliverables.
3. “What would you recommend we not do?”
Good advisors have opinions about what doesn’t work for specific contexts. If the answer is “we’re vendor-agnostic” without substance behind it, they’re not making recommendations — they’re facilitating.
4. “Who does the work?”
This one is simple but critical. At many larger firms, the senior person sells the engagement and a junior person delivers it. Know upfront whether the person you’re meeting is the person you’re hiring.
5. “How do you measure success?”
The answer should include a measurement you can verify independently. If success is defined as “deliverables completed,” that’s a project manager, not an advisor.
How to structure the engagement
A well-structured AI advisory engagement has these properties:
- A defined diagnostic phase. Before any implementation work, the advisor should produce a clear diagnosis of the problem — what workflows to address, in what order, and why.
- Milestones tied to business outcomes. Not “deliver the system” but “the system processes 80% of intake without manual intervention.”
- An explicit handoff plan. Who owns the system after the engagement ends? What training does that person need? What does maintenance look like?
- A right to audit. You should be able to have a technical person of your choosing review what was built.
Most engagements that end badly don’t fail because the advisor was incompetent. They fail because none of these structural pieces were defined upfront, and when the project ran into difficulty, there was no shared definition of success to navigate back to.
Define them before you sign. It protects both sides.