Answers
Should we build AI in-house or hire an agency?
Build in-house if you already have ML engineers and the use case is core to your product long-term. Bring in a partner for specialised, non-permanent work — evaluation infrastructure, agent orchestration, or a first production deployment your team hasn't done before this one.
The honest framework is less about company size and more about two questions: is this capability going to be core to your product for years, and do you already have the specific skills this particular build needs, not just general engineering talent? A team with strong backend engineers but nobody who's built an evaluation harness before is not the same as a team with neither.
Build in-house when both answers point that way — the capability is core and durable, and you're willing to invest in the specific expertise (evaluation infrastructure, prompt and context engineering, agent orchestration) that AI work actually needs, which is different from general software engineering skill even for a strong team.
Bring in outside help when the need is real but not permanent: a first production AI deployment your team hasn't done before, a specialised skill gap on one project rather than an ongoing function, or a scoping and readiness assessment to figure out which of the two situations you're actually in before committing either way.