The Slippery Slope of DIY

Any horse owner can tell you about the slippery slope of horse ownership. It goes like this: first, you buy a horse. It’s fun for a while, but then you want to ride out with a buddy, so you buy a second horse. Now you and your buddy can have fun, but the barn gets boring, and you buy a trailer. Now you’re trailering everywhere to ride, but on one of your rides, one horse comes up lame. So, you buy a third horse as a stand-in. But you can’t leave that horse back at camp all by themselves; they need a buddy, so you buy another horse. But now that 3-horse trailer isn’t big enough, so you buy a bigger trailer. That bigger trailer demands a bigger truck. But by now you’re paying to board 4 horses and park a big trailer. You might as well buy your own farm.

I’m about halfway down this slippery slope, and I am loving the ride. It’s a slippery slope that ultimately leads to a place I want to be: farm ownership.

As part of my Sales Engineering role at Rasa, I recently tried to DIY a Rasa-ish conversational AI framework to see what my prospective customers were actually in for. And I immediately felt the pull of a slippery slope. Here’s the difference: I want to own a farm. I do not want to own a conversational AI framework.

The First Horse

Here’s the thing: the experience is entirely dependent upon what you want out of the first horse you buy… I mean the first agent you build. If you need a relatively simple bot that handles only a few straightforward use cases, you might be perfectly fine DIY’ing your own agent. That’s akin to buying an older gentle horse to hang out with, love on, groom, etc. It’s satisfying a need, and if there are no other needs, you don’t need Rasa.

Down the Slope

The problem rears its head when you’ve built a simple FAQ bot, and you want to add your first transactional use case, but the LLM you’re using hallucinates the inputs, or runs into an error and fakes the outputs. LLMs are getting extremely crafty!

Now you slip down to the second horse. Still, for conversational AI DIY’ers, it looks like guardrail hell: code for observing LLM outputs/actions, or a secondary implementation path for deterministic flows.

Two systems: one for agentic conversations, one for deterministic ones. And the agent is chatting pretty well! Like any sensible developer, you decide it’s time to implement testing before deployment. But your conversations have split paths, maybe even intermingled paths of determinism and openness. Traditional e2e testing frameworks fail on the non-deterministic paths, so you have to implement a full simulation and evaluation suite on top of unit and e2e tests.

Now you’re ready to deploy, but deploying a large-conversation-volume bot requires some level of parallelization. Now you need tooling to guarantee that the same conversation is answered by the same server every time, or a shared database that always responds in milliseconds so the conversation feels snappy.

Once it’s deployed, you obviously need observability tools, deployment and migration tooling, CI/CD integrations. And when you think you’ve got everything built, your non-technical conversation designers start asking for access to everything so they can see and try to improve the bot, but that requires a non-tech-focused UI, and so far everything you’ve built has been CLI-only.

Congratulations; whether you wanted it or not, you now own a complete conversational AI framework.

The Velocity Layer

You own the framework for development, the infrastructure for deployment, and — like a cherry on top — you own the agent itself. Which is the only thing you set out to build in the first place. Alan, one of the co-founders of Rasa, calls the top layer the velocity layer. It’s where you want and need to spend your time. It’s where you make a direct impact on your customers and drive revenue. The layers below it are all very important, but the value they add is only really earned when there are 100s or thousands of bots running on them.

That’s what Rasa does. We own the bottom layers: development and deployment, and offer a platform upon which to build your bot: the velocity layer.

Own the Horse, Not the Farm

If you’re thinking of DIYing a conversational AI agent, peek over the edge to check out the bottom of the slippery slope of DIY and make sure you’re happy there. You very well may be! And if that’s the case, you’re in good company. But if you’d rather DIY just the important stuff, build on Rasa.

One final analogy: plenty of people want to own a horse but not the farm. That’s why boarding exists. You own the horse; you can ride him wherever whenever. You can train him to become whatever you want him to be. You pay his boarding fees, some medical bills, and whatnot. But the amount of shit you shovel goes down by a lot—all the perks of owning a horse without some of the most mind-numbing and constant headaches.