Pilots vs MVPs

Part of our goal, or mission, at AICE Labs is to prevent promising AI initiatives from languishing in “proof of concept” purgatory, where a nice demo in a local test environment gets dumped and never makes it to production. One way to save a project from this fate would be to immediately jump into a heavyweight solution with full integration into the final environment. But that risks going too far in the opposite direction. What’s the secret sauce for bridging the contrasting needs? On the one hand, we want to follow good development practices and get to an end-to-end integrated solution as quickly as possible. On the other hand, if you don’t yet know how to tackle a given problem at all, any work on integration is prone to change at best and a complete waste at worst. What’s more, full estimation of the integrated project often requires that the approach is known. After all, if a given problem can be solved with a simple wrapper around GPT or Claude, that’s on a totally different scale than if custom training, finetuning, or intricate agentic workflows are required.

Here’s our (current and evolving) thinking about this:

• Any project where you are not 100% sure you already know which AI technology you’ll be using needs to start with a pilot phase

• In that pilot phase, do not worry one bit about integration.

  • Where does data come from? From an excel sheet or CSV file, manually exported

  • Where does the code run? On your laptop

  • What about data security? Don’t worry about it. Ask for the data to be de-identified and scrubbed so that it’s not an issue.

• Don’t aim for perfection, aim for a de-risked decision: This is how we’ll go and build the real thing

• Be vigilant about dead ends and stay away from things that only work because you’re in that limited, simplified environment. Security is a good example here. Just because we are not worried about it in the pilot phase doesn’t mean we can explore solutions that would be inherently insecure. “We found a great solution, 99% accurate and blazing fast and you only need to send all your sensitive data to a third party in a sketchy country.” ;)

• For the real thing, do start working on end-to-end integrations from day 1. Now that you’ve verified the initial approach, building the whole system in parallel ensures there won’t be nasty surprises about integration issues three days before launch.

The outcome of the pilot phase is decidedly not an MVP. It’s research and prototyping to ensure that whatever you build next is actually viable.