Strategy & Planning: Identifying, Validating, and Launching Successful AI Models
In this blog we outline our approach to identifying, validating, and launching successful AI models. We’ve used this strategy to build AI features in our own products and applied the approach to projects for our clients.
1. Define Your Business Objectives
The first step is to collect no more than 3 key business objectives so that effort is focused on addressing challenges or opportunities that matter to the business.
2. Collect Ideas & Explore Data to Identify Others
Compile ideas from team members that may move the needle on any of the key business objectives. Aggregate all potentially interesting data into an easily accessible data storage location outside the transactional system (e.g., CSVs , Amazon S3, etc.). Have an experienced data scientist and domain expert explore the data together over a 2 - 4 week period to find potentially interesting patterns that may become new, compelling ideas.
3. Prioritize Ideas
Add ideas to the table below and assign the relevant business objectives and related information for Audience, Data Availability, Potential Benefit, Customer Feedback (if time permits conduct idea and concept testing on each idea), Alpha Scope and Alpha Effort.
4. Draft Roadmap
Draft an initial roadmap with three ideas from the table above with 3 - 6 months set aside for each. Choose the most compelling and most doable idea to start. If resources and culture permits, also consider choosing one “Hail Mary” idea that may have a huge benefit (if it works) but longer effort. Place this “Hail Mary” idea on a parallel track.
For each idea, proceed through the Alpha, Beta, and Launch steps in an iterative fashion to ensure the model and user experience provides expected benefits. If, for some reason, they do not, consider shelving the idea and choosing another. You can always come back to a shelved idea if new data becomes available or a new opportunity surfaces.
5. Develop and Test Alpha Model
The goal of the Alpha (also called “Proof of Concept”) is to validate the viability of the idea using real data. Take a time-boxed approach (4 - 8 weeks) for each idea to build and validate an initial model.
For each idea:
- Explore the source data using data science techniques
- Draw up a hypothesis and what’s possible given the data available
- Experiment with the data by developing an initial model and validate the accuracy of the model
- Consider how the output of the model will present itself in the experience of a customer or internal staff
- Determine what needs to change in the current experience or accuracy of the model to meet requirements (e.g., more data, different algorithm, etc.)
- Make Go / No-Go decision to determine whether to proceed to a Beta
6. Develop and Test Beta Model
The goal of the Beta phase is to improve the alpha model, connect it into a real data pipeline, and build and validate the user experience.
- Implement data pipeline to feed the model
- Optimize and tune the model
- Design the user experience to capture data for the model (if it doesn’t already exist) and deliver model output
- Implement user experience in a standalone environment
- Test the user experience with users and refine
- Make Go / No-Go decision to determine whether to proceed to Launch
The goal of the Launch phase is to finalize the model and the user experience and package them up for production release. A key part of launching the model is to determine how often the model will be retrained and whether this is a manual or automated process. It’s often prudent to first launch with a AI feature internally first, then rollout to customers once the internal team experiences its benefits.
Post launch there is often a long tail of tuning and maintaining models as the data grows and changes. So make sure your team knows what they need to do to maintain it.
8. Refine Roadmap
Demonstrating value as quickly as possible and building momentum is key for AI initiatives. As you’re moving through the process of identifying and validating models and user experiences for each idea, refactor the roadmap so everyone is aware which ideas are at what phase in the process. Don’t be scared to shelve ideas if they aren’t bearing fruit.