The Time to Next Order explorer is a combination of two different modeled attributes—Order Likelihood and Days Until Next Order. The models look at the last 180 days to predict the next 42 days (6 weeks).
Each model is built custom to the data source, retrained monthly, and new predictions are run nightly.
- Each customer's events collected over the last 180 days
- Revenue from that customer to date
- The customer’s order frequency
- Average order value for that customer
How the model predicts
These predictions use a deep sequential neural network trained on a client’s own discrete data events. Part of the power of the model comes from the “sequential” component - the model can account for an unknown number and type of events per input (including events proprietary to a customer, like a product finder or a fit wizard), taking into account the time period when they occur, without attempting to normalize or reclassify into consistent inputs. The model also works on anonymous data, not just on customers who have already purchased.
How the model is evaluated
A sample of your data is held separate from the model while it is being created so that we can understand how well it will perform for new customers and new scenarios (a training set). This training set is used in the resulting to model to generate predictions, which are compared to actual events.
The model is optimized to a measure called recall. When optimizing to recall, the goal is to find everyone likely to convert, even if it means some who are not likely converters are also identified, so that opportunity is maximized.
In general, those with any likelihood to purchase have 40x the conversion rate of those who are unlikely to purchase. Extremely likely purchasers can be as high as 100x (or more) the conversion rate of those unlikely to purchase.
While the model quality is different for every account, we expect “Extremely Likely” category of customers to convert at a very high rate, followed by "Very Likely", and "Likely". Once optimized for recall, the model ranks customers by likelihood and divides into sevenths, where the first seventh is "Extremely Likely"; the next two-sevenths are "Very Likely"; and the final four-sevenths are "Likely". All other customers are considered "Unlikely" to purchase.
For the data scientists...
Finally, if the language of model engineering is your native tongue:
- The Likelihood classifier typically has an AUC greater than 0.9
- The model is biased to have high recall and low precision because the goal is to assist in identifying identify strong candidates to message and not simply to predict the future
- The Mean Average Error of our “Days to Convert” metric is overall around 13 days but more accurate when there’s a strong intent signal or the likely conversion is close in time