ML in Practice: Predicting the Time of Day to Send a Newsletter

It’s generally agreed upon that machine learning is really good at making predictions with extensive data sets.  Humans just can’t sift through billions of data points and turn them into anything intelligible. But machines fall short with respect to running a business.  There is a lot of creativity required to apply machine learning predictions in a way that has a positive net effect on both your goal and the efficiency with which your business operates. The Vidora team recently worked through applying some of this creativity to one of our customer’s use cases – predicting the best time of day to send each of their users a newsletter.  The steps we took are outlined below:

Step 1: Break the problem down

It requires a little thought and planning before getting your hands dirty with machine learning.  In our customer’s case of predicting times to send newsletters, a decision needed to be made: use a regression model or take another approach?  You could have a more complex regression model that predicted the best time down to the exact second, but after some thought, the simplest solution was going to be breaking time of day down into 4 time windows – morning, day, evening, and night.

Step 2: Add features to the pipelines

Vidora provides hooks to inject features directly into your fully automated ML pipelines, providing customers the power to improve performance with their own unique domain knowledge.  Since we were focused on time windows, new features were added that summed the total activity for each time window. One for the morning activity, one for the day, and so forth. Other features were added as well that summed activity for each day of the week.  These features helped the pipelines get a more accurate signal when predicting the user’s likelihood to be active at specific times.

Step 3: Build the pipelines

Since there were 4 time windows, the process required building 4 pipelines that predicted how likely the user would be in each time window.  Pretty straightforward when using Vidora – just log into Cortex, create each pipeline with a few clicks, and wait for them to finish. No need to worry about infrastructure, prepping the data, or trying to pick the best performing model.  All of that is handled for you.

Step 4: Remove the overlaps

There’s a caveat with building 4 pipelines and getting 4 sets of predictions. What if a user is likely to be active in the morning and the evening?  Fortunately, Vidora has made the process of comparing output quite simple by treating all your predictions as probabilities.  Instead of getting a convoluted model score, your pipelines provide predictive probabilities that can be compared across other pipelines.  If the end user is active in the morning and the evening, then just choose the one with the higher probability. Removing the overlaps is an important step when using multiple pipelines because you don’t want to send the same newsletter to a user in the morning and the evening!

Step 5: Uncover insights with Explainable ML

Understanding the output of your pipelines with Explainable ML and communicating it to others in the context of your business is paramount.  You never know what you might find that could influence your deployment strategies.  Here are a few interesting conclusions that the pipelines discovered in this case:

  • Users that were most active in the night time, tended to have higher feature importance scores for Friday and Saturday.
  • Users that were more active during the evening (4 to 8pm) gravitated towards sports content, while the rest of the users were more interested in local news with a splash of entertainment.
  • Users most active during the day tended to be less active on their phones compared to their computers and tablets.

Step 6: Deploy

With the users spread across 4 different time frames, the last and final step is to send those emails.  If inclined to incorporate some of the insights from step 5, you can adjust your strategies a bit. In this example, maybe you want to tailor the evening newsletter with different content compared to other time windows.  Or maybe you can adjust each newsletter depending on what device it is more likely to be consumed on. In either case, there will always be plenty of things you can experiment with and test.

Machine learning can be burdensome for your business if not approached in the right way, but as you can see from the example above, it’s becoming easier and easier to turn what used to be very complicated problems into something more tractable.  Whether you’re having issues with setting up machine learning infrastructure, engineering the right features for your models, or explaining the output, the Vidora platform provides ways to move forward with less headaches and great results.

Posted by Anastasia Turin on April 5, 2019