Identify the problems and make changes in 2021
Without a doubt, consumer behavior has shifted dramatically in 2020. We will be hard-pressed to find a model that predicted the global need for N95 facemasks and other PPE. PPE supply models will need to be completely redesigned and retrained. How are your models doing? Like many applied data science practitioners, you might find that some of the model results haven’t been ‘typical’ in recent months—Start 2021 out by reviewing your models and making an action plan.
Why is there a concern?
Obviously, consumer demand forecasting models fail when there are drastic shifts in actual consumer demand. Outliers are always problematic. 2020 has brought MANY outlier scenarios. The demand for laptops and peripherals soared due to the sudden shift to working and schooling from home. Facility shutdowns along the supply chains have disrupted the normal flow of just in time inventory of computer components. The shift from in-store shopping to online overwhelmed last-mile deliveries. Does the demand model have feature flags that indicate that the supplier has shut down the factory AND the shipping routes were shut down, AND consumer demand would spike after the typical holiday rush? Probably not. And for a good reason. There isn’t modern digital history for a world-wide pandemic with the widespread impact as COVID-19. The models couldn’t predict what they didn’t learn.
Do you have a problem?
These are some basic scenarios that might indicate you have to take action.
Has your measure of success metric dropped? Basically, is your model just not working well anymore? All models degrade once deployed, but have your numbers changed drastically? This is a symptom. See the other scenarios for possible root causes.
Has your data feature distribution changed (data drift)? Part of model monitoring is assessing changes to the data distributions in the data. Many platforms have automatic data drift monitoring with alarms. Have any of your alarms gone off?
Has the relationship between the feature and the target changed (concept drift)? In 2019, your sentiment model scored ‘mask’ as neutral, while in 2020, the model should score it as negative. Simply retraining the data without relabeling the sentiment of the word ‘mask’ will be ineffective.
Consider if the environment has changed so much that the actual business problem has changed. This analysis may require you to consult your business sponsors and Subject Matter Experts (SMEs). If the environment has changed drastically, retraining a model to a now-unattainable business goal is a waste of time, resources, and reputation.
What can you do about it?
You have several routes depending on the extent of the changes you need to make.
If you believe the issue to be data drift and the training and scoring data features themselves to be still relevant, retraining the model with the updated historical data might be enough. If the data continues to shift, consider adjusting the retraining schedule to a more frequent cadence.
If you believe the issue to be concept drift, you have some additional work. You will have to relabel your historical data before retraining.
Do you need new data features? Need a ‘globalPandemicflag’? If so, you may need to do additional data pipeline, analysis, training, and tuning work. The main architecture and design of your algorithm may remain intact.
If the underlying business use case has changed itself, you may have a complete redesign on your hands. Or the model needs to be retired completely. If your model predicts restaurant table reservation rates, you have bigger issues than poor model results. Is retraining the model the best use of resources at this time?
I don’t see consumer data stabilizing early in 2021. There are too many unknown variables regarding the vaccine rollouts, virus mutations, and the biggest unknown of all — human behavior. What consumer behaviors will take years to return to ‘normal,’ and which ones will suddenly flip once people get vaccinated? Prepare to bump up model monitoring, increase retraining cadence, and explore active learning solutions for your business cases.
Kudos for taking model monitoring and maintenance seriously! The data science world needs dedicated practitioners like yourself.