In my previous articles, I talked about comparing operationalising open source to the difference between your kitchen at home and a restaurant kitchen. In this article, I am going to talk about the challenges of operationalisation. In my next article, I will move on to how to overcome them, so that you can achieve three-star worthy open source operationalisation.
The challenges of operationalisation
I’ve talked about four categories of similarities between cooking and open source operationalisation: people, ingredients (or data), equipment (or technology) and processes. I am going to look at the challenges in each of these categories.
First, people. In a restaurant kitchen, or in data science, finding people with the right skillset and then getting them to stick around is a big challenge. A poll on the data science website KDNuggets found that data scientists tend to stay in their jobs for an average of just 1–2 years. What’s more, they command high salaries. Unsurprisingly, businesses don’t want to pay top price for someone who isn’t going to stay for long! This results in managers having to juggle teams with scattered skills and experience with different technologies, reducing time for innovation when covering gaps between one team member leaving and another joining. This is the reason why restaurants with one or multiple stars are associated with a specific chef. They can invest the time to develop the menu and deliver a repeatable experience for diners.
Second, ingredients, or data. A restaurant may spend months finding the right quality and flavour ingredients, but in doing so they ensure delivery is repeatable. Operationalising requires high quality and up-to-date data. We often see data in silos, where data scientists need to know who to ask to get access or even to discover what data are available. This often results in data duplication — and this, in turn, leads to out-of-date data that does not reflect the market conditions. The process of sourcing data is often very time-consuming and can result in specific data pipelines that are much harder to reproduce or scale when the model is moved to production. This leads to more time being wasted when the model needs to be re-built.
Third, we have equipment or technology. The cooking equipment in a professional kitchen is very different from what you will see in your own kitchen. The hobs and ovens are hotter. The pans are stronger and bigger. The knives are sharper. Similarly, when you operationalise models, your equipment needs to step up a notch. The technology you use needs to be able to scale in two ways, horizontally or vertically, based on demand once it is deployed. This is becoming more and more necessary as data volumes and model complexity increase.
In your restaurant kitchen, scaling horizontally is like adding more cooking equipment and vertical scaling is like purchasing more specialist cooking equipment. For analytics, horizontal scaling means adding more servers/machines to an environment to cope with larger volumes of data whereas scaling vertically is when more compute power is added to a server/machine to allow a more complex models to be used. This is a BIG point when considering open source operationalisation. To be able to scale your open source model, you need to overcome several challenges. Most of these are not considered during development, unless there is already an understanding of the entire process.
By default, most people learn how to code, and therefore build models, that are single-threaded. However, to reduce the processing time when running your model, you can multi-thread it. Think of single-thread like a country road and multi-thread like a motorway, where the cars are the data. On a country road, you can only travel as fast as the slowest car, meaning all cars arrive at their destination when the slowest car does. However, lots of cars can travel at different speeds on a motorway, leading to a faster journey for all cars except the slowest. However, multi-threaded models are complicated and it’s not as easy as just ‘converting from one to the other’. In a multi-threaded model, you need to replicate the data for each thread, or cut the data up into chunks across the threads, while still knowing which thread is using and modifying each specific piece of data. This increases the complexity of error handling because the model needs to know which thread is affected, so you can locate and solve the error. To increase the accuracy of your model, you can add layers, but this increases its complexity. To run these more powerful models, you may need a different processing unit — but this means you need to build the model in that environment in the first place — and most laptops don’t have that option. There are often code structure dependencies needed when building a model in the more sophisticated units, which means it can be difficult to transfer from a simpler environment. However, the cost means that building on a more sophisticated unit is not always possible. This is a careful balancing act.
Finally, processes. Identifying and addressing the challenges of processes is a fantastic place to start because effective processes help combat the other challenges. Processes also highlight that operationalising open source is not just a matter of technology. Without processes, you’re just one un-informed decision away from deployment falling apart. Without correct model governance, there is no way to understand who was responsible for what. Who built the model, provided the data pipeline, the model deployment destination or how it’s deployed? Without this information, data scientists are playing ‘Guess Who?’ as they develop models. This makes operationalisation a problem, and also affects regulatory reporting.
It would not be helpful to identify challenges without talking about some solutions. My next article will examine those, to help you achieve three-star operationalisation.