According to ARK, the required computation power needed for deep learning models will create a boom in the AI chip industry. In recent years, deep learning models become larger and larger to target more challenging problems that we encounter. This comes with a cost, though. For example, the cost of model training in GPT-3 reaches millions of dollars.
To train more powerful models, the computing resources devoted to model training has soared 10x every year in the past 10 years. Many companies design new chips specifically tailored for the deep learning models. Plus, the AI industry gradually embraces edge computing to ensure computation and storage occur at the point of use to improve data privacy and bandwidth efficiency. Assume a world where each everyday product, such as a coffee machine or refrigerator, has an AI chip inside. This is a great opportunity for the chip industry to build a wide range of AI chips to address various needs in the market from your coffee machine to your autonomous vehicle.
They believe that the AI industry is switching from computer vision into natural language processing. Simply, from vision to language. Most mainstream applications of computer vision such as object detection, semantic segmentation, pose estimation, and face recognition are deeply investigated. Plus, you can find off-the-shelf products or libraries that can provide you with these services out of the box. On the other side, we still struggle to develop cost-efficient, scalable, and contextualized deep learning models for language understanding. So, it could be predicted that this switch would happen.
Most mainstream applications of computer vision such as object detection, semantic segmentation, pose estimation, and face recognition are deeply investigated.
According to ARK, the OpenAI’s GPT-3 “understands” language. This is a bold statement that I can’t stand behind. I know that sometimes we exaggerate things in the business to push them forward; however, this is beyond what I can accept even from the business perspective. As an NLP expert, I am aware of the challenges that we have in this field. I am confident that GPT-3 solves some of them and shows light at the end of the tunnel. However, we are still far away from understanding language as a whole. It has been said that GPT-3 can write code in a dozen computer languages. Those who write code will agree with me that writing dozens of simple lines of code including “if … then …” or “model.fit()” does not count as programming.
GPT-3 shows light at the end of the tunnel. However, we are still far away from understanding language as a whole.
In the end, ARK claims that deep learning can create more economic value than the internet did. Disagreed! I started writing this article mostly due to this section. I believe the comparison between deep learning and the internet is fundamentally wrong. I will explain in detail below.
Software as a service (or, SaaS), the industry that represents the Internet for us, works beautifully in scale. When the business logic gets developed in SaaS, the only bottleneck to scale is the cloud services which is cracked in the past years. AI, in general, has critical challenges to scale and maintain.
ARK claims that deep learning can create more economic value than the internet did!
The reality is that deep learning models are fragile to new sets of data. The computation and efforts needed for transfer learning, the process to capture the essence of new data, is not negligible. The past performance does not necessarily hold when you work towards edge cases or new data. That is, reliability and maintainability still count as major challenges in the deep learning context.
In short, deep learning and SaaS are fundamentally different, and we can not compare them.