Lately, it has become common to think that the development of AI is slowing down. To improve large language models, you need more and more data and computing power, and the gained advantages are decreasing.
The observation is mostly correct. The conclusion is incorrect.
The first wave of the rapid development of AI was mainly based on scaling. When models were trained with bigger data and bigger computing power, their capabilities grew in a predictable manner. At first, development was fast, but we are now nearing its limits. Increasing training data and computing power only creates diminishing returns.
The development of models has moved from training time to computing during use and specialised models.
The current AI systems improve their results by using more computing during the actual task. They think longer, test options and assess their answers. Less resources are used on easy questions, more on difficult ones.
AI models are also becoming more specialised. A general model that does everything is more expensive to train and use and is of lower quality than a model specialising in a certain task. The best models are already compilations of various specialised parts. Image recognition, speech, memory, retrieval and planning are all done with their own models that produce cleaner and cheaper information for more general reasoning.
For example, issues with image production in text production can be solved by connecting a specialised model into an image producing system that transforms text into visual form, while models that can identify letters and are able to compare the produced text with the provided text are used as reviewers.
The number of specialised models will increase. It improves quality and lowers the costs of AI.
A multi-element model needs an orchestration layer. It is an AI model that decides which modules to use, in which order and for how long. The orchestration layer can also test models for tasks for which they were not originally designed and permanently deploy any functioning combinations.
Intelligence will move from a singular model to the system structure.
The need to improve the quality of the AI and to reduce costs guides developers to deploy the same solutions our brains have been developing for millions of years. These include modularity, hierarchy and the selective use of resources.
The development of AI will continue even though the development created by a singular technology is declining. The next great leap will not be larger models, but specialised models and a structure that can apply these extensively.
