July 23, 2026
Part 4: Waves of Disruption and Creative Destruction
This is the last of four posts discussing the state of AI. The first post provided an analogy to describe the nature of LLM technology as it stands today, while the second and third posts explored the competitive and business dynamics of the industry. This final post looks ahead to potential new technologies that could disrupt the current generation of models.
Let’s return to the dinner-party analogy from our first post. As humans, when we have a conversation at dinner, we can learn from it and adjust our thinking going forward. In contrast, a fundamental limitation of current LLMs is that their core knowledge - the model’s weights - does not update based on interactions. At another dinner party the next day, you or I may refine or change our arguments based on what we learned previously. LLMs, by contrast, start from the same baseline each time a new conversation begins. They can be brilliant conversationalists throughout a dinner party, but when the party ends, it’s like Groundhog Day - they do not carry those insights into their fundamental understanding of the world. (Arguably, some humans are not so different - especially after a few glasses of wine - but at least we retain the capacity to evolve our thinking.)
For LLMs to incorporate new knowledge into their weights, they must undergo a very long and expensive training process each time. From a technical point of view, LLM configurations are very fragile. If you try to naïvely tweak their weights after release, they can become very confused and forget many other things you did not want them to forget. In industry jargon, when you try to tweak model weights to achieve “continual learning,” the model suffers from “catastrophic forgetting”.
The Workarounds: Context Windows and Fine-Tuning
Modern models do have something called a context window, allowing them to hold the proverbial dinner conversation in memory and take the entire discussion into account when generating responses. These context windows have grown dramatically - from a few thousand words at the launch of ChatGPT in late 2022 to hundreds of thousands today - and they are expected to continue expanding. Sam Altman has even suggested that ideally, such context windows should hold all of each person’s interactions forever.
Models can also be further trained after release through a process called fine-tuning, allowing them to acquire new skills or domain-specific knowledge. This technique has been most successful in narrow subject areas. Still, companies like Cursor have been working to fine-tune their models continuously based on user feedback (thus incorporating each dinner party into the core knowledge base of their product as soon as possible). Nonetheless, from an academic perspective, both context window expansion and fine-tuning are better understood as extensions of the original training rather than an inherent ability to learn continuously through interactions.
The Flying Plate Problem
Another limitation of current LLMs is that they can only really focus on one thing at a time. Take our dinner party analogy again. If you were on a roll talking about a subject at dinner and one of the guests threw a plate of food across the table, you would probably notice and react in some way or another*. As humans, our peripheral vision is always casually observing our surroundings without consuming enormous amounts of our brain power – we have an innate understanding of what the normal state of being is and we recognize that a flying plate is an abnormal event. LLMs don’t have such a holistic view of their environment. They have the equivalent of a one-track mind, focusing solely on the problem at hand.
To be able to notice the flying plate, current LLMs would need multiple sub-models focused on detecting such events and passing them to the main model, and they would need the sub-models to predict the behavior of every plate and piece of food in their field of vision – and even then, the main model would need to have seen flying plates before, and it would have needed to build comprehensive training sets to learn how to handle them. If they are not trained on the behavior of flying plates, they would not know to duck if one is flying toward them… and even if they get hit with that first plate, since they don’t have an innate ability to learn, they won’t know what to do with another plate flying toward them at the next dinner party!
That is the irony of the remarkable intelligence we see in the chatbots we use every day. They appear so smart - and yet, how smart can they truly be if they are so one-track-minded, have no peripheral vision, cannot fundamentally learn anything new, and their core understanding does not evolve through experience? This is why some leading AI researchers believe that current LLM technology is unlikely to deliver the next major step-change in intelligence.
Notably, Yann LeCun - one of the three founding fathers of the modern AI wave and former chief AI scientist at Meta - and Ilya Sutskever, former co-founder and chief scientist at OpenAI, have both argued that fundamentally new approaches will be required to achieve the next major leap in intelligence. Both have pointed to the inability of current LLMs to truly learn as a core deficiency which they aim to overcome with the new companies and research programs they are leading.
Modeling the Physical World
LeCun, for example, believes that we need a new generation of world models* that have an inherent understanding of the physical world and can thus predict the consequences of various actions and events. In our example above, a world model would understand that the flying plate is a special event and model out its consequences by simulating potential outcomes and then be able to decide on actions (e.g., duck out of the way).
Waymo, Google’s self-driving spin-off, recently incorporated Gemini’s world model to capture such new scenarios and simulate their potential consequences to be able to react to them. Now, if Waymo sees something for the first time, say a rogue delivery drone flying across a highway, it would capture the event and incorporate it into its learning so it can know how to handle the next drone it sees. Even if Waymo’s world model simulations are not yet happening in real-time, they represent an important milestone because they capture the full end-to-end cycle of observing, autonomously simulating, and then learning. Yes, learning(!). Researchers like LeCun are betting that the concepts and techniques used in the physical world (which are easier to visualize for us) can also be applied to the cognitive world, allowing new models to observe, simulate, and self-learn concepts and ideas natively – fundamental building blocks for true intelligence.
The Efficiency Gap
Another perceived deficiency of current LLMs is their severe inefficiency compared to humans. Humans run an extraordinarily powerful neural network (the brain) on roughly 20 watts of power, less than what a traditional lightbulb consumes. Modern LLMs, by contrast, require vast data centers consuming enormous amounts of energy. Unlike leading LLMs, humans do not need to have read all the books in the world or all the text on the internet to develop basic intelligence. Similarly, it may take a human tens of hours to learn to drive, while autonomous systems (from companies like Waymo and Tesla) have required a million hours of real-world driving data and billions of miles on the road. It seems plausible, then, that smaller, more efficient models should be achievable – models that do not need to memorize every factoid on all of Wikipedia but instead possess stronger cognitive structures that allow them to research, learn, and evolve their understanding dynamically.
At the same time, one way to think about human efficiency is to remember that humans have evolved over millions of years. As Andrej Karpathy has noted, many “training weights” are embedded in our biology through evolution itself. Karpathy points out that a zebra can walk just minutes after birth. It does not learn that skill from training or experience – it is encoded in its DNA. LLMs, by contrast, have only been around for less than a decade. Neural network technology itself was developed only in the mid-20th century. The progress of these systems has been astonishingly rapid, and it is likely that this trajectory will continue as researchers strive to develop smarter and more efficient models.
However, LeCun and Sutskever are also the first to admit that disruptive paradigms remain firmly in the research stage. As mentioned in our first post, scaling laws have proven extraordinarily powerful: the more an LLM learns from specific knowledge, the more generalized its capabilities become. As such, to date, the power of scaled, generalizable knowledge has trumped the efficiency gains from lower-power, smaller specialized models. In other words, over the years, scaling has managed to overcome the various perceived deficiencies of LLMs, so continued scaling of context windows and fine-tuning techniques may also make some of the models’ current limitations irrelevant, “dissolving [them] within the big blob of compute” (to use Amodei’s words).
Front-row Seats to Evolution
At best, it may be years before research efforts for fundamentally new approaches to AI modelling yield implementable step-change improvements that can disrupt the current coterie of LLM leaders. Arguably, it is equally plausible that such breakthroughs emerge from labs like OpenAI, Gemini, or Anthropic, or that the results arrive as incremental advances that incumbents ultimately absorb. For example, it would be quite plausible that Anthropic and OpenAI are already starting to use world models to train Claude and ChatGPT, much like Waymo is using Google Gemini’s world models to improve its self-driving. A host of other startups are also working on technical solutions to address these issues without fundamentally changing the nature of LLM technology – the successful ones will be good acquisition targets for the frontier labs.
If the past is a good predictor of the way LLM technology will evolve, rather than being disrupted by new technologies, it is more likely that the current LLM leaders will be the ones driving the disruption cycles. The ChatGPT moment in 2022 unleashed an incredible wave of investment and innovation that is likely to transform nearly every industry, representing a multi-trillion-dollar market opportunity. These investments in data centers, model training, and new generations of chips will provide compounding improvements in the quality and capabilities of the models. Entrepreneurs are hard at work reimagining how they can use these capabilities to change the structure of whole industries. Already, we are seeing that the Claude Codes and Cursors of the world are increasing the ability to generate software at an unprecedented rate, redefining the way software is built and upending the status quo.
This will likely continue…
… and other industries are bound to follow.
AI is unleashing a mega-wave of creative destruction across industries.
As investors, while we ponder the technological threats facing incumbent LLM leaders and remain cognizant of the broader risks these powerful technologies may pose, we could not be more thrilled to have front-row seats to witness the coming bundling and unbundling of the world economy as AI pushes humanity forward.
AI is on a roll indeed.
Author:
Source:
The description of world models here draws heavily from "World Models: Computing the Uncomputable"
Image Credit:
ChatGPT – “A futurist dinner party in Silicon Valley with a flying plate about to hit a robot at the table. Current generations of AI would have a hard time recognizing the flying plate as an unusual new event, and so they would not know to duck out of the way. Future generations of models may gain spatial awareness and learning capabilities.”
Disclaimer
The information and opinions contained in this article are for background information and discussion purposes only and do not purport to be full or complete. No information in this article should be construed as providing financial, investment or other professional advice.
The information contained herein is intended for the sole use of the recipient and may not be copied or otherwise distributed or published without the express consent of TOP Venture. Although the information contained herein has been established by TOP Venture based on or by reference to sources, documents and systems it believes to be reliable and accurate, TOP Venture does not guarantee its accuracy or completeness and assumes no responsibility for any losses that may arise from the use of this information. The views and opinions expressed herein are based on current market conditions and are subject to change without notice. No representation is made that any forecast or projection will be realized.