RAG, MCP, and the Cost of "Working Around" Data Limits
If proprietary data is the bottleneck, how is the industry responding?
The dominant answer today is not to train models directly on private data, but to route that data into models at inference time. This is the idea behind approaches like Retrieval-Augmented Generation (RAG) and emerging patterns such as Model Context Protocols (MCP), where systems dynamically inject relevant context into prompts.
In principle, this is an elegant workaround: keep sensitive data private; avoid costly retraining; allow models to access up-to-date, domain-specific information. But in practice, this approach comes with a growing cost—literally.
The Token Explosion Problem — RAG and MCP systems rely on feeding large amounts of context into the model. Instead of the model "knowing" the information, it must re-read it every time.
This creates a fundamental inefficiency: every query requires retrieving documents; those documents are serialized into tokens; the model processes them repeatedly, from scratch. As context sizes grow (sometimes tens or hundreds of thousands of tokens), costs scale accordingly. The result is what could be called a token tax on intelligence.
Why This Doesn't Scale Cleanly — This approach works well for: small knowledge bases; narrow use cases; low-frequency queries. But it struggles when: data is large and deeply interconnected; reasoning requires multi-step context accumulation; latency and cost constraints matter.
In these cases, RAG systems become: expensive (due to repeated token usage); slow (due to retrieval + long-context inference); brittle (due to context selection errors).
A Deeper Limitation — More fundamentally, RAG does not truly solve the data problem—it sidesteps it. Instead of integrating knowledge into the model's weights, it relies on "just-in-time reading" rather than "learned understanding."
This leads to weaker: abstraction; generalization; long-horizon reasoning. Because the model is constantly juggling external context instead of internalizing patterns.
The Emerging Tension — We are now seeing a tension at the heart of modern AI systems: Training on proprietary data is powerful—but restricted. Injecting proprietary data at inference is flexible—but inefficient.
RAG and MCP sit in the middle of this trade-off. They are currently the most practical solution, but they expose a deeper issue: The architecture of today's LLMs may not be well-suited for a world where the most valuable knowledge cannot be freely absorbed during training.
What Comes Next? — If this bottleneck persists, we should expect new directions such as: more efficient context compression; hybrid systems with persistent memory; on-device or on-prem fine-tuning; architectures that separate reasoning from storage.
Until then, the industry will continue paying the price—in tokens, latency, and complexity—for accessing the data it cannot fully learn from.