In late 2024, Texas Attorney General Ken Paxton revealed a landmark settlement with Pieces Technologies, a Dallas-based health-tech startup that had promoted its AI-powered clinical assistant as nearly error-free—claiming a “severe hallucination rate” below one in 100,000.

However, an investigation by the state found the company’s claims were unsupported by solid evidence. Authorities determined that Pieces had misled hospitals into believing the tool could accurately summarize patient records with a reliability it didn’t actually deliver.

While no patients were injured and no penalties were imposed, Pieces consented to clearer disclosures around accuracy, risks, and proper usage—a significant legal precedent suggesting that theoretical performance doesn’t equate to real-world effectiveness.

Experts such as cognitive scientist and AI critic Gary Marcus have long cautioned that current large language models are inherently constrained. As he puts it, these systems are "approximations of language use", not true language comprehension—a crucial difference that becomes especially risky when general-purpose models are applied in highly specialized settings and fail to grasp how actual work unfolds.

According to Gal Steinberg, co-founder and CEO of Twofold Health, the root of many AI shortcomings isn’t flawed algorithms. It’s a lack of context. “Because the ‘model’ only detects patterns, not intent,” he explained. “An AI can predict words or clicks with high precision, yet remain blind to the regulations, workflows, and unwritten rules that define a clinic—or any organization. When optimization ignores those realities, the AI meets its KPI but misses the point.”

Context: The Invisible Framework

Steinberg defines context as “everything the spreadsheet omits—objectives, boundaries, jargon, emotional tone, compliance requirements, and timing.”

When AI systems underperform, it's rarely due to insufficient processing power, but rather insufficient situational awareness. They lack the cultural knowledge, domain-specific subtleties, or time-sensitive understanding that human professionals absorb naturally. For instance, 90 seconds of silence in a therapy session might signal distress. To an AI transcript generator, it’s just empty space. In financial audits, a missing abbreviation could indicate deception. To a model trained on broad internet text, it may seem like a trivial acronym.

That’s why, at Twofold Health, the team begins by asking three foundational questions: Who is involved? What are they trying to achieve? And what are the consequences if the AI gets it wrong?

Another key issue, Steinberg emphasized, is that most organizations treat context as a one-time setup task. But environments evolve. Policies shift. Needs transform. “If you don’t continuously refine your prompts and retrain your models, the AI drifts,” he said.

This is why so many early AI initiatives now sit abandoned. RAND Corporation reports that over 80% of AI projects fail or stall—not because the technology doesn’t function, but because the context in which they were trained no longer reflects the operational reality. The result? An AI that looks correct on paper but fails in practice—like an actor reciting lines on the wrong stage.

Engineering Real-World Intelligence

The fix, Steinberg argues, isn’t just building smarter models, but embedding them with deeper environmental awareness. “It starts by involving domain experts directly in the AI development process. At Twofold, clinicians—not engineers—lead critical parts of the work. They teach the AI about medical language, ethical boundaries, and regulatory frameworks through lived experience,” he said.

Then there’s the overlooked, unglamorous labor that rarely makes headlines: determining which rare cases matter, standardizing informal speech, or realizing that the layout of a form can be more important than the data it contains. These choices may seem minor—until they cascade into systemic errors.

Prior research has shown that AI models trained on broad datasets often behave unpredictably when placed in niche environments—a challenge known as domain shift. In a well-known study, scientists from Google and Stanford observed that modern machine learning models are frequently “underspecified,” meaning they pass internal tests but break down under real-world conditions.

In high-stakes fields like healthcare and finance, where decisions carry legal responsibility, even small inaccuracies are unacceptable. That gap is a lawsuit in the making.

Even Meta’s chief AI scientist, Yann LeCun, has openly criticized the rush to deploy massive models without grounding them in real domains. Speaking at the National University of Singapore in April 2025, LeCun challenged the widespread assumption that bigger models equal smarter AI: “You can’t assume more data and more computing power automatically lead to more intelligent systems.”

He stressed that while scaling helps with basic tasks, it doesn’t solve the complexities of real life—ambiguity, adaptation, and reasoning. Instead, he called for “AI systems capable of planning, reasoning, and understanding environments the way humans do.”

Yet, according to Cisco’s 2024 AI Readiness Index, 98% of business leaders reported increased pressure to adopt AI—often without clear metrics, oversight, or accountability structures. In such a climate, it’s no surprise that context becomes an afterthought.

That’s the danger Steinberg wants to highlight: Not just that AI might generate false information, but that no one within the organization is prepared to take responsibility when it does. “We focus too much on precision and too little on ownership,” he said. “Context isn’t just knowing the correct answer—it’s knowing who answers for the damage when the answer is wrong. Establish that chain of accountability first, and your AI will be fed a richer, more responsible diet of context from day one.”

Anchoring AI in Reality

Context isn’t created by adding more parameters or GPU power. It comes from treating AI as a dynamic system that requires ongoing human guidance—not just initial training. And it comes from placing people—not just prompts—into the feedback loop.

AI isn’t inherently flawed. But without context, it acts as if it is. The answer isn’t blind trust. It’s better nourishment, regular monitoring, and ensuring there’s always someone watching when the AI grows overconfident.

“Because a model that hits its target but misses its purpose isn’t just wasteful. It’s dangerous,” Steinberg said.

The above is the detailed content of Starved Of Context, AI Is Failing Where It Matters Most. For more information, please follow other related articles on the PHP Chinese website!