// model limitations

All signals tagged with this topic

Aug 18

Google Finds LLMs Stumble When Questions Reverse Entity Order

Google's research exposes a structural vulnerability in how large language models retrieve factual knowledge—they perform significantly worse when a question inverts the typical subject-object relationship (e.g., "Who directed Jaws?" versus "What did Steven Spielberg direct?"). LLMs don't learn facts as flexible, bidirectional knowledge but rather as brittle, syntax-dependent patterns from training data. Their reliability depends heavily on how you phrase the query, not on what they actually know. For enterprises building AI systems that need to answer customer questions reliably, this reveals a hard limit on current models' reasoning depth. Fine-tuning and retrieval-augmented approaches may not solve the underlying problem.

theme-ai llm capability mathematical reasoning model limitations

Aug 16

Fields Medalist: LLMs Solve Math Problems Mostly Through Counterexamples, Not Proofs

Source: Wordpress

Gowers distinguishes between disproving conjectures via counterexamples—computationally tractable—and constructing proofs, which requires deeper conceptual reasoning. LLMs excel at verification and refutation where exhaustive search works, but haven't demonstrated the synthetic reasoning that drives major mathematical breakthroughs. The limitation is real: they're pattern-matchers that can find what doesn't work, not architects of why something must be true.

theme-ai generative ai capability model limitations ai safety

Aug 1

Google Earth's AI Generator Turns Satellite Data Into Unreliable Hallucinations

Source: The Verge

Google's integration of generative AI into Earth's imagery tools creates a credibility problem: users can prompt the system to fabricate photorealistic but entirely false satellite imagery, blurring the line between documented reality and computational invention. Earth functions as both a professional tool—urban planning, environmental monitoring, journalism—and a consumer reference point. AI-generated artifacts could spread unchecked through both channels, undermining the foundational trust that makes satellite imagery valuable as evidence. The vulnerability exposes a tension in Google's AI strategy: rushing generative features into established products without architectural safeguards, rather than building verification layers that distinguish indexed reality from model-generated synthesis.

theme-ai autonomous vehicle safety model limitations emergency response

Jul 18

Zoox's Autonomous Taxis Can't Handle Emergency Scenes

Source: SFist - San Francisco News, Restaurants, Events, & Sports

A Zoox robotaxi drove directly into an active fire with smoke and flames. The vehicle lacked the contextual reasoning to recognize and avoid the emergency—it followed its routing logic despite environmental signals that any human driver would interpret as impassable. The incident exposes a gap in the decision-making layer, not a sensor failure. Level 4 autonomy requires more than competence in normal driving; it demands systems that recognize when standard routing rules should be overridden.

theme-ai ai safety model limitations prompt injection

May 23

Google's AI Overviews break on basic command words

Source: The Verge

Google's summarization feature fails on simple imperatives like "disregard," "ignore," and "skip," suggesting the underlying models either lack instruction-following capability or are overcorrecting against prompt injection. This reveals a core design tension: making AI outputs responsive to user intent versus resistant to adversarial manipulation. Google has chosen lockdown over functionality in these cases.