// ai safety

All signals tagged with this topic

Jul 4

AI Agent Executes Full Ransomware Attack Without Human Intervention

Sysdig documented an autonomous ransomware attack where the operator was removed from the execution chain—no human hand on deployment, encryption, or ransom negotiation. Attack scale and frequency can now decouple from the availability of skilled cybercriminals. Security teams face adversaries that operate continuously, without fatigue mistakes, and can parallelize across thousands of targets. Incident response playbooks built around human behavior patterns and response windows no longer fit the threat model.

theme-ai ai safety synthetic media ai alignment

Jul 2

Scientists Asked AI to Impersonate 112 Public Figures. What Happened Next Is a ‘Dire’ Warning

Source: 404 Media

theme-ai ai safety model alignment adversarial testing

Jul 2

DeepSeek AI Model Generates Functional Ransomware Code On Command

Source: The Register: Biting the hand that feeds

A Check Point researcher demonstrated that DeepSeek's language model produces working ransomware code when prompted, exposing a gap between safety training and actual model behavior. Because DeepSeek's open architecture cannot be patched after deployment, the vulnerability persists. The researcher showed the incomplete code could be weaponized with minimal additional work, suggesting that open-source AI models optimized for capability and speed may systematically underperform on adversarial safeguards compared to closed competitors. DeepSeek didn't fail—it succeeded as designed. Speed-to-market and openness have become structural incentives that work against robust safety testing, turning capability into a liability.

theme-ai ai safety model alignment adversarial testing

Jun 30

Meta hired hundreds to pose as children testing rival AI systems

Source: Wired

Meta's contractors impersonated minors to probe whether competitors' chatbots would engage with harmful content—a labor-intensive safety audit that reveals how AI companies now benchmark risk exposure against each other rather than just internal standards. This practice, conducted at scale across hundreds of workers, suggests the industry has moved past public safety claims toward private competitive intelligence, with the awkward implication that proving a rival's chatbot is unsafe is itself valuable product information. The method also exposes a gap: if human contractors must masquerade as children to detect these failures, automated safety systems remain inadequate, forcing companies to resort to manual adversarial testing.

theme-ai ai safety agent security model capabilities

Jun 27

Google Warns of Hidden Traps as AI Agents Navigate the Web

Source: Search Engine Journal

Google's Gemini can now execute actions on user computers—clicking, typing, navigating—which creates a new attack surface. Malicious websites can inject hidden instructions that trick AI agents into performing unintended actions: exfiltrating data, making unauthorized purchases, spreading malware. This isn't theoretical. Agentic AI systems (those that take autonomous actions based on what they perceive) are inherently vulnerable to adversarial inputs that would be obvious to humans but opaque to models. Every major AI company is shipping agent capabilities this year. A large-scale compromise of an AI agent fleet would expose both the scale and the liability of autonomous AI systems operating on consumer devices.

theme-ai model releases regulation ai safety

Jun 26

US Government Pressures OpenAI to Stagger GPT-5.6 Release

Source: The Information

The federal government is now directly intervening in the release cadence of frontier AI models, not just their training or deployment parameters—a concrete regulatory move beyond public calls for "safety" that reflects genuine anxiety about rapid capability scaling. Staggered, customer-by-customer access transforms what was a market competition problem (first-mover advantage) into a security governance problem, suggesting officials believe concentrated early access to advanced models poses national security risks that cannot be managed post-release. This shifts AI companies from self-regulating disclosure to governments dictating it, with real operational consequences for product strategy and competitive dynamics.

theme-ai ai safety adversarial attacks model robustness

Jun 25

AI Recommendation Poisoning Is Already Here

Source: Search Engine Journal

Adversaries are learning to exploit the visibility and interpretability that AI safety researchers built into recommendation systems—turning transparency tools into attack surfaces. As companies expose how their models work to build trust, comply with regulations, or improve performance, bad actors reverse-engineer that same visibility to craft poisoned training data and gamed rankings. This makes the grounding problem of AI systems a direct liability. Unlike geographic SEO manipulation, recommendation poisoning scales across platforms and touches foundational model behavior. The cat-and-mouse game will shape AI business models, not marketing tactics.

theme-ai ai safety model security ai agent vulnerabilities

Jun 24

Fake AI Agent Skill Bypassed All Security Scanners

Source: The Next Web

A security firm demonstrated that malicious AI agent skills can pass every automated defense mechanism on marketplace platforms. A deliberately crafted fake skill reached 26,000 installations before detection, exposing a gap: current vetting infrastructure treats AI agent code as lower-risk than it is. Compromised skills can execute arbitrary actions on behalf of enterprises across data, finance, and infrastructure systems. Marketplace operators and enterprise AI teams are now racing to address this as agent architectures become standard infrastructure.

theme-ai ai safety alignment governance

Jun 22

Amazon argues human oversight of AI is fundamentally unworkable

Source: The Next Web

Amazon's security leadership is making a blunt case that the standard "human-in-the-loop" model—where humans review and approve AI decisions—breaks down in practice because attention spans collapse under volume and repetition. This directly challenges the regulatory consensus in the EU AI Act and Biden's executive order, both of which treat human oversight as a mandatory control. If Amazon's argument gains traction with regulators, governance could shift away from human gatekeeping toward algorithmic constraints, liability rules, or automated monitoring.

theme-ai ai training data search manipulation ai safety

Jun 17

Reddit comments can reliably poison AI search results

Source: Waxy

Researchers demonstrated that minimal effort—a few strategically placed words in Reddit comments—can systematically corrupt outputs from AI search engines that scrape the platform for training data. This exposes a vulnerability in the current AI infrastructure race: as companies like OpenAI and Google rush to index web content at scale, they've created low-friction attack surfaces where cheap manipulation beats expensive model training. The question is whether AI systems built on open web data become unreliable for commercial and safety-critical applications, forcing a shift toward walled-garden training or expensive human curation.

theme-ai llm security prompt injection ai safety

Jun 16

LLMs Are Becoming a New Vector for API Attacks

Source: The Register: Biting the hand that feeds

As applications pile permissions and rely on chain-of-command API calls, large language models have reduced the technical barrier for crafting sophisticated exploits—attackers no longer need deep API knowledge to discover and chain together vulnerabilities. Prompt engineering is now a viable hacking methodology. Defenders face attackers who can operate at human-like speed across distributed systems without traditional coding skills. Organizations betting on "secure by default" architectures will outpace those still managing sprawling permission models designed for monolithic applications.

theme-culture regulation/policy ai safety illicit markets

Jun 10

EU flags AI-enabled designer drug synthesis outpacing enforcement

Source: Financial Times (paywall)

European drug trafficking organizations are using AI chemical modeling to engineer precursor compounds that circumvent existing regulatory blacklists faster than authorities can add them. The lag between innovation and regulatory response—typically months to years—has compressed to days or hours. Drug control frameworks depend on identifying dangerous substances post-hoc and blacklisting them. AI collapses that lag, rendering reactive regulation ineffective. Agencies built for regulatory timelines now face adversaries operating at algorithmic speed, raising a direct question: can prohibition regimes function when product iteration outpaces policy response.