A Wake-Up Call: Google's AI Security Concerns Unveiled
In a recent revelation, Google's Threat Intelligence Group (GTIG) has sounded the alarm, highlighting a surge in attempts to steal and replicate its AI models. This development is part of a broader trend where state-backed and financially motivated attackers are leveraging generative AI for quicker reconnaissance, more sophisticated phishing, and even malware development.
But here's where it gets controversial... GTIG's AI Threat Tracker reveals a rise in "model extraction attempts" or "distillation attacks." This intellectual property theft occurs through legitimate access points like APIs, not network breaches. Google claims to have detected, disrupted, and mitigated such activities, but the question remains: are they enough to deter determined attackers?
Model Theft: The New Frontier
Model extraction is a sophisticated process where attackers repeatedly query a mature model to collect outputs that can train a separate "student" model. GTIG explains that this approach significantly reduces the time and cost required to build a competing model, targeting its behavior and, in some cases, its internal reasoning.
One intriguing case study involves "reasoning trace coercion," where prompts were designed to force Google's Gemini model to output full reasoning processes instead of user-facing summaries. Google identified over 100,000 prompts associated with this campaign, showcasing the scale and sophistication of these attacks. The company's systems recognized the activity in real-time and mitigated the risk, protecting internal reasoning traces.
GTIG emphasizes that the risk of model theft is concentrated on model developers and AI service providers, not average users. They recommend that organizations offering AI models as a service monitor API access patterns resembling extraction or distillation.
State-Backed Misuse: A Growing Concern
The report highlights how large language models have become essential tools for government-backed threat actors, aiding in technical research, targeting, and the rapid generation of nuanced phishing lures. GTIG identifies activity linked to actors associated with North Korea, Iran, China, and Russia, as well as unattributed clusters.
In one instance, an unattributed actor tracked as UNC6148 was observed using Gemini for targeted intelligence gathering, including searches for sensitive account credentials and email addresses. GTIG later detected phishing attempts against those accounts, focusing on Ukraine and the defense sector. Google took action by disabling assets associated with this activity.
Another example involves Temp.HEX, a China-based actor, according to GTIG. This group used Gemini and other tools to compile information on individuals, including targets in Pakistan, and to collect data on separatist organizations in multiple countries. While GTIG did not observe direct targeting from the research, they later noticed similar targets in Pakistan included in a campaign. Google disabled assets linked to this activity.
Phishing Evolved: The Power of AI
The report warns that language quality is no longer a reliable indicator for defenders as attackers use AI to generate tailored messages in local languages and professional tones. It introduces the concept of "rapport-building phishing," where an attacker uses multi-turn interactions to build credibility before delivering a malicious payload.
GTIG describes how the Iranian actor APT42 utilized generative AI models, including Gemini, for reconnaissance and targeted social engineering. The group used Gemini to search for official emails, research targets, and potential business partners, as well as for translation and understanding local references. Google disabled assets connected to this activity.
GTIG also identifies the North Korean actor UNC2970 as using Gemini to synthesize open-source intelligence and profile high-value targets. The activity included research on major cybersecurity and defense companies, as well as mapping job roles and salary information. Google disabled assets associated with this activity.
Tooling and Malware: The Dark Side of AI
The report reveals that state-sponsored groups continue to use Gemini for coding and scripting tasks and post-compromise research. It also notes a growing interest in "agentic AI" features, which are systems designed to act with a higher degree of autonomy. While GTIG has seen tools advertised as offering autonomous agents, they have not observed evidence of these claimed capabilities being used in the wild.
GTIG observed activity linked to a China-based actor tracked as APT31, which used structured prompts framing the user as an expert security researcher, seeking automated analysis of vulnerabilities and testing plans. Google disabled assets associated with this activity.
GTIG also tracked a China-based actor, UNC795, using Gemini several days a week to troubleshoot code and conduct research. Safety systems triggered, and Gemini did not comply with attempts to create policy-violating outputs. Google disabled assets associated with this activity.
In the realm of malware experimentation, GTIG describes a downloader and launcher framework, HONESTCUE, which used Gemini's API to receive C# source code that carried out second-stage actions. This approach complicates network-based detection and static analysis, and the secondary stage compiled and executed code in memory without writing the payload to disk.
Underground Services: A New Threat Landscape
GTIG points to an underground market for services claiming to provide custom offensive AI models but relying on commercial systems. One such toolkit, Xanthorox, was investigated by GTIG, who found that it used several third-party products, including Gemini, and drew on open-source tools and Model Context Protocol servers. Google's Trust & Safety team disabled identified accounts and AI Studio projects associated with Xanthorox.
The report also describes a campaign that abused the public-sharing features of AI chat services to host social engineering content. Attackers staged instructions encouraging users to copy and paste malicious commands into terminals, known as ClickFix. This activity used multiple chat platforms, including Gemini, and distributed malware variants targeting macOS. Google worked with its Ads and Safe Browsing teams to block malicious content and restrict the promotion of such responses.
In conclusion, Google emphasizes the need for a bold yet responsible approach to AI development, maximizing positive societal benefits while addressing challenges. The company plans to continue using threat intelligence and product enforcement to disrupt abuse, anticipating further experimentation with AI-enabled techniques across phishing, malware development, and credential theft as tools and services evolve.
What are your thoughts on these revelations? Do you think Google's measures are sufficient to tackle these emerging threats? Share your insights in the comments below!
