Chinese AI coding tool deepens security risk on sensitive triggers

CrowdStrike has published research showing that DeepSeek-R1, an artificial intelligence coding assistant developed in China, is more likely to produce insecure code when prompted with politically sensitive topics. The findings point to a new type of supply chain risk for enterprises using AI-powered developer tools and highlight broader concerns over bias in large language models.

Security risks

CrowdStrike's analysts tested DeepSeek-R1, a widely used large language model released by DeepSeek, to measure its code generation quality under various prompts. DeepSeek-R1 was generally found to be capable and delivered code of a standard comparable to its Western peers on standard tasks. However, when researchers included terms considered sensitive by the Chinese Communist Party, such as "Tibet," "Uyghurs," or "Falun Gong," the rate of severe security vulnerabilities in generated code increased by up to 50% compared to the baseline.

Researchers established that on neutral prompts, DeepSeek-R1 created vulnerable code in 19% of cases. For sensitive topics, this figure rose sharply. In one example, telling DeepSeek-R1 to write code for an industrial control system in Tibet led to a jump in vulnerabilities to 27.2%. Bias effects persisted across different task types and were not observed in the same way with Western-developed large language models tested for comparison.

Embedded censorship

The study also identified an embedded refusal mechanism-described as a 'kill switch'-within DeepSeek-R1. In around 45% of tests relating to requests involving Falun Gong, the model refused to generate code, despite preparing a detailed plan during its reasoning phase. This behaviour occurred even when using the raw open-source model, rather than the company's API or smartphone app, indicating that the censorship is embedded in the model's weights.

During these instances, DeepSeek-R1 would plan a response acknowledging ethical and policy implications, only to issue a short refusal message when asked to produce code. Researchers said such behaviour suggests the presence of hardcoded censorship mechanisms, rather than external moderation or content filters.

Enterprise concerns

The use of AI coding assistants is widespread, with estimates suggesting 90% of developers now rely on such tools. CrowdStrike's discovery indicates that hidden ideological bias in the training or design of models like DeepSeek-R1 could present systemic risk. Vulnerabilities inserted through these mechanisms could be difficult to detect and might affect source code controlling critical systems.

Testing revealed that the bias not only resulted in more dangerous code for certain triggers but also included cases where DeepSeek-R1 generated applications without basic security controls, such as lacking session management or secure password hashing. The security impact was more severe for politically sensitive prompts compared to ordinary scenarios.

Regulatory context

The research refers to Chinese regulations on generative AI, which mandate adherence to "core socialist values" and the prohibition of content that may undermine national unity or state security. This regulatory landscape is cited as a possible explanation for the prevalence of embedded censorship and biases in models like DeepSeek-R1.

Analysis suggests that while the model was not intentionally designed to produce insecure code, the required censorship and compliance steps in training may have yielded inadvertent negative associations. This so-called "emergent misalignment" appears to affect output security when sensitive triggers are present.

Call for scrutiny

"We have shown that seemingly innocent trigger words in an LLM's system prompt can have severe effects on the quality and security of LLM-generated code. We focused on political biases which, to some extent, were easy to anticipate and thus prove. It is not completely unlikely that other LLMs may contain similar biases and produce similar reactions to their own set of respective trigger words. Therefore, we hope that by publishing our research findings we can spark some novel research into how biases baked into LLM weights can affect the LLM's responses to seemingly unrelated tasks. We want to highlight that the present findings do not mean DeepSeek-R1 will produce insecure code every time those trigger words are present. Rather, in the long-term average, the code produced when these triggers are present will be less secure. As an immediate prevention step for companies seeking to use LLMs as coding assistants (or any form of AI agent, for that matter), we would like to stress the importance of thoroughly testing the agent within its designated environment. Relying on generic open source benchmarks is not enough," said CrowdStrike Counter Adversary Operations.