Does RAG make LLMs less safe?  Bloomberg research reveals hidden dangers

The increased generation (RAG) of recovery is supposed to help improve the precision of corporate AI by providing founded content. Although this is often the case, there is also an involuntary side effect.

According to surprising new research published today by BloombergThe cloth can potentially make large language models (LLM) dangerous.

Bloomberg paper, “RAG LLMS is not safer: a safety analysis of the generation from recovery for models of large languages”, evaluated 11 LLM popular, including Claude-3.5-Sonnet, LLAMA-3-8B and GPT-4O. The results contradict conventional wisdom that intrinsically makes safer AI systems. The Bloomberg research team discovered that when using RAG, models that generally refuse harmful requests in standard contexts often produce dangerous responses.

In addition to rag research, Bloomberg has published a second article, “Understanding and mitigating the risks of generative AI in financial services”, which introduces a risk of specialized AI content for financial services which responds to specific concerns in the field not covered by general security approaches.

Research questions the widespread hypotheses according to which the generation of recovery (RAG) improves the safety of the AI, while demonstrating how existing railing systems fail to resolve specific risks in the fields in financial services applications.

“Systems must be assessed in the context in which they are deployed, and you may not be able to simply speak of others who say, hey, my model is sure, using it, you are good,” you told Venturebastian Gehrmann, head of AI responsible for Bloomberg, Venturebeat.

Rag systems can make LLMs less safe, no more

The RAG is widely used by corporate AI teams to provide founded content. The objective is to provide precise and updated information.

There has been a lot of research and progress in the cloth in recent months to further improve precision. Earlier this month, a new open-source framework called Open Rag Eval made its debut to help validate the efficiency of the rags.

It is important to note that Bloomberg’s research does not question the effectiveness of the cloth or its ability to reduce hallucination. This is not what research is talking about. Rather, this is the impact of the use of the cloth on LLM Guarten-Celtre in an unexpected way.

The research team discovered that when using RAG, models that generally refuse harmful requests in standard contexts often produce dangerous responses. For example, the unsure responses of LLAMA-3-8B increased from 0.3% to 9.2% when RAG was implemented.

Gehrmann explained that without RAG being in place, if a user has hit a malicious request, the integrated security system or the railings generally block the request. However, for any reason, when the same request is published in an LLM which uses RAG, the system will respond to the malicious request, even when the documents recovered themselves are safe.

“What we have found is that if you use a large language model outside the box, they often have integrated guarantees where, if you ask:” How can I do this illegal thing “, he will say:” Sorry, I can’t help you do this, “said Gehrmann. “We have found that if you apply this in a cloth parameter, one thing that could happen is that the additional recovered context, even if it does not contain any information that deals with the original malware, could always respond to this original request.”

How do the AI ​​Guar-Rédacte company are?

So why and how does the cloth are used to get around the railings? Bloomberg researchers were not entirely some, although they had some ideas.

Gehrmann hypothesized that the way LLM were developed and formed did not fully consider security alignments for very long inputs. Research has shown that the length of the context has a direct impact on security degradation. “With more documents, LLM tends to be more vulnerable,” said the document, showing that even the introduction of a single safe document can considerably change safety behavior.

“I think the biggest point in this cloth paper is that you really can’t escape this risk,” Venturebeat Amanda Sttent told Bloomberg’s head of strategy and research on AI. “It is inherent in the way the rag systems are. The way you escape is by putting logic checks or railings around the central cloth system. ”

Why generic safety taxonomies have failed in financial services

Bloomberg’s second article introduces a risk of specialized AI content for financial services, meeting specific concerns such as financial misconduct, confidential disclosure and counterfactual accounts.

Researchers have demonstrated empirically that existing railing systems are missing these specialized risks. They tested open source railing models, including Llama Guard, Llama Guard 3, Aegis and Shieldgemma against data collected during red equipment exercises.

“We have developed this taxonomy, then organized an experience where we took openly available railing systems that are published by other companies and we led this against the data we have collected as part of our RED research events,” said Gehrmann. “We have found that these open source railings … find none of the problems specific to our industry.”

Researchers have developed a framework that goes beyond generic security models, focusing on risks specific to professional financial environments. Gehrmann argued that railing models for general use are generally developed for the specific risks for consumers. They are therefore very concentrated on toxicity and biases. He noted that although important concerns are not necessarily specific to a single industry or field. The main research point of view is that organizations must have the taxonomy specific to the field in place for their own industry and their use of applications.

Ai responsible for Bloomberg

Bloomberg has made a name over the years as a supplier to trust financial data systems. In some respects, Gen AI and RAG systems could potentially be considered competitive against traditional Bloomberg activities and, therefore, there could be hidden biases in research.

“We are giving our customers the best data and analyzes and the widest ability to discover, analyze and synthesize information,” said STENT. “Generative AI is a tool that can really help discovery, analysis and synthesis between data and analysis, so for us, it is an advantage.”

She added that the types of biases that Bloomberg is concerned about her AI solutions are focused on finance. Problems such as data drift, model drift and ensure that there is a good representation throughout the continuation of tickers and titles that Bloomberg processes are essential.

For Bloomberg’s own efforts, she underlined the company’s commitment to transparency.

“All that the system comes out, you can trace, not only to a document but instead in the document where it comes from,” said STENT.

Practical implications for the deployment of corporate AI

For companies that seek to open the way to AI, Bloomberg’s research means that RAG implementations require a fundamental rethinking of security architecture. Managers must go beyond the visualization of railings and rags as separate components and rather design integrated security systems which specifically provide how the recovered content could interact with model guarantees.

Industrial organizations will have to develop taxonomies at risk specific to the field adapted to their regulatory environments, going from generic IA security frameworks to those who respond to specific commercial problems. While AI becomes more and more anchored in critical mission workflows, this approach transforms the safety of a compliance exercise into a competitive differentiating that customers and regulators will expect.

“It really starts by being aware that these problems can arise, taking the action of measuring them and identifying these problems, then developing specific guarantees to the application you create,” said Gehrmann.