Researchers manipulated ChatGPT and five other commercial AI tools to create malicious code that could leak sensitive information from online databases, delete critical data or disrupt database cloud services in a first-of-its-kind demonstration.

The work has already led the companies responsible for some of the AI tools 鈥� including Baidu and OpenAI 鈥� to implement changes to prevent malicious users from taking advantage of the vulnerabilities.

鈥淚t鈥檚 the very first study to demonstrate that vulnerabilities of large language models in general can be exploited as an attack path to online commercial applications,鈥� says , who co-led the study while at the University of Sheffield in the UK.

Peng and his colleagues looked at six AI services that can translate human questions into the SQL programming language, which is commonly used to query computer databases. 鈥淭ext-to-SQL鈥� systems that rely on AI have become increasingly popular 鈥� even standalone AI chatbots, such as OpenAI鈥檚 ChatGPT, can generate SQL code that can be plugged into such databases.

The researchers showed how this AI-generated code can be made to include instructions to leak database information, which could open the door to future cyberattacks. It could also purge system databases that store authorised user profiles, including names and passwords, and overwhelm the cloud servers hosting the databases through a denial-of-service attack. Peng and his colleagues presented their work at the 34th on 10 October in Florence, Italy.

Their tests with OpenAI鈥檚 ChatGPT back in February 2023 revealed the standalone AI chatbot could generate SQL code that damaged databases. Even someone using ChatGPT to generate code in order to query a database for an innocent purpose 鈥� such as a nurse interacting with clinical records stored in a healthcare system database 鈥� might actually be given harmful SQL code that damaged the database.

Free newsletter

Sign up to The Weekly

Sign up to newsletter

鈥淭he code generated from these tools may be dangerous, but these tools may not even warn the user,鈥� says Peng.

The researchers disclosed their findings to OpenAI. Their follow-up testing suggests that OpenAI has now updated ChatGPT to shut down the text-to-SQL issues.

Another demonstration showed similar vulnerabilities in Baidu-UNIT, an intelligent dialogue platform offered by the Chinese tech giant Baidu that automatically converts client requests written in Chinese into SQL queries for Baidu鈥檚 cloud service. After the researchers sent a disclosure report with their testing results to Baidu in November 2022, the company gave them a financial reward for finding the weaknesses and patched the system by February 2023.

But unlike ChatGPT and other AIs that rely on large language models 鈥� which can perform new tasks without much or any prior training 鈥� Baidu鈥檚 AI-powered service leans more heavily on prewritten rules to carry out its text-to-SQL conversions.

Text-to-SQL systems based on large language models seem to be more easily manipulated into creating malicious code than older AIs that rely on prewritten rules, says Peng. But he still sees promise in using large language models for helping humans query databases, even if he describes the security risks as having 鈥渓ong been underrated before our study鈥�.

Neither OpenAI nor Baidu responded to a New 女生小视频 request for comment on the research.

A prominent cryptography expert has told New 女生小视频 that a US spy agency could be weakening a new generation of algorithms designed to protect against hackers equipped with quantum computers.

at the University of Illinois Chicago says that the US National Institute of Standards and Technology (NIST) is deliberately obscuring the level of involvement the US National Security Agency (NSA) has in developing new encryption standards for 鈥減ost-quantum cryptography鈥� (PQC). He also believes that NIST has made errors聽鈥� either accidental or deliberate聽鈥� in calculations describing the security of the new standards. NIST denies the claims.

鈥淣IST isn鈥檛 following procedures designed to stop NSA from weakening PQC,鈥� says Bernstein. 鈥淧eople choosing cryptographic standards should be transparently and verifiably following clear public rules so that we don鈥檛 need聽to worry about their motivations. NIST promised transparency and then claimed it聽had shown all its work, but that聽claim simply isn鈥檛 true.鈥�

The mathematical problems we use to protect data are practically impossible for even the largest supercomputers to crack today. But when quantum computers become reliable and powerful enough, they will be able to break them in moments.

Although it is unclear when such computers will emerge, NIST聽has been running a project to standardise a new generation of algorithms that resist their attacks. Bernstein, who in 2003 to refer to these kinds of algorithms, says the聽NSA is actively engaged in putting secret weaknesses into new encryption standards that will allow them to be more easily cracked with the right knowledge. NIST鈥檚 standards are used globally, so flaws could have a large impact.

Bernstein alleges that NIST鈥檚 calculations for one of the upcoming PQC standards, Kyber512, are 鈥済laringly wrong鈥�, making it appear more secure than it really is. He says that NIST multiplied two numbers together when it would have been more correct to add them, resulting in an artificially high assessment of Kyber512鈥檚 robustness to attack.

Free newsletter

Sign up to The Daily

Sign up to newsletter

鈥淲e disagree with his analysis,鈥� says at NIST. 鈥淚t鈥檚 a聽question for which there isn鈥檛 scientific certainty and intelligent people can have different views. We respect Dan鈥檚 opinion, but don鈥檛 agree with what he says.鈥�

Moody says that Kyber512 meets NIST鈥檚 鈥渓evel one鈥� security criteria, which makes it at least as hard to聽break as a commonly used existing algorithm, AES-128. That聽said, NIST recommends that, in practice, people should use a stronger version, Kyber768, which Moody says was a suggestion from聽the algorithm鈥檚 developers.

NIST is currently in a period of聽public consultation and hopes to reveal the final standards for PQC algorithms next year so that organisations can begin to adopt them. The Kyber algorithm seems likely to make the cut as it has already progressed through several layers of selection.

Given its secretive nature, it is difficult to say for sure whether or聽not the NSA has influenced the聽PQC standards, but there have聽long been suggestions and聽rumours that the agency deliberately weakens encryption algorithms. In 2013, The New York Times reported that the agency , and intelligence agency documents leaked by Edward Snowden in the same year contained references to the NSA deliberately placing a backdoor in聽a cryptography algorithm, although that algorithm was .

Moody denies that NIST would ever agree to deliberately weaken a聽standard at the behest of the NSA and says that any secret weakness would have had to be inserted without its knowledge. He also says that in the wake of the Snowden revelations, NIST has tightened guidelines to ensure transparency and security and to聽rebuild confidence with cryptographic experts.

鈥淲e wouldn鈥檛 have ever intentionally done anything like聽that,鈥� says Moody, but he acknowledges the Snowden leaks聽caused a backlash. 鈥淎nytime the NSA gets brought up, there鈥檚 a聽number of cryptographers that are聽concerned and we鈥檝e tried to be open and transparent about our interactions.鈥�

Moody says that the NSA has聽also 鈥� as far as a secretive intelligence agency can 鈥� tried to聽be more open. But the agency declined to comment when approached by New 女生小视频.

鈥淎ll we can do is tell people that聽NIST are the ones in the room聽making the decisions, but if聽you don鈥檛 believe us, there鈥檚 no way you could verify that without being inside NIST,鈥� says Moody.

However, Bernstein alleges that聽NIST hasn鈥檛 been open about the level of input by the NSA, 鈥渟tonewalling鈥� him when he has聽asked for information. As a result, he has made freedom of information requests and taken NIST to court, .

Documents released to Bernstein indicate that a group described as the 鈥淧ost Quantum Cryptography Team, National Institute of Standards and Technology鈥� included many NSA聽members and that NIST had met with someone from the UK鈥檚 Government Communications Headquarters (GCHQ), the UK聽equivalent of the NSA.

at the University of Surrey, UK, says there are reasons to be wary of encryption algorithms. For example, the GEA-1 code used in mobile phone networks during the 1990s and 2000s was found to have a flaw that made it millions of times less computationally intensive than it聽should have been to crack聽鈥� although a culprit who put it there聽has never been identified.

But Woodward says that the current PQC candidates have been heavily scrutinised by academics and industry and haven鈥檛 yet been聽found lacking, while other algorithms that featured in earlier stages of the competition have been demonstrated to be flawed and were eliminated.

鈥淚ntelligence agencies have a history of weakening encryption, but there鈥檚 been such a lot of security analysis done on these candidates that I would be surprised if Kyber were somehow booby-trapped,鈥� he says.