Adversarial AI: Cybersecurity Implications

Adversarial AI is the use of artificial intelligence (AI) or machine learning (ML) to facilitate cyber attacks. A huge volume of new cybersecurity products tout their use of AI for good purposes, such as more effective threat detection and response, but AI can also be used for nefarious purposes, such as disrupting a vital industrial process.

According to SecurityWeek, the use of AI and ML in the marketing of many new cybersecurity products is seen as essential. However, while AI and ML can be used for good, they can equally be used for bad. It states that, like most areas of cybersecurity, it is an arms race between attackers and defenders. According to Adversa, attacks involving AI are currently less common than traditional attacks on software, but will likely be responsible for higher losses in the longer term.

Attacks that use AI can come in two forms. One of these is the use of AI in attacks, such as the use of deepfakes as part of a business email compromise attack. The other is attacks against AI itself, such as the poisoning of data underlying AI decisions in order to cause wrong conclusions to be drawn. To date, the use of such attacks is considered to be embryonic, but many believe that their use will grow and that the results can be severe. Nation states and well-resourced criminal operations are well placed to be able to deploy adversarial AI.

The US National Security Commission on AI notes that only a small percentage of current AI research focuses on defending AI systems against adversarial efforts. However, AI is being widely deployed for many use cases. The most advanced sectors are information technology and telecommunications, closely followed by automotive.

A recent survey by IBM shows that 45 percent of large enterprises and 29 percent of small and medium businesses have adopted AI, as it is likely viewed as an indispensable tool for managing cyber threats. Meticulous Research recently found that the market for AI in cybersecurity is likely to grow to $46.3 billion by 2027, an average annual growth rate of 23.6 percent per year. According to Palo Alto Networks, in 2021, 55 percent of organizations expect to increase cyber budgets, with a large proportion going to AI applications and solutions. Yet McKinsey estimates that 60 percent of all companies that are adopting AI recognize security risks generated by AI as the most significant that they face. Through the use of ML and AI, attackers can develop a deeper understanding of how organizations are attempting to stop them penetrating their networks.

Attacks Using AI and Targeting AI

Attacks Using AI

Cyber adversaries are using artificial intelligence and machine learning to create more aggressive, coordinated attacks. They are also leveraging intelligence such as personal information on targets gathered from social media and other sources to generate more effective phishing campaigns, achieving email open rates as high as 60 percent.

Adversaries are much more likely than end user organizations to share information among themselves, thus spreading their AI attack techniques. Adversaries are also more prone to sharing information on the Dark Web, where there are also marketplaces offering a wide range of AI and ML hackers tools, some even providing help desk support. Recently, the Dark Web has seen the emergence of cyber attacks-as-a-service.

While organizations are using AI and ML in an attempt to prevent malware and other attack modes, attackers are also using AI to create more malicious payloads for malware. An early example of this was DeepLocker, which was demonstrated by IBM in the summer of 2018. DeepLocker uses AI to hide malicious payloads within popular applications that are considered to be benign, such as web conferencing applications.

Sophos agrees when it comes to using AI to create more targeted malware. It states that researchers at both Google and OpenAI have demonstrated independently that neural networks can be leveraged to produce source code that is based on unstructured, natural language instructions. They postulate that adversaries will begin to adopt neural networks to reduce the cost of generating new, highly effective malware variants.

Another example of the use of AI by adversaries involves deepfakes, which can be used to impersonate, by voice and appearance, a senior executive. According to IntSights, discussions regarding deepfakes on the Dark Web have increased from around 40 posts in 2019 to 100 posts in 2021 and the number is expected to grow.

Deepfakes can be rendered more easily through the development of generative adversarial networks (GANs) that can synthesize fabricated images. GANs have application in industries such as art, fashion, video gaming and science. They can be used to produce art, for example, or to improve images used for scientific purposes. However, they can also be used for sinister purposes, such as producing fake photographs or videos, sometimes of people who don’t exist, or for creating false social media profiles. Attempts to address this problem are being made by lawmakers in California and the Defense Advanced Research Projects Agency (DARPA) is studying ways to counteract fake media generated using GANs.

This is particularly worrying owing to the increase of facial recognition as a biometric identifier. Increasingly, videos are being used for this purpose rather than photos as they are believed to be superior to photos alone. However, they can also be spoofed for the purposes of committing fraud or propagating disinformation.

Attacks Targeting AI

The effects of an attack targeting AI can be catastrophic. For example, financial models could be poisoned with wrong data so that they are no longer reliable. In the automotive sector, a self-driving car could be hit by an evasion attack, leading to serious injury or even death. Some refer to the “brittleness” of AI since just small modifications that are almost untraceable can lead to attackers misleading or disrupting a system. Such attacks are also extremely difficult to detect. For example, AI can be manipulated so that malicious activity appears as benign, so security professionals will be led to believe that the system is working properly. Difficulties in detecting attacks could give adversaries extra dwell time on a network, giving them more opportunity to cause serious damage.

There are three main types of attacks against AI that are currently known.

Poisoning attacks are where an attacker focuses on the data used to train a machine learning model, changing data that already exists or looking to introduce new data that is incorrectly labelled or even malicious, with the aim of causing incorrect decisions regarding the data. This type of attack is called adversarial contamination and is particularly prevalent where models are constantly being retrained as new data is introduced. An example of such as attack is fraud cases, where cases are relabeled as not being fraudulent so that the system will not reject them.

Evasion attacks are the most commonly used. They focus on the model itself and are used to modify data in order to evade detection or to be classified as legitimate. It can be compared to the way in which spammers look to obfuscate content in a spam email, by embedding content within an attached image for example. Researchers at Google have shown how attackers can make an image of a panda appear to be that of a gibbon. It works because image recognition models are trained to associate certain pixels with a particular variable, so attackers tweak those pixels to lead to an incorrect reading.

The third main type is model stealing or model extraction . These attacks focus on the model after it has been trained and aim to either reconstruct the model or to extract data that the model has been trained on. For example, the attacker could use such an attack to extract confidential data such as addresses or other personal data to use for their financial gain, or a stock trading model could be copied and used to trade stocks. These attacks are most often used when an attacker is looking to steal training data that is sensitive and confidential.

Combating AI Attacks

There are a number of approaches that can be taken to combat AI attacks. Adversarial training is used to train a model so that it can identify adversarial examples such as misclassified images. The aim is to be able to identify future adversarial attacks, although it is considered to be difficult to discover adversarial examples in the first place. IBM has developed an Adversarial Robustness Toolbox that aims to simplify the process.

Switching models is an approach that uses multiple models within a system, changing at random among the models to make predictions. This aims to make an attack harder to pull off since the adversary will not know which model is being used and would have to poison all the models to be successful.

One further approach uses a generalized model defense using multiple models but, instead of switching models, multiple models are combined into one generalized model. In this approach, all the models contribute to the final prediction. This is considered to be the most robust approach since an adversarial example might be able to trick one model, but is unlikely to be able to trick them all.

EU Policy Measures for Easing the Adoption of AI in Cybersecurity

The Centre for European Policy Studies (CEPS) has noticed a definite uptick in cyber attacks using AI and postulates that these will increase further with increasing uptake of the IoT, which will further expand the attack surface. It believes that, in order to manage cybersecurity risks in the face of technical challenges and resource constraints, there are a number of conditions that must be met in order to improve the robustness and resilience of systems using AI:

• Collaboration between policymakers, technical community and key corporate representatives should be enhanced in order to better investigate, prevent and mitigate the potentially malign uses of AI in cybersecurity.
• Public procurement policies should incorporate an assessment of security requirements for AI systems.
• The lack of predictability in AI systems should be countered by ensuring operational control over their use by developing and monitoring practices such as in-house development of AI models and greater testing of data, perhaps through the use of clone systems.
• Incentives should be offered for cross-border information sharing in the private sector to ensure a governance framework to enable legal certainty around the sharing of data
• AI certification should be supported at an international level, with efforts led by ENISA, including the use of assessments both prior to deployment and during the lifecycle of a service or process.
• The cybersecurity level of libraries and tools should be reviewed to prevent misuse before publication of research. Data sharing practices should be reviewed, taking into account the requirements of regulations such as GDPR.
• AI-related career paths should be developed in order to address the skills shortage and the cybersecurity sector should be monitored to ensure the smooth incorporations and understanding of AI tools into existing professional practices and architectures.

Proposals for Increasing the Security of AI Systems

The European Commission has recently published its proposed Regulation on a European approach for artificial intelligence, intended for use with high-risk AI systems¹. Among the proposed requirements are development of high-quality data sets, documentation and record keeping, transparency and provision of information, human oversight, and robustness, accuracy and security.

While the CEPS task force on AI and cybersecurity shares these views, it believes that more concrete guidance should be developed regarding how to secure AI systems. Among its proposals are:

• Secure logs related to the development and coding of systems are a requirement, showing who has changed what, when and why, which means that even older versions of software can be secured by checking and reversing differences and additions.
• All software libraries that are linked to code should have secure pedigrees.
• Such pedigrees also apply to data libraries used for training machine learning algorithms, which can also show compliance with privacy laws and other principles.
• The model parameters and training procedures for all machine learning that is used should be tracked.
• Records that demonstrate due diligence in the testing of technology should be a requirement, preferably including actual test suites so they can be checked by the organization or third party involved.
• Techniques such as randomization, noise prevention, defensive distillation, and ensemble learning should be used, in addition to logs, to enhance AI reliability and reproducibility.
• The full auditability of AI models at the time or point of failure should be proposed and made available for subsequent analysis, including by courts.

Conclusion

Attacks against AI are likely to expand rapidly, but the fact that they are difficult to detect could lead to many being missed with potentially catastrophic consequences. Such attacks are also likely to expand beyond cybersecurity to other areas that could include national security. For example, AI techniques could be used to reverse engineer data from a sensor to find the underlying model used in order to decipher sensitive parameters of systems such as nuclear reactors. Critical industries such as oil and gas, nuclear and chemical, electric power grids and water treatment are all likely to expand their use of AI. While their aim would be to increase the security of their operations, there is the very real danger that AI could also be used against them to cause significant harm.

Resource File

Adversa AI: http://www.adversa.ai/
Centre for European Policy Studies: http://www.ceps.eu/
US National Institute of Standards and Technology: http://www.nist.gov/
US National Security Commission on AI: http://www.nscai.gov/

References

¹ “A European approach to artificial intelligence.” European Commission. 2022.