Understanding AI in Cybersecurity and Secure AI Challenges, Strategies and Trends (Dilli Prasad Sharma, Arash Habibi Lashkari etc.) (Z-Library)

📄 File Format: PDF

💾 File Size: 3.8 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

Progress in IS Dilli Prasad Sharma · Arash Habibi Lashkari · Mahdi Daghmehchi Firoozjaei · Samaneh Mahdavifar · Pulei Xiong Understanding AI in Cybersecurity and Secure AI Challenges, Strategies and Trends

📄 Page 2

Progress in IS

📄 Page 3

Progress in IS encompasses the various areas of Information Systems in theory and practice, presenting cutting-edge advances in the field. It is aimed especially at researchers, doctoral students, and advanced practitioners. The series features both research monographs, edited volumes, and conference proceedings that make substantial contributions to our state of knowledge and handbooks and other edited volumes, in which a team of experts is organized by one or more leading authorities to write individual chapters on various aspects of the topic. Individual volumes in this series are supported by a minimum of two external reviews. The Series is SCOPUS-indexed.

📄 Page 4

Dilli Prasad Sharma • Arash Habibi Lashkari • Mahdi Daghmehchi Firoozjaei • Samaneh Mahdavifar • Pulei Xiong Understanding AI in Cybersecurity and Secure AI Challenges, Strategies and Trends

📄 Page 5

ISSN 2196-8705 ISSN 2196-8713 (electronic) Progress in IS ISBN 978-3-031-91523-9 ISBN 978-3-031-91524-6 (eBook) https://doi.org/10.1007/978-3-031-91524-6 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2025 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland If disposing of this product, please recycle the paper. Dilli Prasad Sharma University of Toronto Toronto, ON, Canada Mahdi Daghmehchi Firoozjaei MacEwan University Edmonton, AB, Canada Pulei Xiong National Research Council of Canada Ottawa, ON, Canada Arash Habibi Lashkari York University Toronto, ON, Canada Samaneh Mahdavifar McGill University Montreal, Canada

📄 Page 6

v Preface As part of the Understanding Cybersecurity Series (UCS) knowledge mobilization program, this book explores AI for security and secure AI, covering major vulner- abilities, attack vectors, and security solutions to protect AI systems, making it an essential resource for researchers, industry professionals, and security practitioners. In 2020, the first team released the initial online article series, Understanding Canadian Cybersecurity Laws, which received recognition and was awarded the Gold Medal for Best Blog Column in the Business Division at the 2020 Canadian Online Publishing Awards. Building on this success, the team published the first book, Understanding Cybersecurity Law and Digital Privacy: A Common Law Perspective, in 2021 through Springer Nature Switzerland AG. Continuing the research efforts, the second team launched the second article series in 2021, titled Understanding Cybersecurity Management for FinTech (UCMF), accompanied by the publication of the related book Understanding Cybersecurity Management in FinTech: Challenges, Strategies, and Trends. This book highlights the significance of cybersecurity in financial institutions by show- casing recent cyber breaches, attacks, and financial losses. Starting in 2022, the third UCS team embarked on the third online series, Understanding Current Cybersecurity Challenges in Law, addressing emerging trends and critical legal issues concerning cybersecurity globally. This series, con- sisting of six parts, explores digital jurisdictional authority and user-generated digi- tal content ownership. The series is complemented by the publication of the third book, Understanding Cybersecurity Law in Data Sovereignty and Digital Governance: An Overview from a Legal Perspective, which offers an in-depth understanding of current cybersecurity challenges and their legal implications. Simultaneously, another team also worked on the fourth book, Understanding Cybersecurity Management in Decentralized Finance: Challenges, Strategies, and Trends. This book comprehensively reviews cybersecurity in blockchain technolo- gies, analyzing platforms like Ethereum, Binance Smart Chain, Solana, Cardano, Avalanche, and Polygon. It explores cybersecurity issues in smart contracts, and related blogs are currently being published through the IT World Canada website.

📄 Page 7

vi Beginning in 2023, the fifth UCS team embarked on the development of the fifth book, Understanding Cybersecurity on Smartphones: Challenges, Strategies, and Trends. This book focused on understanding cyber threats and adversaries on smart- phones, examining cybersecurity threats, vulnerabilities, and risk management. The book offers practical solutions for securing and protecting smartphones while rais- ing awareness of the importance of smartphone security. In 2024, the sixth UCS team has focused on understanding the criticality of cybersecurity in healthcare, advocating for robust measures to protect patient data, maintain system integrity, and mitigate evolving cyber threats in a book entitled Understanding Cybersecurity Management in Healthcare: Challenges, Strategies and Trends. The book offers practical solutions for securing and protecting health- care data and the environment for patients, doctors, and hospital IT teams while raising awareness of the importance of healthcare environment security. The seventh UCS team has dedicated the last 2 years to exploring AI for security and secure AI, addressing the vulnerabilities, attack vectors, and security challenges associated with AI-driven systems. Their work culminates in the book Understanding AI for Cybersecurity and Secure AI: Challenges, Strategies, and Trends, which pro- vides an in-depth analysis of threats targeting AI models, adversarial attacks, and robust defense mechanisms. The book serves as a comprehensive guide for research- ers, industry professionals, and security practitioners, offering a broad study about the vulnerabilities and threats along with the strategies to enhance AI security while ensuring the safe and ethical deployment of AI technologies. Toronto, ON, Canada Dilli Prasad Sharma Toronto, ON, Canada Arash Habibi Lashkari Edmonton, AB, Canada Mahdi Daghmehchi Firoozjaei Montreal, Canada Samaneh Mahdavifar Ottawa, ON, Canada Pulei Xiong Feb 2025 Preface

📄 Page 8

vii Dilli Prasad Sharma For My family—my beloved wife, Brinda, and son, Suhan, And my father, the late Pratiman Sharma, and mother, Gita Devi Sharma, And my teachers, for your unwavering love, support, and invaluable mentorship. Arash Habibi Lashkari For My family—my beloved wife, Farnaz, and children, Kourosh and Kianna, And my father, Bahman, mother, Zeynab, and sister, Ziba, And my teachers and lecturers, for all the lessons you’ve taught me. Mahdi Daghmehchi Firoozjaei For My family—my beloved wife, Marzi; my beautiful daughters, Mahdis and Meloreen; And my beloved mother, Leila, thank you for your unconditional love and unwaver- ing support. Samaneh Mahdavifar For My family—my beloved son, Hesam, and my mom and dad, And my teachers and mentors, thank you for your love, unwavering support, guid- ance, and patience. Pulei Xiong For my family, collaborators, and colleagues, thank you for your unwavering love and support. Acknowledgments

📄 Page 9

ix Part I General 1 Why AI and Security? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 Cyber Threat Landscape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2.1 Threat Predictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.2.2 The Cyber Attacker’s Motivation . . . . . . . . . . . . . . . . . . . . . . 7 1.3 What Is AI? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.4 Traditional Approach to Cybersecurity . . . . . . . . . . . . . . . . . . . . . . . 8 1.5 AI-Centric Cybersecurity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.6 Applications of AI to Cybersecurity . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.7 Secure AI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2 Understanding AI and ML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.2 Overview of ML Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.2.1 Supervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.2.2 Unsupervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.2.3 Semi-Supervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.2.4 Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.3 Fundamental Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.3.1 Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.3.2 Logistic Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.3.3 Decision Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.3.4 Support Vector Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.3.5 k-Nearest Neighbors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.4 Neural Networks and Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . 24 2.4.1 A Brief History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.4.2 Overview of Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . 25 2.4.3 Deep Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Contents

📄 Page 10

x 2.5 Deep Learning Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.5.1 Feedforward Neural Network . . . . . . . . . . . . . . . . . . . . . . . . 28 2.5.2 Convolutional Neural Networks . . . . . . . . . . . . . . . . . . . . . . 29 2.5.3 Recurrent Neural Networks and LSTM . . . . . . . . . . . . . . . . . 30 2.5.4 Autoencoders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 2.5.5 Transfer Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Part II AI in Security 3 AI in Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.2 General Framework of AI in Security . . . . . . . . . . . . . . . . . . . . . . . . 39 3.2.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.2.2 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.2.3 Pre-Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.2.4 Feature Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.2.5 Model Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.2.6 Model Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.2.7 Model Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.2.8 Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.2.9 Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.3 AI-Driven Security Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.3.1 AI-Driven Cyberattacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.3.2 AI-Driven Security Solutions . . . . . . . . . . . . . . . . . . . . . . . . . 52 3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4 AI for Network Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 4.2 Network Protection Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.2.1 Signature-Based Detection. . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.2.2 Anomaly-Based Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.2.3 Policy-Based Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.2.4 Reputation-Based Protection . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.3 AI/ML-Based Protection Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.3.1 Supervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 4.3.2 Semi-Supervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 4.3.3 Unsupervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 4.3.4 Reinforced Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 4.4 Future of AI in Network Security . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 4.5 Challenges and Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 Contents

📄 Page 11

xi 5 AI for Software Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 5.2 Smartphone App Security Threats and Vulnerabilities . . . . . . . . . . . 70 5.3 Web Application Security Threats and Vulnerabilities . . . . . . . . . . . 71 5.3.1 Broken Access Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 5.3.2 Cryptographic Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 5.3.3 Injection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 5.3.4 Insecure Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 5.3.5 Security Misconfiguration . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 5.3.6 Vulnerable and Outdated Components . . . . . . . . . . . . . . . . . 76 5.3.7 Identification and Authentication Failures . . . . . . . . . . . . . . . 77 5.3.8 Software and Data Integrity Failures . . . . . . . . . . . . . . . . . . . 77 5.3.9 Security Logging and Monitoring Failures . . . . . . . . . . . . . . 78 5.3.10 Server-Side Request Forgery (SSRF) . . . . . . . . . . . . . . . . . . 78 5.4 Desktop Application Security Threats and Vulnerabilities . . . . . . . . 79 5.5 AI and ML Methods for Application Security . . . . . . . . . . . . . . . . . . 81 5.5.1 User Authentication and Authorization . . . . . . . . . . . . . . . . . 81 5.5.2 Threat Detection and Prevention . . . . . . . . . . . . . . . . . . . . . . 87 5.5.3 Vulnerability Assessment and Patch Management . . . . . . . . 90 5.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 6 AI for Cloud Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 6.2 Security and Privacy Issues with the Cloud Computing . . . . . . . . . . 97 6.2.1 Misconfiguration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 6.2.2 Unauthorized Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 6.2.3 Data Breach (Loss/Leakage) . . . . . . . . . . . . . . . . . . . . . . . . . 99 6.2.4 Malware Injections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 6.2.5 Insecure APIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 6.2.6 Abuse of Cloud Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 6.2.7 Account Hijacking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 6.2.8 Insider Threat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 6.2.9 Security Protection Solutions in Cloud Computing . . . . . . . . 103 6.3 Future of AI-Based Security Protection Solutions in Cloud Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 6.4 Challenges and Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 6.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 7 AI for IoT and OT Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 7.2 IoT Security Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 7.2.1 Security Issues in the Process Layer . . . . . . . . . . . . . . . . . . . 116 7.2.2 Security Issues in the Cloud Layer . . . . . . . . . . . . . . . . . . . . 117 7.2.3 Security Issues in the Network Layer . . . . . . . . . . . . . . . . . . 118 Contents

📄 Page 12

xii 7.3 Industrial Internet of Things- IIoT . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 7.4 Operational Technology (OT) Network . . . . . . . . . . . . . . . . . . . . . . . 121 7.4.1 OT Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 7.4.2 AI-Based OT Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 7.5 AI-Based IoT (AIoT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 7.5.1 AI-Based IoT Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 7.5.2 AI-Based IoT Security Challenges . . . . . . . . . . . . . . . . . . . . 128 7.5.3 AI-Driven Attacks on IoT Systems . . . . . . . . . . . . . . . . . . . . 128 7.6 Future of AI-Based IoT Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 7.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 Part III Secure AI 8 AI Security and Privacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 8.2 AI Vulnerabilities and Security Threats . . . . . . . . . . . . . . . . . . . . . . . 138 8.3 AI Security and Privacy Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 8.3.1 AI Security Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 8.3.2 AI Privacy Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 8.4 Adversarial Attacks Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 8.5 A Common Adversarial Attack Analysis Framework . . . . . . . . . . . . 152 8.5.1 AI Attack Surface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 8.5.2 Attacker’s Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 8.5.3 Attacker’s Knowledge and Capabilities . . . . . . . . . . . . . . . . . 154 8.5.4 Attack Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 8.5.5 Attack Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 8.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 9 Defense Methods for Adversarial Attacks and Privacy Issues in Secure AI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 9.2 Model Robustness and Defense Requirements . . . . . . . . . . . . . . . . . 160 9.2.1 Robustness Against Adversarial Attacks . . . . . . . . . . . . . . . . 161 9.2.2 Privacy Preservation in Secure AI . . . . . . . . . . . . . . . . . . . . . 161 9.2.3 Explainability & Interpretability . . . . . . . . . . . . . . . . . . . . . . 162 9.2.4 Secure Deployment & Monitoring . . . . . . . . . . . . . . . . . . . . . 162 9.3 Crafting Adversarial Samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 9.3.1 Adversarial Samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 9.3.2 Framework for Crafting Adversarial Samples . . . . . . . . . . . . 164 9.4 Robust Against Adversarial Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . 166 9.4.1 Adversarial Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 9.4.2 Certified Robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 9.4.3 Gradient Masking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 Contents

📄 Page 13

xiii 9.4.4 Input Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 9.4.5 Defensive Distillation Network . . . . . . . . . . . . . . . . . . . . . . . 170 9.4.6 Ensemble Defenses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 9.4.7 Adversarial Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 9.4.8 Classifier Robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 9.4.9 Network Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 9.5 Privacy-Preserving Methods in AI . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 9.5.1 Data Anonymization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 9.5.2 Homomorphic Encryption . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 9.5.3 Federated Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 9.5.4 Secure Multi-Party Computation . . . . . . . . . . . . . . . . . . . . . . 181 9.5.5 Differential Privacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 9.6 Explainability & Interpretability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 9.6.1 Explainable AI (XAI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 9.6.2 Robust Model Interpretability . . . . . . . . . . . . . . . . . . . . . . . . 186 9.7 Secure Deployment & Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 9.8 Challenges Applying Adversarial Defense Methods in Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 9.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 10 General Framework for AI Security and Privacy . . . . . . . . . . . . . . . . . 199 10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 10.2 AI Lifecycle Stages with Their Security Threats and Defenses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 10.2.1 Data Collection Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 10.2.2 Data Preprocessing Phase . . . . . . . . . . . . . . . . . . . . . . . . . . 203 10.2.3 Model Training Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204 10.2.4 Inference Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 10.2.5 System Integration Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . 208 10.3 NIST’s AI Risk Management Framework (AI RMF) . . . . . . . . . . . 210 10.3.1 Foundational Information . . . . . . . . . . . . . . . . . . . . . . . . . . 210 10.4 Core Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 10.5 Google’s Secure AI Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 10.6 AI Security, Privacy, Standards, and Regulations . . . . . . . . . . . . . . 216 10.6.1 ISO/IEC AI Security and Privacy Standards . . . . . . . . . . . . 217 10.6.2 European Telecommunications Standards Institutes . . . . . . 217 10.6.3 EU Artificial Intelligence Act. . . . . . . . . . . . . . . . . . . . . . . . 218 10.6.4 OECD AI Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 10.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 11 AI Safety and Fairness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 11.2 AI Risks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 11.3 Transparency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 Contents

📄 Page 14

xiv 11.3.1 AI Systems Are Opaque . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 11.3.2 Ethical Obligations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 11.3.3 Accountability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 11.4 AI Alignment and Machine Ethics . . . . . . . . . . . . . . . . . . . . . . . . . . 227 11.4.1 Four Key Principles (RICE) . . . . . . . . . . . . . . . . . . . . . . . . . 228 11.4.2 Human Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 11.4.3 Machine Ethics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230 11.5 Bias and Fairness in AI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230 11.5.1 Bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 11.5.2 Fairness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 11.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 12 AI Security Challenges, Opportunities and Future Work . . . . . . . . . . 239 12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 12.2 AI Algorithms Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240 12.3 AI Data Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 12.4 AI Applications Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 12.5 Classification of Key AI Ethical Challenges . . . . . . . . . . . . . . . . . . 242 12.6 AI Ethical Challenges at the Individual Level . . . . . . . . . . . . . . . . . 243 12.7 AI Ethical Challenges at the Societal Level . . . . . . . . . . . . . . . . . . . 243 12.8 AI Ethical Challenges at the Environmental Level . . . . . . . . . . . . . 244 12.9 Stages of AI System’s Lifecycle and Challenges . . . . . . . . . . . . . . . 244 12.10 Future Research Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 12.10.1 Transferability of Adversarial Examples . . . . . . . . . . . . . . . 246 12.10.2 Evaluating the Robustness of Defense Methods . . . . . . . . . 246 12.10.3 Difficulty in Controlling the Magnitude of Adversarial Perturbations . . . . . . . . . . . . . . . . . . . . . . . . 247 12.10.4 Lack of Research Focus on Attacks Beyond Classification Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 12.10.5 Evolving Threat of Unknown Unknowns . . . . . . . . . . . . . . 248 12.10.6 Randomization of Classifier’s Decision Boundary . . . . . . . 248 12.11 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250 13 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251 Contents

📄 Page 15

Part I General

📄 Page 16

3© The Author(s), under exclusive license to Springer Nature Switzerland AG 2025 D. P. Sharma et al., Understanding AI in Cybersecurity and Secure AI, Progress in IS, https://doi.org/10.1007/978-3-031-91524-6_1 Chapter 1 Why AI and Security? 1.1 Introduction Artificial intelligence (AI) has dominated our world by incorporating human intel- ligence and behavior into machines and systems. AI has a pivotal role in automating processes and creating smart and intelligent systems according to today’s needs and can be trained to solve specific problems (Sarker, 2022). There are a wide variety of cutting-edge applications of AI in our everyday lives. They have noticeably evolved over the past few years and have made their way in almost every business sector. AI applications include but are not limited to navigation, robotics, healthcare, market- ing, agriculture, finance, and transportation (SIMPLILEARN, 2024). Today, security organizations face many challenges such as sophisticated cyber threats, a continuously growing attack surface, an abundance of data, and increasing infrastructure complexity (IBM, 2024). These factors hinder their capacity to pro- tect data, oversee user access, and promptly identify and address security threats. As a result, traditional ways of attack identification, such as signature-based methods, no longer work. There should be an intelligent way of finding patterns in the under- lying threat data and updating the model as a zero-day vulnerability arises, a mal- ware variant evolves, or a phishing email attack happens. We can analyze millions of events using AI, detect attacks, and mitigate threats. One can keep track of the users’ behaviors and gradually construct profiles for users and assets in an organiza- tion to find deviations from the normal state over a long period. AI encompasses a spectrum of technologies like machine learning, neural networks, deep learning, and expert systems. Using AI-powered solutions, security experts can analyze the captured log files, find attack patterns, extract domain knowledge, and update and retain the intelligence. In this book, we study the application of AI to cybersecurity and investigate sev- eral AI-centric models that have been applied to network security, application secu- rity, cloud security, and Internet of Things (IoT)/Operational Technology (OT)

📄 Page 17

4 security. We further dive into AI security, which focuses on defending AI infrastruc- ture from cyberattacks. AI security risks encompass multiple aspects, such as data breaches and leaks, susceptible development pipelines, data poisoning, adversarial attacks, and expansion of the attack surface (WIZ, 2024). 1.2 Cyber Threat Landscape As technology advances, the cyber-attack surface is growing. With more users, we have more smart devices connections, and subsequently, more data is poured into the network traffic. Cyber threats have always had an accumulative trend in that new ones do not replace the attacks in 1 year in the next year. However, the new attacks are combined with the old ones into a broader spectrum of threats. Based on a report RapidScale published in 2022, the origin of most cyber threats is caused by human error. Adversaries exploit human weaknesses and lure them into achieving what they seek (PRNEWSWIRE, 2024). For instance, based on a report by McAfee Lab 2016 threat prediction, “63% of confirmed data breaches involve using weak, default, or stolen passwords, and 26% of miscellaneous human errors involve peo- ple mistakenly sending sensitive information to the wrong person!” (Rapidscale, 2024). While the most prevalent cyber threats in 2023 include but are not limited to phishing, ransomware, and extortion hacks, as discussed below: • Phishing: A phishing attack is a cyberattack that employs social engineering to lure Internet users into revealing sensitive information, such as login credentials, credit card numbers, or personal information. In this attack, the malicious actors impersonate legitimate individuals, organizations, or entities and trick the users into clicking on a fraudulent link that is redirected to an adversary’s website where she can steal the victim’s personal information. Phishing attacks often take the form of fraudulent emails, messages, or websites designed to appear trustworthy. • Ransomware: Ransomware is malicious software (malware) that encrypts a vic- tim’s files or locks them out of their computer or network, rendering the data inaccessible. Attackers then demand a ransom from the victim, typically in cryp- tocurrency, in exchange for a decryption key or to unlock the compromised sys- tem. Ransomware attacks can have severe consequences, causing data loss, financial harm, and operational disruptions for individuals, businesses, and orga- nizations. A ransomware attack happens in different stages, including infection, encryption, ransom demand, payment, and decryption (Trellix Threat Labs, 2024). • Extortion hacks: An extortion attack, also known as extortionware or simply extortion, is a type of cyberattack in which malicious actors threaten to reveal sensitive or confidential information, damage data, disrupt services, or carry out other harmful actions unless a specific demand is met. Extortion attacks are a form of cyber extortion where the attacker seeks monetary or non-monetary ben- efits in exchange for not carrying out their threats. Some common extortion 1 Why AI and Security?

📄 Page 18

5 attacks are ransomware, denial of service attacks, data theft, non-monetary extortion, and physical threats. 1.2.1 Threat Predictions Now that we are in 2024, we have witnessed significant shifts in the cyber threat landscape over the past year. Throughout 2023, cyberattacks became more sophisti- cated and targeted, with emerging threats such as AI-powered malware, advanced phishing campaigns, and critical infrastructure attacks dominating headlines. The rise of state-sponsored attacks and the increasing use of ransomware-as-a-service (RaaS) have further exposed vulnerabilities across industries. Social engineering has become more sophisticated and widespread by exploiting human behavior to steal personal information. Nation-states launch cyberattacks with economic, politi- cal, and territorial ambitions by executing espionage, warfare, and disinformation through threat activities across Israel, Ukraine, Taiwan, and other regions (Trellix, 2024). As we step into 2024, it is evident that cyber resilience, zero-trust frameworks, and proactive threat intelligence are no longer optional but essential for safeguard- ing digital assets. Reflecting on the lessons learned in 2023, organizations must remain vigilant, adaptive, and collaborative to stay ahead in an ever-evolving cyber- security landscape. Trellix Advanced Research Center team has predicted the following threats as the leading trends in 2024 (Trellix, 2024): • The threat of AI: One significant security concern would be the underground development of malicious Large Language Models (LLMs) and the security con- cerns associated with their potential misuse by cyber criminals. Advanced LLMs like GPT-4 are being utilized in phishing campaigns, creating counterfeit web pages, and developing malware, making large-scale attacks more accessible to individuals with limited technical skills. Cybersecurity researchers anticipate accelerating the development and malicious usage of such tools in 2024. The second concern would be related to Script Kiddies, individuals with limited tech- nical expertise who pose a growing threat due to the availability of advanced generative AI tools. There is a potential for unskilled actors to execute sophisti- cated attacks at scale using unrestricted generative AI that can write malicious code, create deepfake videos, and assist with social engineering schemes. The use of AI-generated voices in scams involving psychological manipulation poses significant risks. The improved quality of AI-generated voices makes it challeng- ing to differentiate between real and fake voices. The accessibility and afford- ability of AI-voice generation tools empower scammers to automate and amplify their fraudulent activities, targeting victims across diverse linguistic back- grounds. There would be an increase in the use of AI-generated voices in live 1.2 Cyber Threat Landscape

📄 Page 19

6 phone calls to impersonate legitimate entities and enhance the effectiveness of phishing and vishing attacks. • Shifting trends in threat actor behavior: One of the threats predicted in 2024 is Supply Chain Attacks Against Managed File Transfers Solutions. Designing effective Managed File Transfer (MFT) solutions is paramount in securely exchanging sensitive data for businesses. There exist inherent risks associated with MFT systems, including the potential for ransomware attacks due to the valuable information they handle. Recent incidents, such as the Cl0P group exploiting GoAnywhere MFT and the MOVEit breach, underscore the vulnera- bilities in these systems. The complexity of MFT integration into business net- works adds to security weaknesses. It is advised that organizations review and secure their MFT solutions, implement Data Loss Prevention (DLP) measures, and encrypt sensitive data to mitigate the risk of operational disruptions, reputa- tional damage, and financial losses. • Emerging threats and attack methods: • Insider threats are employees, contractors, or partners who have legitimate access to an organization’s critical assets and can intentionally or unintentionally harm the organization’s resources, personnel, facilities, information, networks, and systems (CISA, 2024). Based on Trellix 2024 Threat Predictions published in Oct. 2023, insider threats have increased by 47% over the last two years, impos- ing a significant loss of $15.38 million to contain these threats over organiza- tions. Therefore, it is essential for organizations to identify, detect, and mitigate these threats to retain their stakeholders’ satisfaction. Another emerging threat is related to the rising usage of QR codes by the public everywhere. QR code-based phishing attacks are on the rise due to the trust people place in QR codes, which cybercriminals exploit to distribute malware or lead victims to fake websites. QR codes are easy to create and are widely used in daily activities, therefore, they have become an attractive tool for cybercriminals to exploit in conducting phish- ing attacks. Edge devices such as firewalls, routers, and switches are prone to be targeted by Advanced Persistent Threats (APT) due to their inherent vulnerabili- ties and lack of intrusion detection capabilities. The number of connected devices over the Internet has increased substantially, making edge devices a crucial but vulnerable component of the digital infrastructure. Another target that threat actors are lured into is new attack vectors, such as Python scripts in Microsoft Excel which are alternatives to old Macros. Although Microsoft claims security measures, potential vulnerabilities or misconfigurations might exist which could be exploited by threat actors. Signed vulnerable drivers pose significant threats by allowing attackers to achieve kernel-level privilege escalation. Despite some mitigation efforts, such as the Vulnerable Driver Blocklist by Microsoft (Microsoft, 2024), these attacks remain simple to execute and are likely to increase in 2024, impacting security systems widely. 1 Why AI and Security?

📄 Page 20

7 1.2.2 The Cyber Attacker’s Motivation Normally, cyber attackers have a wide range of motivations for conducting attacks. There could be political reasons behind a cyber-attack campaign or a script kiddie wanting to show off their coding talents. Usually, the main motivation behind cyber- attacks is to earn money by stealing personal information or blackmailing users to hide some secret information. Some common motivations for cyberattacks include (SOPHOS, 2024): • Financial gain: Many cybercriminals are primarily motivated by financial incentives. They may engage in activities such as stealing credit card informa- tion, perpetrating fraud, or conducting ransomware attacks to extort money from victims. • Espionage: State-sponsored cyber attackers, or APTs, aim to steal sensitive gov- ernment, military, or corporate information for political, economic, or military purposes. Their motivation is often tied to espionage or gaining a competitive advantage. • Hacktivism: Some individuals or groups use cyberattacks to promote a political or social agenda. They may deface websites, leak sensitive information, or dis- rupt services to draw attention to their cause. • Revenge: In some cases, individuals may launch cyberattacks as revenge against a specific person or organization. This can be motivated by personal grievances or vendettas. • Intellectual property theft: Competing companies or nations may seek to steal intellectual property, research, or proprietary information for economic or tech- nological advantage. • Challenge: Some hackers are motivated by the thrill of outsmarting security systems or the challenge of breaking into highly secure networks. They may not have specific malicious intent but are driven by curiosity or a desire to prove their skills. • Cyber warfare: Nation-states may engage in cyber warfare to disrupt critical infrastructure, sabotage military operations, or gain strategic advantages in conflicts. Understanding these motivations is crucial for devising effective defense mecha- nisms and implementing mitigation strategies accordingly to contain the risk in an organization. 1.3 What Is AI? AI refers to the simulation of human intelligence in machines, enabling them to perform tasks that typically require human intelligence, such as visual perception, speech recognition, decision-making, and language translation. AI is a 1.3 What Is AI?