Understanding Chapter 1 of the AI Security Guidebook: AI Systems and 15 Major Security Threats

1. Why should security be considered before adopting AI?

Artificial intelligence is rapidly expanding beyond simple analytical tools into public administration, civil-service support, document generation, data analysis, search and recommendation, facility management, and many other public-sector applications.

The emergence of generative AI has accelerated this transformation. Users can now provide prompts and receive documents, reports, code, images, summaries, and other outputs from AI systems.

However, broader AI adoption also introduces new security risks.

For example:

• Sensitive internal documents may be entered into an external AI service.

• Malicious data may be inserted into AI training data.

• Attackers may manipulate AI behavior through malicious prompts.

• AI may be given excessive permissions and unintentionally modify another system.

• An external AI model or open-source library may contain vulnerabilities or malicious code.

The AI Security Guidebook for National and Public Institutions emphasizes that AI security must cover not only the AI system itself but also AI models, training data, input/output data, infrastructure, and supply chains.

For public-sector IT officials, this means AI should be regarded not simply as "new software," but as a new type of information system with a new and expanded attack surface.

2. What is an AI system?

The AI Security Guidebook defines an AI system as a system that uses AI models, training data, and infrastructure to achieve a given objective and provide users with outputs such as predictions, recommendations, or decisions.

In simple terms:

AI system = AI model + data + AI infrastructure + users

AI model

The AI model analyzes input data and performs tasks such as prediction, classification, or generation.

Examples include:

• Civil-service inquiry classification

• Image anomaly detection

• Document summarization

• Question-answering systems

• Traffic prediction

AI training data

Training data is used to teach an AI model.

It may include:

• Text

• Images

• Audio

• Video

• Databases

• Sensor data

• Administrative records

Because training data can influence model behavior, its security and integrity are critical.

AI infrastructure

AI infrastructure provides the computing and software environment required to develop, train, and operate AI systems.

It can include:

• CPUs

• GPUs

• NPUs

• Servers

• Storage

• Networks

• AI software

• Databases

• Vector databases

• RAG components

The Guidebook describes AI infrastructure as computing resources, networks, and related software used for AI development, training, and inference.

3. How are AI systems evolving?

AI has evolved from predictive AI toward generative AI, and increasingly toward agentic AI and physical AI.

The Guidebook presents this development as follows.

Predictive AI → Generative AI → Agentic AI → Physical AI

The most important change is that AI is moving from simply answering questions toward performing tasks.

An ordinary generative AI request might be:

"Summarize this meeting."

An agentic AI system might instead:

"Analyze the meeting → search related documents → create a report → send it to the responsible official."

If the AI has access to internal systems, the security implications become significantly greater.

OWASP's LLM security guidance also identifies Excessive Agency as an important risk because unchecked permissions in agentic systems can lead to unintended or risky actions.

4. How is AI security different from traditional cybersecurity?

Traditional information-security programs generally focus on protecting:

• Servers

• Networks

• Databases

• Applications

• User accounts

• Communication channels

AI systems add additional assets that must be protected:

• AI models

• Model parameters

• Training data

• RAG data

• Vector databases

• Prompts

• AI inputs and outputs

• Agent permissions

• AI supply chains

Therefore:

Traditional cybersecurity + AI-specific security

is the appropriate way to think about AI security.

The Guidebook specifically notes that AI security includes threats to AI models, model structures and parameters, input/output data, and training data in addition to conventional cybersecurity threats.

NIST's AI Risk Management Framework likewise provides a structured approach for organizations that design, develop, deploy, or use AI systems to manage AI-related risks.

5. The 15 major AI security threats identified by the Guidebook

The AI Security Guidebook identifies 15 categories of AI security threats, T01 through T15.

6. T01 — Training Data Poisoning

An attacker deliberately modifies or inserts malicious information into data used for AI training or inference.

This is particularly relevant to RAG systems because external documents and vector databases may also become attack surfaces.

The Guidebook explains that poisoned data can cause performance degradation, incorrect decisions, bias, or intentional unwanted behavior.

For public institutions, officials should ask:

• Is the data source trustworthy?

• Can data modifications be detected?

• Are hashes managed?

• Are changes logged?

• Is integrity verification applied to RAG data?

7. T02 — Unauthorized Sensitive-Information Training

This occurs when sensitive or confidential information is included in AI training without authorization.

For example, an institution should not assume that all internal information is automatically appropriate for AI training.

The purpose of the AI system, security classification, data sensitivity, and intended use must all be considered.

The Guidebook warns that internal confidential administrative information could be learned by an AI system and subsequently exposed to unauthorized users.

8. T03 — AI Backdoor Insertion

An AI backdoor causes a model or AI component to behave normally under ordinary conditions but perform malicious actions when a specific trigger condition is satisfied.

The Guidebook describes the possibility of backdoors being inserted into AI models, training data, or libraries and remaining hidden within the AI system.

Public institutions should therefore verify:

• Model origin

• Distributor identity

• File integrity

• Hash values

• Digital signatures

• Software provenance

before using externally obtained AI models.

9. T04 — Training Data Extraction

An attacker may repeatedly query an AI system and analyze its outputs to reconstruct information from training data.

This becomes particularly important when training data contains personal or confidential information.

Monitoring should therefore consider whether the AI is repeatedly revealing unusually specific information.

10. T05 — Unauthorized Access to Training Data

AI access does not mean that every user should have access to every piece of data.

Consider data such as:

• Personnel records

• Audit materials

• Security information

• Personal information

• Internal policy documents

If an AI system can access all of these sources, user-specific authorization must also be reflected in AI retrieval and responses.

The Guidebook warns that inadequate access controls for training data or RAG vector databases may result in disclosure to unauthorized users.

11. T06 — AI Model Extraction

Attackers may analyze AI outputs to infer model architecture, parameters, or weights.

For government-developed or highly customized AI models, the model itself may be an important institutional asset.

API access, usage limits, authentication, and output exposure should therefore be managed carefully.

12. T07 — Sensitive Information Input and Leakage

This is one of the most practical risks for public institutions.

Suppose an employee asks a public AI service:

"Summarize this audit document."

If the uploaded document contains personal or confidential information, the action may create a security problem.

The Guidebook specifically warns that sensitive information may be entered through prompts or file uploads and subsequently exposed to unauthorized users.

Institutions should therefore clearly define:

What information must never be entered into AI systems?

13. T08 — Prompt Injection

Prompt injection has become a major security issue with generative AI.

An attacker inserts malicious instructions into prompts or external content to change AI behavior.

The Guidebook explains that prompt injection may cause an AI system to ignore internal safeguards, generate malicious code, execute malicious commands, or disclose sensitive information.

OWASP identifies Prompt Injection as a major risk for LLM applications, alongside sensitive information disclosure, supply-chain risks, data/model poisoning, excessive agency, vector and embedding weaknesses, and unbounded consumption.

An important point for RAG systems is:

The data that an AI reads can itself become an attack vector.

A malicious instruction hidden in a retrieved document may potentially influence the AI's behavior.

14. T09 — Evasion Attack

An evasion attack manipulates input data so that an AI model makes an incorrect decision.

The Guidebook describes this in terms of adversarial examples designed to induce AI misclassification or malfunction.

For AI used in access control, facility security, surveillance, or other critical functions, testing should therefore include adversarial inputs rather than relying only on normal functional testing.

15. T10 — Communication-Channel Attack

This involves attacking the communication channel between users and AI systems.

Potential consequences include exposure of user inputs or AI outputs and theft of authentication credentials.

Public institutions should examine:

• Encryption

• Authentication

• Session management

• API-key protection

• Credential storage

• External AI service connections

AI security does not replace conventional network security.

16. T11 — Denial-of-Service Attack

Attackers may send excessive or malicious requests to exhaust AI resources.

Because AI workloads can require significant computing resources, this can result not only in service disruption but also increased resource consumption and cost. The Guidebook specifically notes the possibility of resource exhaustion, delays, service interruption, and excessive usage charges.

17. T12 — Lack of Incident and Anomaly Monitoring

If an AI system is attacked but the organization cannot detect it, incident response becomes difficult.

The Guidebook highlights the need to record and monitor AI inputs, outputs, and usage history.

Useful monitoring questions include:

• Who used the AI?

• When?

• From which device?

• What was requested?

• What was returned?

• Was the request unusual?

• Did usage suddenly increase?

18. T13 — Poor AI System Permission Management

This becomes particularly important when AI systems can control other systems.

An AI agent might have access to:

• Databases

• Document systems

• Email

• Business applications

• APIs

• Facility-control systems

If excessive permissions are granted, an incorrect AI decision or manipulated AI behavior could have significant consequences.

The Guidebook identifies situations where AI may modify systems or data without appropriate human decisions or where users cannot immediately control or stop the AI.

For agentic AI, the following principles are especially important:

Least privilege + human approval for sensitive actions + emergency stop capability

19. T14 — Supply-Chain Attack

AI systems often depend on many external components:

• Open-source AI models

• AI frameworks

• Libraries

• Vector databases

• External datasets

• APIs

• Cloud services

• Development tools

A compromise of one component may therefore affect the larger AI system.

The Guidebook describes supply-chain threats involving vulnerabilities or malicious code in AI models, training data, libraries, and other components.

MITRE ATLAS provides a knowledge base of adversary tactics and techniques for AI systems and can support threat assessments and AI red-team activities.

20. T15 — Poor Security Management of Contractors

Many public-sector AI projects involve external organizations for development, data labeling, system integration, maintenance, or operation.

Even when the government organization maintains strong internal controls, weaknesses at a contractor can result in exposure of AI models or training data.

The Guidebook explicitly identifies contractor security-management weaknesses as a threat across the AI lifecycle.

AI security therefore needs to cover the entire supply chain, not just the government organization's own infrastructure.

21. AI security must cover the entire lifecycle

The AI Security Guidebook describes the typical AI lifecycle as:

Data Collection → AI Training → AI System Construction → AI System Operation → AI System Disposal

Data collection

Data is gathered and prepared.

Security concerns include malicious data and unauthorized sensitive information.

AI training

Models are trained and evaluated.

Training-data integrity, access control, and model provenance become important.

AI system construction

The trained model is integrated into the production environment and connected to existing systems.

APIs, RAG, vector databases, and external-system connections may become important security considerations.

AI system operation

Users interact with the AI system.

The Guidebook identifies prompt injection, sensitive-information leakage, model extraction, evasion attacks, communication attacks, denial of service, monitoring failures, and permission-management problems among the risks that may arise during operation.

AI system disposal

Models, infrastructure, and data must be securely disposed of.

Ending operation does not automatically end the security risk. The Guidebook notes that poor disposal procedures can create new vulnerabilities and enable reuse or misuse.

22. What should public-sector IT officials check?

Before adopting AI, officials should ask:

• What is the purpose of the AI system?

• What data will it use?

• What security classification applies to the data?

• Will an external AI service be used?

• Will internal and external networks be connected?

• Where does the AI model come from?

• Which systems will the AI be allowed to access?

During implementation:

• Has the AI model's provenance been verified?

• Has the integrity of open-source components been checked?

• Does training data contain sensitive information?

• Are RAG and vector-database permissions separated?

• Are API connections protected?

• Are AI permissions limited?

During operation:

• Are AI inputs and outputs logged?

• Are anomalies monitored?

• Has prompt injection been tested?

• Are model and data changes tracked?

• Are vulnerabilities and security updates managed?

• Can the AI system be stopped during an incident?

During disposal:

• Has the model been securely deleted?

• Has training data been securely disposed of?

• Have backups been addressed?

• Has contractor-held data also been disposed of?

23. AI security is not about protecting only the AI model

AI systems are frequently connected to existing information systems.

A possible architecture is:

User → AI Service → RAG → Vector Database → Internal Database → Business System

In such an architecture, protecting only the AI model is insufficient.

The organization must consider the security of:

Data → Model → Infrastructure → User → External Connections → Supply Chain → Operations → Disposal

The Guidebook focuses on the safe management of AI models, data, and infrastructure while noting that other cybersecurity measures should also follow applicable national security guidelines.

24. International AI security frameworks

The AI Security Guidebook states that it references NIST AI RMF, ISO/IEC 42001, OWASP Top 10 for LLM, MITRE ATLAS, the UK NCSC guidance, and other domestic and international materials.

NIST AI RMF

NIST AI RMF provides a structured framework for managing AI risks across the design, development, deployment, and use of AI systems. NIST also published a Generative AI Profile in 2024.

OWASP GenAI Security Project

OWASP focuses specifically on security risks associated with LLMs and generative AI applications. Its 2025 guidance covers prompt injection, sensitive-information disclosure, supply-chain issues, data/model poisoning, excessive agency, vector and embedding weaknesses, and unbounded consumption, among others.

MITRE ATLAS

MITRE ATLAS provides a living knowledge base of AI-specific adversary tactics and techniques based on real-world observations and AI red-team activities.

ISO/IEC 42001

ISO/IEC 42001 specifies requirements for establishing, implementing, maintaining, and continually improving an Artificial Intelligence Management System within organizations that develop or use AI.

25. Five key takeaways for public-sector organizations

1. AI is also an information system.

AI introduces new capabilities, but it remains part of the organization's broader information-system environment.

2. AI creates additional assets that must be protected.

Security must cover not only servers and networks but also models, training data, prompts, inputs, outputs, vector databases, and agent permissions.

3. Data is a core security issue.

Compromised, unauthorized, or improperly classified data can directly affect AI behavior and outputs.

4. AI permissions must be minimized.

As agentic AI becomes more capable of interacting with other systems, least privilege, human approval, and emergency controls become increasingly important.

5. Security must cover the entire lifecycle.

Security should be built into:

Collection → Training → Construction → Operation → Disposal

rather than added only after deployment.

26. Conclusion

AI can significantly improve productivity and public services. However, securely adopting AI is a different challenge from simply adopting AI.

The central message of Chapter 1 of the AI Security Guidebook can be summarized as follows:

Adopting AI and adopting AI securely are two different tasks.

Public-sector organizations should therefore look beyond the AI model itself and consider the entire ecosystem:

Data → Model → Infrastructure → User → External Connections → Supply Chain → Operations → Disposal

As AI evolves from generative AI toward agentic AI and physical AI, the key security questions will increasingly become:

• What can the AI see?

• What data can it access?

• What systems can it control?

• What actions can it perform?

• Who must approve sensitive actions?

• How can the AI be monitored?

• How can the system be stopped when something goes wrong?

For public-sector IT officials, incorporating these questions into AI project planning from the earliest stage can help establish a more systematic approach to AI security.

This article is a copyright-conscious, independently written explanation based on the AI Security Guidebook and publicly available international AI-security materials. It does not reproduce the source documents verbatim. For actual government projects, the latest version of the applicable AI Security Guidebook, national security guidelines, institutional security policies, and relevant procedures should be reviewed together.