Advanced AI Security: Red Teaming, Adversarial Testing, and ML Supply Chain Hardening

Basic AI security — encryption, access controls, network segmentation — is table stakes. It protects against generic infrastructure threats but doesn't address the attacks unique to AI systems: adversarial inputs that fool models, supply chain compromises in ML frameworks, training data poisoning, and model extraction attacks. Advanced AI security requires specialized techniques that most security teams haven't yet adopted.

AI Red Teaming

AI red teaming goes beyond traditional penetration testing. It combines infrastructure security testing with AI-specific attack techniques to evaluate the complete threat surface of an AI system.

Red Team Scope for AI Systems

Red Team Methodology

Phase 1: Reconnaissance

• Map the complete AI system: data sources, training pipelines, model registry, serving infrastructure, monitoring systems
• Identify model types, frameworks, and deployment patterns
• Enumerate API endpoints, input formats, and output structures
• Review public information: model cards, documentation, research papers referencing the model architecture

Phase 2: Threat Modeling

• Apply STRIDE (Spoofing, Tampering, Repudiation, Information Disclosure, DoS, Elevation of Privilege) to each AI component
• Map MITRE ATLAS (Adversarial Threat Landscape for AI Systems) tactics and techniques to the target system
• Identify the most valuable assets: training data, model weights, inference data, customer data
• Prioritize attack scenarios by impact and feasibility

Phase 3: Attack Execution

Evasion Testing:

• Generate adversarial examples using established techniques (FGSM, PGD, C&W for image models; TextFooler, BERT-Attack for NLP)
• Test with realistic perturbation budgets — adversarial examples should be imperceptible or plausible to humans
• Evaluate transferability: do adversarial examples crafted against a surrogate model also fool the target?

Extraction Testing:

• Query the model API systematically to build a dataset of input-output pairs
• Train a surrogate model on the collected data and evaluate agreement with the target model
• Assess whether rate limiting and query budgets effectively prevent extraction

Prompt Injection Testing (LLMs):

• Direct injection: attempt to override system prompts with user input
• Indirect injection: place malicious instructions in documents/data the LLM processes
• Jailbreaking: attempt to bypass content policies through roleplay, encoding tricks, or multi-turn conversations
• Data exfiltration: attempt to extract system prompts, RAG content, or other users' data

Phase 4: Reporting

• Document each finding with: attack technique, steps to reproduce, impact assessment, evidence, and recommended remediation
• Classify findings by severity: Critical (immediate remediation), High (remediate before next deployment), Medium (remediate within quarter), Low (accept or address opportunistically)
• Provide executive summary suitable for board reporting and regulatory communication

Red Team Frequency

• High-risk AI systems: Annual comprehensive red team + continuous automated testing
• Medium-risk: Biannual red team + quarterly automated testing
• Low-risk: Automated testing in CI/CD pipeline; manual red team on significant changes

ML Supply Chain Security

The ML supply chain is broader and more complex than traditional software:

Attack Surface

• ML frameworks: PyTorch, TensorFlow, Hugging Face Transformers — large codebases with native code, frequent updates, and extensive dependency trees
• Pre-trained models: Models from Hugging Face Hub, TensorFlow Hub, or PyTorch Hub. Pickle-based serialization allows arbitrary code execution during model loading
• Training data: Public datasets, web-scraped data, third-party data providers — each a potential poisoning vector
• Container images: NVIDIA CUDA base images, ML framework images, custom training images — each layer is a potential compromise point
• Python packages: ML projects have hundreds of Python dependencies. Typosquatting and dependency confusion attacks target ML-specific packages

SLSA (Supply-chain Levels for Software Artifacts) for ML

Apply SLSA framework principles to the ML supply chain:

SLSA Level 1: Documentation

• Document all dependencies: ML frameworks, libraries, pre-trained models, datasets, container base images
• Generate SBOM (Software Bill of Materials) for all ML pipeline components
• Generate ML-BOM: equivalent for data and model artifacts (training data sources, pre-trained model provenance)

SLSA Level 2: Tamper Resistance

• Build ML pipelines in hosted, authenticated build environments (not developer laptops)
• Generate provenance metadata linking model artifacts to their source code, data, and build environment
• Use reproducible builds where feasible (challenging for ML due to non-determinism, but achievable with seed fixing and deterministic operations)

SLSA Level 3: Hardened Infrastructure

• Training pipelines run on hardened, ephemeral infrastructure with no persistent state
• Build environments isolated from development environments and the internet
• All inputs (code, data, models, libraries) verified against signed checksums before use

Sigstore for ML Artifacts

Sigstore provides keyless signing and verification for software artifacts. Apply it to ML:

• Sign container images: All training and serving container images signed with cosign. Admission controllers verify signatures before deployment
• Sign model artifacts: Model weights, configurations, and metadata signed at the model registry. Serving infrastructure verifies before loading
• Sign training data: Dataset hashes signed at creation. Training pipelines verify data integrity before use
• Transparency log: All signing events recorded in Rekor (transparency log) for auditability

Safe Model Serialization

• Avoid pickle: Python's pickle format allows arbitrary code execution. Pre-trained models serialized as pickle files can contain malicious payloads
• Use safetensors: A safe serialization format for tensor data that does not allow code execution. Recommended for all model weight storage
• Verify checksums: Always verify model file checksums against trusted sources before loading
• Sandbox model loading: If pickle must be used (legacy models), load models in sandboxed environments with restricted system access

Confidential Computing for AI

For the most sensitive AI workloads, encryption at rest and in transit isn't sufficient — data must be protected while being processed.

Technologies

• AMD SEV-SNP: Secure Encrypted Virtualization — encrypts VM memory with per-VM keys. Available on Azure Confidential VMs and AWS EC2 instances
• Intel TDX: Trust Domain Extensions — hardware-isolated VMs with attestation. Available on Azure and GCP
• NVIDIA Confidential Computing: GPU TEE (Trusted Execution Environment) protecting data on GPU memory. Available on H100 GPUs
• ARM CCA: Confidential Compute Architecture for ARM-based processors. Emerging for edge AI deployment

Use Cases

• Multi-party model training: Multiple organizations contribute data to train a shared model without any party accessing the others' raw data
• Regulated data processing: Process data subject to strict localization or sovereignty requirements with hardware-guaranteed isolation from the cloud provider
• Model IP protection: Protect model weights from extraction by the cloud provider or other tenants
• Privacy-preserving inference: Process customer data for inference without exposing it to the model operator

Deployment Architecture

• Confidential Kubernetes: Azure offers Confidential AKS with AMD SEV-SNP nodes. Deploy AI workloads in confidential pods with encrypted memory
• Attestation: Remote attestation verifies that the confidential environment is genuine and running expected code before releasing sensitive data or model weights
• Key management: Customer-managed keys stored in HSM, released only to attested confidential environments

Compliance Mapping

Building the Advanced Security Practice

Team Skills Required

• AI red team lead: Combined ML engineering and offensive security expertise. Rare skill set — often built through cross-training
• Adversarial ML specialist: Deep understanding of adversarial attacks and defenses for ML models
• Supply chain security engineer: Experience with SLSA, Sigstore, SBOM, and container security
• Confidential computing engineer: Knowledge of TEE technologies, attestation, and secure enclave programming

Phased Implementation

1. Months 1-3: ML supply chain hardening — pin dependencies, scan containers, implement Sigstore signing, switch to safetensors
2. Months 3-6: Adversarial testing integration — add adversarial testing to CI/CD for high-risk models. Implement automated prompt injection testing for LLMs
3. Months 6-9: First AI red team exercise — comprehensive assessment of highest-risk AI system
4. Months 9-12: Confidential computing pilot — deploy most sensitive AI workload on confidential infrastructure