Security & Compliance for AI¶

Why AI workloads need extra security attention¶

AI workloads introduce attack surfaces that traditional containerized applications do not. A compromised model serving endpoint is not just a data breach — for physical AI, it is a potential safety incident. A tampered robot policy could cause physical harm. A poisoned training dataset could produce a model that behaves incorrectly in subtle, hard-to-detect ways.

This chapter covers the security controls that address these AI- specific risks, using the tools and patterns available on OpenShift.

Container image signing with Sigstore¶

The problem¶

When a model serving container starts on a factory-edge cluster, how do you verify that it is the exact image your CI pipeline built and signed — not a tampered version injected by a compromised registry or man-in-the-middle attack?

The solution: Sigstore¶

Sigstore is an open-source project (hosted by the Linux Foundation) that provides tools for signing, verifying, and protecting software supply chains:

Cosign: Signs container images with cryptographic signatures.
Fulcio: Issues short-lived signing certificates based on OIDC identity (keyless signing — the CI runner's identity becomes the signing identity).
Rekor: An immutable transparency log that records every signing event. Provides tamper-resistant proof that a specific image was signed by a specific identity at a specific time.

ClusterImagePolicy on OpenShift¶

OpenShift 4.18+ supports ClusterImagePolicy CRDs that enforce signature verification cluster-wide. When a ClusterImagePolicy is created, the Machine Config Operator updates the container runtime configuration on all nodes to require valid signatures before pulling images from specified registries.

apiVersion: config.openshift.io/v1alpha1
kind: ClusterImagePolicy
metadata:
  name: require-signed-images
spec:
  scopes:
    - "registry.example.com/production/*"
  policy:
    rootOfTrust:
      policyType: PublicKey
      publicKey:
        keyData: <base64-encoded-public-key>

If someone attempts to deploy an unsigned or tampered image, the container runtime rejects the pull. The rejection is visible in pod events and auditable.

Supply chain security¶

SBOMs (Software Bill of Materials)¶

An SBOM lists every software component in a container image — libraries, their versions, and their dependencies. Generated by tools like Syft in SPDX JSON format, SBOMs are attached as image attestations.

For AI workloads, the SBOM captures not just the application code but also the ML frameworks (PyTorch, TensorRT), CUDA libraries, model runtime dependencies, and any vendored components. This enables vulnerability scanning across the entire dependency tree.

SLSA provenance attestations¶

SLSA (Supply-chain Levels for Software Artifacts) provenance records capture how an artifact was built: the source commit, the build environment, the builder identity, and the dependencies used. Tekton Chains (part of Red Hat OpenShift Pipelines) automates this — it observes every pipeline run, generates in-toto provenance attestations, signs them with Cosign, and stores them.

The result: for any deployed container, you can verify not just its signature but its entire build provenance — which commit, which pipeline, which builder, at what time.

The provenance chain for physical AI¶

For a deployed robot policy, the full provenance chain traces:

Deployed InferenceService
  └─ Signed container image (Cosign signature, SBOM, SLSA provenance)
      └─ Model checkpoint (registered in Model Registry)
          └─ Training run (MLflow experiment, hyperparameters, metrics)
              └─ Training dataset (synthetic data from Isaac Sim)
                  └─ Simulation scene (USD scene, domain randomization config)
                      └─ Source assets (Nucleus, version-controlled)

Every link is cryptographically verifiable. An auditor can navigate from the serving endpoint back to the source simulation scene and verify that every intermediate step is authentic and unmodified.

Compliance scanning¶

The Compliance Operator¶

The Compliance Operator installs from OperatorHub and runs OpenSCAP- based compliance scans against predefined profiles. For defense and regulated industry contexts, the key profiles are:

DISA STIG: Security Technical Implementation Guides published by the Defense Information Systems Agency. The profiles ocp4-stig-v2r3 and ocp4-stig-node-v2r3 scan the OpenShift platform and RHCOS nodes against DISA requirements.

The operator creates scan CRDs, runs the scans as pods, and produces ComplianceCheckResult CRs for each rule. Results categorize as PASS, FAIL, MANUAL (requires human verification), or NOT-APPLICABLE.

Failed checks can be auto-remediated via ComplianceRemediation CRs, which generate MachineConfigs or other corrective resources. However, auto-remediation should be disabled in production and remediations reviewed by a human before application — a safety practice that itself satisfies compliance requirements for change management.

ACM policy integration¶

ACM policies can require managed clusters to maintain compliance scanning. A hub-level Policy enforces that every spoke cluster has the Compliance Operator installed, the correct scan profiles configured, and scans running on schedule. Violations surface in the ACM console for fleet-wide compliance visibility.

Compliance Operator

Secrets management with Vault¶

HashiCorp Vault + Vault Secrets Operator (VSO)¶

Vault provides centralized secrets management. VSO (installed from OperatorHub) syncs secrets from Vault into native Kubernetes Secrets:

VaultConnection: Points to the Vault server.
VaultAuth: Configures authentication (Kubernetes auth — pods authenticate via their ServiceAccount token, no static credentials).
VaultStaticSecret: Declares which Vault KV path to read and which Kubernetes Secret to create.

VSO automatically refreshes secrets on a configurable interval and overwrites manual edits to Kubernetes Secrets, ensuring the Vault value is always authoritative.

Why Vault matters for AI¶

AI workloads need credentials for:

Object storage: S3 keys for accessing model weights, training data, and pipeline artifacts.
Model registries: NGC API keys for pulling NIM containers and gated model weights from HuggingFace.
Git repositories: Tokens for GitOps access to the infrastructure repository.
Inter-service auth: Tokens for Kafka, Nucleus, and API endpoints.

None of these should be in Git. Vault provides a single source of truth for credentials, with audit logging, rotation support, and fine-grained access control.

Vault Secrets Operator

Network segmentation¶

NetworkPolicies¶

Kubernetes NetworkPolicies provide namespace-level traffic isolation. For AI workloads:

Inference services only accept traffic from the API gateway
Training jobs only access object storage and the training data namespace
Fleet management services only accept traffic from known producers (Kafka, API endpoints)
Monitoring agents have broad read access but no write access to workload namespaces

A default-deny-ingress policy per namespace, with explicit allow rules for each required communication path, is the baseline.

Service Mesh (Istio)¶

OpenShift Service Mesh (based on Istio) provides mTLS between services — every pod-to-pod communication is encrypted and authenticated with mutual TLS certificates. The mesh also provides:

Traffic observability (who is talking to whom, at what volume)
Rate limiting (protect inference endpoints from overload)
Circuit breaking (isolate failing services)

For physical AI, the mesh is particularly valuable for the inference path: camera feeds, perception results, and action commands all flow over mTLS-encrypted connections.

AI-specific security considerations¶

Model poisoning¶

A tampered training dataset or modified model weights can cause the model to behave incorrectly in subtle ways — correct most of the time, but wrong in specific triggered scenarios. Defenses:

Data provenance: Track training data lineage through MLflow. Verify that the dataset used for training is the expected one.
Model signing: Sign model checkpoints with Cosign. Verify signatures before serving.
Evaluation diversity: Evaluate models on diverse test scenarios, including adversarial inputs, before promotion.

Adversarial inputs¶

Inference endpoints are attack surfaces. Carefully crafted inputs can cause models to produce wrong outputs. Defenses:

Input validation: Type checking, range validation, and anomaly detection on inference inputs.
Rate limiting: Service Mesh rate limits prevent brute-force probing of model behavior.
Monitoring: Track inference distributions for anomalies that suggest adversarial probing.

Credential exposure¶

Model serving pods often need access to multiple external systems (storage, registries, databases). Vault + VSO ensures credentials are never committed to Git, are short-lived, and are rotated automatically.

Key takeaways¶

AI workloads have unique security surfaces: model poisoning, adversarial inputs, and the physical consequences of compromised robot policies.
Sigstore + ClusterImagePolicy provides container image verification at the admission level — unsigned or tampered images are rejected.
SBOMs and SLSA provenance provide the full supply chain audit trail from build to deployment.
Compliance scanning (DISA STIG) provides evidence of security posture for regulated industries.
Vault manages all credentials centrally, with Kubernetes-native projection and automatic rotation.
Network segmentation (NetworkPolicies + Service Mesh) isolates AI workloads from each other and from non-AI services.