In today's digital landscape, the security of container images is a critical yet often overlooked aspect of software development. The rise of AI and its integration into various systems has brought this issue to the forefront, highlighting the urgent need for robust image signing practices.
The Problem: Unsigned Images and Security Risks
Unsigned container images create a significant vulnerability in the delivery pipeline. They provide an open invitation for attackers to infiltrate systems at any stage, from the initial pull of a package to the injection of tampered artifacts into production builds. The lack of cryptographic evidence of modification leaves organizations vulnerable to malicious activities, including credential theft and impersonation.
The issue is further compounded by inconsistent practices within organizations. Some teams sign their images, while others skip this crucial step, creating gaps in the chain of trust. Additionally, the inheritance of base image security posture means that a single compromised base image can propagate across multiple services, spreading potential threats.
AI and the Changing Workload Landscape
The AI era has brought about a rapid evolution in workloads, with model weights, training datasets, and inference runtimes now shipping as OCI artifacts. This shift presents unique challenges, as traditional scanning methods struggle to keep up. For instance, a pickled PyTorch checkpoint has no CVE to match against, and safer serialization formats, while reducing the risk of code execution, do not provide information about the origin or integrity of the weights.
Real-world examples, such as the malicious PyTorch model found on Hugging Face in 2024, demonstrate the urgency of the problem. This model, which opened a reverse shell upon loading, highlights the need for proactive provenance checks. The emergence of model-specific scanning tools, while helpful, is also being evaded by attackers, as seen with the nullifAI models described by ReversingLabs.
AI also widens the attack surface by suggesting dependencies that may not pass human threat models. Tampered AI model artifacts can have far-reaching consequences, corrupting predictions, poisoning recommendations, or even taking actions in production environments, such as making API calls or incurring costs.
The Registry: A Key Player in Image Security
The registry plays a crucial role in image security, as it is the last system in the path that sees every artifact, knows who pushed it, and controls who can pull it. It holds the identity context, enforces access policies, and stores metadata about the image contents. Therefore, it is the ideal layer for implementing image signing practices that are consistent across all teams and pipelines without causing delays.
The Benefits of Image Signing
Image signing does not eliminate the risk of forgery, but it significantly reduces the attack surface. With signing and enforcement, tampering only works if the attacker compromises a narrowly scoped signer, making it an auditable event tied to an identity. Revoking the identity stops the entire fleet from trusting the compromised signer. This turns an invisible, unbounded problem into a scoped, attributable, and revocable issue.
Making Signing Invisible: Amazon ECR Managed Signing
Amazon ECR Managed Signing aims to remove the operational tax associated with image signing. It does this by absorbing the cost within the registry, making signing a property of pushing an image rather than a project each team has to undertake. The process involves creating a registry-level signing configuration with rules pairing signing profiles with repository filters. Every matching push is then signed automatically.
Managed Signing addresses the custody question by not giving users direct access to the signing keys. Instead, AWS Signer keeps the certificate and private key, ensuring that signing keys are never exposed in repositories, runners, or build logs. This enhances security and simplifies key management.
The verification and enforcement processes happen downstream, with a trust policy dictating the rules. This policy ensures that workloads only run if they carry a signature produced by a trusted identity and verified against the AWS Signer root. Admission controllers, such as Kyverno, block any image not meeting these criteria from running.
Conclusion: The Future of Image Security
The AI era demands a shift in how we approach image security. Vulnerability scanning, while important, is no longer sufficient. We must prove the origin and integrity of our images to ensure they have not been tampered with. Making cryptography the path of least resistance, as Amazon ECR Managed Signing does, is a crucial step towards achieving this goal. By letting the registry carry the operational tax, we can ensure that image signing becomes a standard practice, enhancing the security of our delivery pipelines and protecting against potential threats.
