3 Kubernetes Risks and What to Do About Them

Today, speed trumps nearly everything when it comes to software development. And as digital business requirements evolve, developers are being asked to work even faster and with more agility than ever before. Take Netflix, for example: to meet society’s need for new, binge-worthy watches, the company delivers hundreds of microservices and thousands of deployments each day.

Containers are often used by developers to meet this need for speed by packaging both software and its necessary dependencies together. And according to a 2021 Red Hat study, 88% of those teams rely on Kubernetes to manage their containerized workloads and services, automate processes and move faster and at scale.

Kubernetes and the Not-so-Secret Security Challenge

Despite its popularity, and perhaps partly because of it, the Kubernetes orchestration platform introduces new identity-centric security challenges that must be tackled early in the development cycle. Otherwise, small issues can grow quickly, resulting in costly delays and defects.

The same Red Hat study referenced above found that almost all developers (94%) experienced at least one Kubernetes-related security incident in the last 12 months, while more than half have delayed deploying applications into production due to security concerns.

In dynamic cloud-native environments, secrets — such as passwords, SSH Keys, certificates and API tokens — are used widely to grant access to anyone (such as an application developer) or anything (such as an application) that knows them. Organizations become vulnerable when these secrets are leaked to logs, exposed in application code or mishandled in other ways. Exacerbating the issue, secrets have the unique ability to grant access to additional secrets and privileges, which is known as the “secret zero problem.”

Three Key Kubernetes Risks and Mitigation Steps 

Securing secrets in containerized environments requires the collaboration of both development and security teams. Here are three potentially vulnerable areas within Kubernetes to focus on as part of a true DevSecOps approach:

Kubernetes Risk #1: Hardware

Whether it runs on-premises or in a cloud managed by a third party, Kubernetes still requires a hardware platform. If an attacker can breach the virtual machine running Kubernetes and gain access to root privileges, the attacker can go on to breach the Kubernetes cluster.

Security best practices:

• Enforcing the principle of least privilege can go a long way in protecting the hardware that underpins both Kubernetes and the containers themselves. Provision virtual machines with the least number of privileges necessary to perform their function to make it difficult for attackers to gain root access.
• Rotate credentials often and also consider vaulting to further strengthen protections.

Kubernetes Risk #2: The Kubernetes API Server

Beyond the physical machines, the Kubernetes control plane must also be secured. This control plane has access to all the containers running in a cluster, including the Kubernetes API server, which acts as the front end of the control plane and facilitates user interaction within the cluster.

An attack on the API server can have big implications. Even a few stolen secrets or credentials can be used to escalate an attacker’s access and privilege, and what was first a small crack in the armor can quickly turn into a network-wide issue.

Security best practices:

• To minimize risk, start with blocking credential theft and malware threats on the endpoint — where many attacks begin — paying close attention to local machines used by users with administrative privileges in Kubernetes.
• Use multi-factor authentication (MFA) to authenticate access to the Kubernetes API server. That way, even if a credential is stolen, it can’t be authenticated or used to access Kubernetes. Development teams can also set the experimental-encryption-provider-config flag on the Kubernetes API server to bolster secrets protection.
• Once a user is authenticated in Kubernetes, they can access all the resources within the cluster, so managing permissions is critical. Role-based access control (RBAC) will help ensure users only have the access rights they need and aren’t overly permissioned.
• Similarly, least privilege should be enforced across Kubernetes service accounts, which are automatically created when a cluster is set up to help authenticate Pods. Secrets should also be rotated on a regular basis to keep access out of the hands of those who don’t need it.

Kubernetes Risk #3: Containers

Pods and the containers within them are the building blocks of a Kubernetes cluster and contain the information needed to run the application. There are several key areas of vulnerability within this container ecosystem and workflow, as illustrated below.  To explore them in depth, read our eBook “Securing Containers in a Kubernetes Environment.”

Security best practices: Consider these container security best practices, along with those outlined above:

• While it may seem logical, do not build secrets into the code or the container image. Otherwise, anyone with access to the source code will also have access to information in code repositories, logs and elsewhere.
• Avoid using environment variables for sensitive information. This will help prevent attackers from discovering clear text secrets with simple commands if they compromise access to the container API.
• Implement RBAC, limit secret access to the processes running inside a given container, and delete secrets/revoke access to resources when they are no longer needed to minimize exposure.
• Log usage, including when a secret is injected, rotated or removed from a container and regularly audit access to critical systems. Also, consider centralizing secrets management to make audit, access control and secrets management more manageable.

Arm your development teams with these best practices to help strengthen security across your organization’s entire Kubernetes environment. And look for ways to incorporate self-service capabilities into developers’ everyday processes (like code scanning, for example) to make life easier for everyone — and keep things moving quickly.

When security and development teams partner to help defend against attacks, drive operational efficiencies, and satisfy audit and compliance requirements, everyone wins.