Defense-in-Depth for Secrets Management: Discovery, Visibility, Leak Detection and AI

In the past, many security teams considered securing secrets enough – if your secrets were secured, you were good. While you’re still kind-of-good staying on this course, security professionals increasingly recognize that just securing secrets is not enough – organizations require a more sophisticated solution to help protect themselves in today’s increasingly sophisticated threat landscape.

For large organizations with vast application portfolios, safeguarding these secrets and the newly created secrets from agile dev teams is challenging. And the growing number of secrets can increase the cyber debt associated with unsecured applications if not addressed. Fortunately, with a defense-in-depth approach, AI and automation, security teams increasingly have opportunities to further address and improve the security of the organization’s machine identities and reduce cyber debt.

“Security professionals increasingly recognize that just securing secrets is not enough – organizations require a more sophisticated solution to help protect themselves in today’s increasingly sophisticated threat landscape.”

The High Cost of Unknown Secrets and Leaks

Security teams recognize that they can only secure the secrets and machine identities they know. The problem is that security is often unaware of all the secrets, especially with cloud workloads. And essentially, if you’re unaware of a secret, you’re probably not securing it.

Organizations also increasingly recognize that when secrets management best practices are not followed, there is an increased likelihood of secrets being leaked and exposed publicly.

And with all the ongoing breaches of human and machine identities, there’s increased awareness that the consequences of a secrets breach can be devastating for the enterprise. Organizations look to remediate the situation rapidly when a problem, such as an unsecured secret, is discovered or a leaked secret is detected.

Secrets and Vault Sprawl: A Growing Challenge

But why now? What’s changed, that makes just vaulting secrets no longer enough? While cloud, digital transformation and DevOps adoption have driven incredible business benefits, they’ve also created many secrets and, too often, vault sprawl. Cloud service providers’ (CSPs) built-in secrets stores, such as AWS Secrets Manager (ASM) and Azure Key Vault (AKV), are becoming increasingly popular and widely used by development teams – they can be very effective tools. Dev teams can easily start using the built-in secrets stores, creating a store for each project or on an account or subscription basis. This is often encouraged by the CSP – Azure, for example, recommends using separate key vaults. The result is multiple secrets stores, running into potentially thousands of secret stores in a large enterprise. In many organizations, the challenge for security teams is that the vaults can be created without their knowledge. So, security teams are too often, especially when less familiar with the cloud, unaware of the secret stores and the secrets in these stores.

The Challenge of Securing Secrets for Security Teams

So now developers and cloud teams are establishing new secrets stores and securing secrets in them, without security being aware that the secret stores even exist. Okay, but is that so bad? The secrets are at least vaulted in a secrets store. Still, while this can provide some level of security, this approach doesn’t scale and is unlikely to meet corporate compliance and security policies. Some critical problems with this approach are:

1. Security teams are unaware of the secrets in the secret stores, the policies followed (or not followed), and how effectively they are being secured – e.g., what are the secrets used for? What does the secret provide access to? What are the security policies? Which apps are using these secrets?
2. Multiple secrets stores create vault sprawl and secrets sprawl. Instead of the enterprise having a single copy of a secret that’s secured and managed in a single centralized vault, with vault sprawl, there are multiple copies of the same secret in numerous vaults and even across multiple cloud environments. If attackers can’t steal from one vault, they will just find another that’s less secure.
3. Secrets rotation is typically not supported by the cloud providers’ built-in secrets stores. And with secrets sprawl, secrets rotation often becomes essentially impossible when multiple apps potentially use the same secrets in production environments. Unless the secrets are rotated in a coordinated manner, one or more production apps will go down because they are still using the old credentials.
4. Meeting compliance and audit requirements is challenging without a centralized view of secrets. How will auditors be comfortable that applications and secrets are in compliance if the same secret is stored in multiple secrets stores, and with the uncertainty that there may be some secrets stores they are not even aware of?

So why doesn’t the corporate security team simply get developers to secure secrets in a central store? Well, let’s get real – security is rarely going to be able to change the developers’ workflows and behavior. So, how can security teams secure these secrets scattered across potentially thousands of secrets stores? Well, the good news is that there are solutions that can centrally secure secrets without forcing changes in the developers’ workflows.

Protecting Secrets: Are You Responsible for What You Don’t Know?

In many organizations, security is likely to be held responsible for secrets they may not even know exist. It’s not a great situation, so first, let’s figure out what secrets exist. Fortunately, security teams can increasingly get visibility and secure the secrets in these built-in secrets stores. Discovery tools are becoming increasingly important because they can give security visibility into all the secrets stores across the enterprise.

These discovery tools can also derive insights such as when the secret was last used, whether it is idle or unused – and when it was last rotated – if at all. They can also provide a mechanism to track the owner of the secrets stores and the secrets. Most importantly, security teams can now assess and prioritize which secrets and secrets stores they want to centrally manage in a vault and potentially even work with the cloud and development teams to ensure that best practices are followed.

Automating Onboarding Secrets: The Ideal Solution

Of course, in an ideal coding environment, there would be no unmanaged secrets to discover. For example, as soon as the developer uses their AI-assisted IDE (integrated development environment) to write the code that requires a secret, simply having the secret request in the code would ensure that the secret is centrally secured or a policy established for securing the secret automatically at run time. Effectively, this approach shifts left and creates the secret within the infrastructure as code so that the newly created secret is automatically onboarded and centrally secured when the infrastructure is spun up.

Unfortunately, in most situations today, this is not the case and secrets are not automatically onboarded to a central secrets store managed by the application security team. Instead, too often, they are unsecured (e.g., hard-coded) or dropped into one of the CSP’s built-in secrets stores – a store that security is potentially unaware of. So, in this case, the security team will need first to discover the secrets and then, once found, centrally secure them.

AI Tools: Offering Insights and Promising More

Once you can discover secrets across all the various secrets stores, AI tools coupled with discovery, other secrets tools and automation promise to detect anomalies, force rotations and prioritize which secrets need to be onboarded, providing another layer of defense. Today, we’re still in the early stages of determining the most effective use cases for AI to improve the security of machine identities and their associated secrets, and it’s a fast-moving area.

Leak Detection: Preventing Incidents When Best Practices Aren’t Followed

Of course, development and cloud teams do not always follow best secrets management and coding practices. We’re all human, and too often, there’s the potential for human error, workarounds and quick fixes that can expose secrets that should have been secured across public code repositories.

Consider a developer prototyping an automation script or app for a quick demo that needs some credentials – it’s just too easy to hard-code the secret. Attackers don’t care if it’s a simple script or a complex app – they just want to steal that database credential or cloud access key. Of course, once the secret is rotated, the hard-coded secret is useless, but until it’s rotated, it’s something an attacker can exploit. In another scenario, the hard-coded secret has been removed and vaulted.

So, is it all good?

Well, no, not until it’s rotated.

What happens when the code with a hard-coded secret is posted to a public Git repository?

It’s exposed.

Yes, these edge cases shouldn’t happen; the problem is that they occur. Over 10 million secrets and credentials were detected in Git repositories in 2022, according to GitGuardian.

These are examples of secrets you thought you’d safely secured but were leaked by human error. An added layer of defense is to secure secrets in a central vault and continually check that none of these secrets have been inadvertently leaked because of human error. Of course, once a leaked secret is detected, it needs to be rotated so that any existing copies are invalidated.

Automated Remediation: Reducing Cyber Debt

Another challenge for security teams is the vast number of applications, secrets stores and cloud use cases. There are many secrets and machine identities, and more are being added constantly. Unsecured applications are often referred to as cyber debt, and reducing cyber debt is becoming an increasing challenge for security teams, forcing them to at least keep pace with development teams to prevent it from growing further.

Cyber debt and the desire for multiple layers of defense make for an increasingly compelling need to not only detect problems such as unsecured secrets in the cloud providers’ secrets stores or leaked secrets but also to provide automated tools for remediating these challenges. Without a doubt, AI tools are going to help security teams detect potential issues and anomalies, and this will, of course, make automated remediation of anomalies even more critical.

While attackers will inevitably become more aggressive and innovative in exploiting machine identities, security teams have the tools to protect their organizations more effectively. Fortunately, with layered defenses, discovery, leak detection, AI tools, automation and other innovative approaches, we can prevent an unsecured secret from exposing the entire organization to an attack.

Chris Smith is a director of product marketing at CyberArk.