As technologists, it's our job to keep the systems that we work on secure. That means protecting sensitive data, keeping systems up to date with patches and security updates, and being mindful of the various ways that an attacker could potentially breach our defenses. Thinking like an attacker is a great perspective for bolstering security posture. You've likely heard "Security is everyone's job!" everywhere. It is. I try to avoid speaking in absolutes... but it is. Security is everyone's job.
Secrets management is often an afterthought or simply overlooked. Secrets are anything that should not be shared with unauthorized eyes. They may be an API key, a password, an AWS Secret Access Key, or the code to the vault in the CEO's office with fourteen million dollars in cash. If these secrets fall into the wrong hands, it can spell disaster for an organization.
Imagine this scenario: you're working on a new feature for your company's website, and you need to use an API key to access a third-party service. You hardcode the key into your code, thinking nothing of it, and push the code live to production.
A few days later, you get a call from your boss. "Hey, there's some weird activity happening on our website. Sales are through the roof, but we have no idea why."
You take a look at the logs and realize that someone has managed to steal the API key you hardcoded into the code. There's an argument to be made here that it wasn't stolen but, rather, that it was given to them. They're using it to scrape the site and make fake purchases, racking up thousands of dollars in fraudulent charges.
This might sound like a far-fetched scenario, but it's actually a fairly common occurrence. Secrets that are hardcoded into code are often stored in version control systems like Git, which means that anyone with access to the repository can see them. Or... maybe they're stored as an environment variable on a CI/CD tool that suffers a security breach.
That's why it's so important to use a secrets management system. These systems allow you to store secrets in a secure location, and retrieve them as needed at runtime. This means that secrets are never stored in version control, and are much less likely to fall into the wrong hands. Here are some considerations that have helped me and teams I've worked with make productive decisions:
Maintain as few secrets management solutions as necessary. This reduces confusion about which secrets live where.
Adopt and agree on a secrets naming convention and follow it militantly. Making a guess about if you're rotating the correct secret can be critically dangerous in production environments. These mishaps may end with a post-mortem meeting where the lack of system availability cost $THE_COMPANY $500,000 per hour of downtime.
Where possible and prudent, select a system which allows you to maintain some level of history. Rollbacks and history can be lifesavers.
Restrict access to secrets to non-human accounts whenever and wherever possible. This reduces error and, frankly, the less that someone has to manage a secret... it's less likely to get compromised or exposed.
For secrets running in services, inject the secrets at runtime.
Automated secrets rotation is a great way to kick this up a notch. But all of this needs buy-in. Write the stories with precision, consider the customer benefit over your own, prioritize the stories in a feature/epic, get the work planned and execute. You have to sell the vision to clients and stakeholders so you may achieve mutual understanding for why all of this is so important. An investment of one sprint will eventually pay back in reduced production downtime and reduced time-to-recovery as a consequence of a vendor breach.
So next time you're working on a project and need to use a sensitive piece of information, remember the importance of secrets management. Don't hardcode that API key. Use the .gitignore file. Rotate that Personal Access Token that you've had since 1993. Prevent that on-call event that you'll get while you're spending time with your family.
Time is a resource for attackers and a risk consideration for
all others. In today's world it is not "if a breach happens" but rather "when a breach happens".
