Secret management in CI/CD: Stop Using Long-Lived Credentials

We’ve all been there: staring at a CI/CD pipeline configuration file, wondering if that AWS_ACCESS_KEY_ID sitting in an environment variable is actually as "temporary" as the documentation claimed. Last month, while cleaning up a legacy repo, I found a database password hardcoded in a YAML file that hadn't been rotated since 2021. It’s a common reality, but it’s one we need to move past if we want to take our cicd security seriously.

Static credentials are a liability. When a secret lives forever, the window of opportunity for an attacker is infinite. The solution isn't just better storage; it's changing the lifecycle of the secret itself.

Why Static Secrets Fail You

Initially, we tried storing sensitive keys in encrypted CI/CD variables. It felt safe enough until a developer accidentally logged the environment variables during a failed build. The secret was now in the plain-text build logs, accessible to anyone with read access to the CI dashboard.

That experience taught me that secret management isn't about hiding things; it's about reducing the duration that a secret is valid. If a secret is leaked, it should be useless by the time an attacker realizes they have it.

The Shift to Ephemeral Secrets

Ephemeral secrets are generated on-the-fly, granted a specific TTL (time-to-live), and expire automatically. Instead of a hardcoded AWS key, your CI runner requests a temporary token from a secrets manager like HashiCorp Vault.

If you’re already running on Kubernetes, you might be familiar with the workflow of syncing secrets. I’ve written before about using HashiCorp Vault and External Secrets Operator: Secure Kubernetes Secrets to handle static values, but dynamic injection takes this a step further.

Here is how you can implement dynamic secret injection in your pipeline:

1. Identity Federation: Configure your CI/CD provider (like GitHub Actions or GitLab CI) to authenticate with your secret manager using OIDC (OpenID Connect). This removes the need for a "master secret" to access your vault.
2. Dynamic Generation: Configure your secret backend to generate credentials on demand. For example, when the pipeline starts, it asks Vault to create a temporary database user with SELECT permissions that expire in 15 minutes.
3. Injection: The pipeline injects this short-lived credential directly into the runtime memory of the build process.

Implementing Dynamic Secret Injection

I prefer using the Vault API directly within the pipeline shell to fetch these credentials. It avoids cluttering the CI environment variables.

Bash
# Example: Fetching a dynamic AWS STS token in a GitHub Action
export VAULT_ADDR="https://vault.internal.com"
export AWS_ROLE="my-ci-role"

# Authenticate via OIDC
VAULT_TOKEN=$(vault write -field=token auth/jwt/login role=$AWS_ROLE jwt=$ACTIONS_ID_TOKEN)

# Request dynamic creds
CREDS=$(vault read -format=json aws/sts/deploy-role)

export AWS_ACCESS_KEY_ID=$(echo $CREDS | jq -r .data.access_key)
export AWS_SECRET_ACCESS_KEY=$(echo $CREDS | jq -r .data.secret_key)
export AWS_SESSION_TOKEN=$(echo $CREDS | jq -r .data.security_token)

By the time the deploy job finishes—roughly 3 minutes later—those credentials are revoked. Even if someone dumps the build logs, the keys are dead.

Hard-Won Lessons

Moving to dynamic injection isn't a silver bullet. You’ll encounter a few hurdles:

• Latency: Every pipeline now has to "talk" to your secret manager. If your Vault instance is down or slow, your entire deployment pipeline stalls.
• Complexity: Debugging dynamic credentials is a nightmare because they expire before you can manually test them. I usually keep a "break-glass" role that lasts for 60 minutes for local troubleshooting, but that's a security trade-off you must document.
• Tooling: If you are already using Kubernetes Secret Management with HashiCorp Vault and ESO Guide, stick to the patterns your team already knows. Don't add a second secret manager just because you read a blog post about it.

Infrastructure as Code and Security

Effective credential rotation is impossible to do manually at scale. As you build out your infrastructure as code, ensure that your secrets management strategy is part of the architecture, not an afterthought. Treat the secret engine configuration as code—if it’s not in version control, you don’t have a repeatable security posture.

When I look back at my early projects, I realize we spent too much time worrying about how to encrypt the secrets and not enough time worrying about how to expire them. If you’re just starting, don't try to solve everything at once. Start by OIDC-enabling your CI/CD pipelines. It’s the single biggest step you can take to stop using long-lived credentials.

I’m still experimenting with how to handle service-to-service auth in non-Kubernetes environments, as that remains the "wild west" of internal networking. It's a messy problem, but one that’s worth solving.

FAQ

Q: Does dynamic injection add significant latency to my builds? A: In my experience, it usually adds about 200-500ms per secret request. It’s negligible compared to the time it takes to pull a Docker image or run npm install.

Q: What happens if the secret manager goes down? A: Your pipelines fail. This is why you need a highly available secrets manager. If you can't afford a multi-region Vault cluster, maybe stick to static secrets with frequent, automated rotation using a tool like AWS Secrets Manager.

Q: Do I need Vault to do this? A: Not strictly. Cloud providers have built-in IAM roles for CI/CD that provide ephemeral tokens (like GitHub OIDC to AWS IAM Roles). Start there before building a complex self-hosted secrets infrastructure.