Safeguarding AI Secrets: A Practical Guide to API Key Rotation After a Breach

Overview

In the wake of the recent security incident at Braintrust, an AI services firm, attackers gained access to an Amazon Web Services (AWS) account and exfiltrated secrets used to access AI provider APIs. The breach underscores a critical reality: API keys are the lifeblood of modern AI pipelines, and once compromised, they expose your models, data, and cloud resources to misuse. This guide walks you through a structured process for rotating API keys after a breach, using the Braintrust case as a cautionary example. You'll learn how to identify compromised credentials, generate new keys, update dependent services, revoke old keys, and audit your environment to prevent recurrence. By following these steps, you can minimize damage and restore security quickly.

Prerequisites

Before beginning the key rotation process, ensure you have the following:

• AWS Console or CLI access – with credentials that have permissions to read and modify secrets in AWS Secrets Manager, Parameter Store, or S3 buckets where keys may be stored.
• Inventory of all API keys – a list of keys used for AI providers (e.g., OpenAI, Anthropic, Hugging Face), along with their purpose and the services consuming them.
• Incident response plan – a documented procedure for breach containment, communication, and recovery.
• Access to application source code – to update environment variables, configuration files, or CI/CD pipelines that reference the old keys.
• Monitoring tools – CloudTrail, CloudWatch, or third‑party SIEM to detect unauthorized usage of rotated keys.

If you lack any of these, pause the rotation until you can obtain them. Rushing without a complete inventory can lead to missed credentials and prolonged exposure.

Step‑by‑Step Guide to API Key Rotation

1. Identify Compromised Keys

The Braintrust breach started when attackers accessed an AWS account. Your first step is to determine exactly which keys were exposed. Review your AWS CloudTrail logs for unusual activity, such as unauthorized DescribeSecret or GetSecretValue calls. Use the AWS CLI to list recent secret access:

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=GetSecretValue --start-time 2025-03-01T00:00:00Z

Cross‑reference the results with your secret inventory. Also check for any keys stored in plaintext in code repositories, environment variables, or logs. If your breach involved lateral movement, assume all keys in the same AWS account or project are compromised.

2. Generate New API Keys

Once you've identified the compromised keys, generate replacements from each provider's dashboard or API. For AI providers like OpenAI, use the API key management page to create a new key with the same permissions. For example, using OpenAI's CLI (if available):

openai api keys.create --description "rotated-after-breach" --org org-abc123

Record the new key value securely – it may be displayed only once. Never store it in plain text. Use a password manager or temporary encrypted file.

3. Update Secrets in AWS

With new keys in hand, update your secret storage in AWS. If you use Secrets Manager, update the secret value with the new key:

aws secretsmanager put-secret-value --secret-id ai-provider-keys --secret-string '{"openai_key":"sk-new...","anthropic_key":"sk-ant-new..."}'

If you store keys in Parameter Store or S3, update those accordingly. Do not delete the old key yet – you need to update applications first to avoid service disruption.

4. Update Dependent Applications

Now update all services, microservices, and scripts that read the compromised keys. The safest approach is to redeploy applications with an updated configuration that points to the new secret. For example, in a Kubernetes deployment, update the env section of your pod spec:

env:
  - name: OPENAI_API_KEY
    valueFrom:
      secretKeyRef:
        name: ai-keys
        key: openai_key

Then apply the change and restart pods. For serverless functions (like AWS Lambda), update environment variables directly or use Secrets Manager integration. If your applications read keys at runtime (e.g., via the AWS SDK), they will automatically pick up the updated secret from the store, but you must ensure the IAM role has the correct permissions.

After updating all production environments, test each service to confirm it can make API calls with the new key.

5. Revoke Old Keys

Once all applications are using the new keys, revoke the old ones. On cloud provider dashboards, delete or deactivate the old key. For OpenAI, you can delete via the API:

openai api keys.delete --key sk-old-exposed-key

Also consider updating any IAM policy that restricts key usage by matching key patterns. After revocation, monitor logs for any failed authentication attempts using the old keys – this may indicate an attacker still trying to use them, which you can block at the network level as well.

6. Audit and Monitor for Residual Risks

Rotation is not the end. Conduct a thorough audit to ensure no keys were missed. Use AWS Config rules or custom scripts to scan all code repositories for hardcoded secrets. Enable automatic rotation on AWS Secrets Manager for future use. Finally, review IAM policies and AWS account permissions. The Braintrust breach likely exploited excessive permissions – follow the principle of least privilege for all secrets access.

Set up CloudWatch alarms for unexpected API key usage, such as calls from unusual IPs or at odd hours. Consider using a service like Vault or HashiCorp Boundary for dynamic secrets management to reduce the blast radius of future breaches.

Common Mistakes

• Not rotating all keys simultaneously – if you rotate only some keys, attackers may still have access through others that were also exposed. Always assume all secrets in the compromised account are compromised.
• Delaying rotation – every minute an old key is valid increases the window for abuse. Begin rotation as soon as the breach is confirmed, even if you haven’t fully identified the scope.
• Forgetting to update CI/CD pipelines – automated deployment scripts often store keys in environment variables or encrypted files. A missed pipeline can fail after revocation, causing production outages.
• Storing new keys in the same insecure location – after rotation, move keys to a managed secret store with encryption and access logging.
• Ignoring third‑party dependencies – if your application calls external APIs that use an API key indirectly (e.g., a SaaS tool), those keys may also need rotation.
• Failing to communicate with teams – notify all developers, operations, and security stakeholders about the rotation plan and timeline to avoid accidental overwrites or confusion.

Summary

The Braintrust data breach is a stark reminder that API key hygiene is a cornerstone of AI security. By following this guide – identifying compromised keys, generating replacements, updating applications, revoking old keys, and auditing – you can contain a breach and restore trust in your infrastructure. Implement automatic rotation policies and least privilege access to reduce the risk of future incidents. Remember: rotate early, rotate often, and never leave secrets exposed.