Published on

Automated Secrets Rotation — Zero-Downtime Credential Rotation in Production

Authors

Introduction

Static secrets are a ticking time bomb. Developers copy database passwords into .env files, API keys sit unused for years, and when (not if) they leak, rotating them at 2 AM becomes a scramble. Automated secrets rotation eliminates this—new credentials are generated, deployed, and old ones are retired without downtime or manual intervention.

This post covers AWS Secrets Manager rotation, PostgreSQL password rotation without restart, blue/green secret versioning, and API key rotation with overlap periods for zero downtime.

AWS Secrets Manager Rotation Lambda

AWS Secrets Manager can automatically invoke a Lambda function to rotate credentials. Here's a production setup:

import {
  SecretsManager,
  SecretsManagerRotationRules,
} from '@aws-sdk/client-secrets-manager';

interface RotationConfig {
  secretId: string;
  rotationLambdaArn: string;
  rotationRules: {
    automaticallyAfterDays: number;
  };
}

async function enableRotation(config: RotationConfig): Promise<void> {
  const client = new SecretsManager({ region: 'us-east-1' });

  await client.rotateSecret({
    SecretId: config.secretId,
    RotationLambdaARN: config.rotationLambdaArn,
    RotationRules: {
      AutomaticallyAfterDays: config.rotationRules.automaticallyAfterDays,
      Duration: 3,
      DurationUnit: 'HOURS',
    },
  });

  console.log(`Rotation enabled for ${config.secretId}`);
}

// Example: Enable daily rotation for database password
enableRotation({
  secretId: 'prod/rds/postgres/password',
  rotationLambdaArn:
    'arn:aws:lambda:us-east-1:123456789012:function:rotate-postgres-password',
  rotationRules: {
    automaticallyAfterDays: 1,
  },
});

Rotation Lambda for PostgreSQL

The Lambda function handles the actual rotation logic:

import { SecretsManager, RDSData } from '@aws-sdk/client-secrets-manager';
import { RDSDataServiceV2 } from '@aws-sdk/client-rds-data';
import pg from 'pg';

interface RotationSecret {
  username: string;
  password: string;
  host: string;
  port: number;
  dbname: string;
}

interface RotationEvent {
  SecretId: string;
  ClientRequestToken: string;
  Step: 'create' | 'set' | 'test' | 'finish';
}

const secretsClient = new SecretsManager({ region: 'us-east-1' });
const rdsDataClient = new RDSDataServiceV2({ region: 'us-east-1' });

// Step 1: Create new password
async function createSecret(
  secretId: string,
  clientToken: string
): Promise<string> {
  const response = await secretsClient.getSecretValue({ SecretId: secretId });
  const currentSecret = JSON.parse(response.SecretString!) as RotationSecret;

  const newPassword = generateSecurePassword(32);
  const newSecret: RotationSecret = {
    ...currentSecret,
    password: newPassword,
  };

  await secretsClient.putSecretValue({
    SecretId: secretId,
    ClientRequestToken: clientToken,
    SecretString: JSON.stringify(newSecret),
    VersionStages: ['AWSPENDING'], // Staging stage
  });

  return newPassword;
}

// Step 2: Set new password in database
async function setSecret(
  secretId: string,
  clientToken: string
): Promise<void> {
  const response = await secretsClient.getSecretValue({
    SecretId: secretId,
    VersionId: clientToken,
    VersionStage: 'AWSPENDING',
  });

  const pendingSecret = JSON.parse(
    response.SecretString!
  ) as RotationSecret;
  const currentResponse = await secretsClient.getSecretValue({
    SecretId: secretId,
    VersionStage: 'AWSCURRENT',
  });
  const currentSecret = JSON.parse(
    currentResponse.SecretString!
  ) as RotationSecret;

  // Connect as current user to update password
  const conn = new pg.Client({
    host: currentSecret.host,
    port: currentSecret.port,
    database: currentSecret.dbname,
    user: currentSecret.username,
    password: currentSecret.password,
  });

  await conn.connect();

  try {
    // Update password in PostgreSQL
    await conn.query(
      `ALTER USER "${currentSecret.username}" WITH PASSWORD $1`,
      [pendingSecret.password]
    );
    console.log(`Password updated for user ${currentSecret.username}`);
  } finally {
    await conn.end();
  }
}

// Step 3: Test new credentials
async function testSecret(
  secretId: string,
  clientToken: string
): Promise<void> {
  const response = await secretsClient.getSecretValue({
    SecretId: secretId,
    VersionId: clientToken,
    VersionStage: 'AWSPENDING',
  });

  const secret = JSON.parse(response.SecretString!) as RotationSecret;

  const conn = new pg.Client({
    host: secret.host,
    port: secret.port,
    database: secret.dbname,
    user: secret.username,
    password: secret.password,
  });

  try {
    await conn.connect();
    await conn.query('SELECT 1');
    console.log('New credentials validated successfully');
  } finally {
    await conn.end();
  }
}

// Step 4: Finalize rotation
async function finishSecret(
  secretId: string,
  clientToken: string
): Promise<void> {
  const current = await secretsClient.describeSecret({ SecretId: secretId });
  const currentVersionId = current.VersionIdsToStages
    ? Object.entries(current.VersionIdsToStages).find(
        ([, stages]) => stages?.includes('AWSCURRENT')
      )?.[0]
    : null;

  await secretsClient.updateSecretVersionStage({
    SecretId: secretId,
    VersionStage: 'AWSCURRENT',
    MoveToVersionId: clientToken,
    RemoveFromVersionId: currentVersionId,
  });

  // Mark old version as deprecated after grace period
  setTimeout(() => {
    secretsClient.updateSecretVersionStage({
      SecretId: secretId,
      VersionStage: 'AWSDEPRECATED',
      MoveToVersionId: currentVersionId,
    });
  }, 3600000); // 1 hour grace period
}

// Lambda handler
export const handler = async (event: RotationEvent): Promise<void> => {
  const { SecretId, ClientRequestToken, Step } = event;

  console.log(
    `Rotation ${Step} for secret ${SecretId} with token ${ClientRequestToken}`
  );

  try {
    switch (Step) {
      case 'create':
        await createSecret(SecretId, ClientRequestToken);
        break;
      case 'set':
        await setSecret(SecretId, ClientRequestToken);
        break;
      case 'test':
        await testSecret(SecretId, ClientRequestToken);
        break;
      case 'finish':
        await finishSecret(SecretId, ClientRequestToken);
        break;
      default:
        throw new Error(`Invalid step: ${Step}`);
    }
  } catch (error) {
    console.error(`Rotation failed at step ${Step}:`, error);
    throw error;
  }
};

Connection Pool Reconnection Without Restart

Applications must reconnect to databases with new credentials. Use connection pool lifecycle hooks:

import { Pool, PoolClient } from 'pg';

class RotationAwarePool extends Pool {
  private secretId: string;
  private secretsClient: SecretsManager;
  private lastSecretVersion: string = '';

  constructor(secretId: string, region: string) {
    super();
    this.secretId = secretId;
    this.secretsClient = new SecretsManager({ region });
    this.startSecretMonitor();
  }

  private startSecretMonitor(): void {
    // Check for secret updates every 5 minutes
    setInterval(() => this.checkAndReconnect(), 5 * 60 * 1000);
  }

  private async checkAndReconnect(): Promise<void> {
    try {
      const secret = await this.secretsClient.describeSecret({
        SecretId: this.secretId,
      });

      const currentVersion = Object.entries(secret.VersionIdsToStages || {})
        .find(([, stages]) => stages?.includes('AWSCURRENT'))
        ?.at(0);

      if (
        currentVersion &&
        currentVersion !== this.lastSecretVersion
      ) {
        console.log('Secret rotated, draining old connections...');

        // Drain old connections
        this.idleClients.forEach((client) => client.end());
        this.idleClients = [];

        // Force reconnection with new credentials
        await this.refreshConnections();
        this.lastSecretVersion = currentVersion;

        console.log('Pool refreshed with new credentials');
      }
    } catch (error) {
      console.error('Failed to check secret version:', error);
    }
  }

  private async refreshConnections(): Promise<void> {
    const secretValue = await this.secretsClient.getSecretValue({
      SecretId: this.secretId,
    });

    const credentials = JSON.parse(secretValue.SecretString!);

    this.options = {
      ...this.options,
      password: credentials.password,
    };
  }
}

// Usage
const pool = new RotationAwarePool('prod/rds/postgres', 'us-east-1');

app.get('/api/data', async (req: express.Request, res: express.Response) => {
  const result = await pool.query('SELECT * FROM users LIMIT 10');
  res.json(result.rows);
});

Blue/Green Secret Versions

Maintain multiple secret versions to enable fast rollback:

interface SecretVersion {
  versionId: string;
  stage: 'AWSCURRENT' | 'AWSPENDING' | 'AWSDEPRECATED' | 'STABLE';
  createdAt: Date;
}

class BlueGreenSecretsManager {
  private secretsClient = new SecretsManager({ region: 'us-east-1' });

  async getCurrentSecret(secretId: string): Promise<string> {
    const response = await this.secretsClient.getSecretValue({
      SecretId: secretId,
      VersionStage: 'AWSCURRENT',
    });
    return response.SecretString!;
  }

  async getPendingSecret(secretId: string): Promise<string | null> {
    try {
      const response = await this.secretsClient.getSecretValue({
        SecretId: secretId,
        VersionStage: 'AWSPENDING',
      });
      return response.SecretString || null;
    } catch {
      return null;
    }
  }

  async listVersions(secretId: string): Promise<SecretVersion[]> {
    const response = await this.secretsClient.describeSecret({
      SecretId: secretId,
    });

    const versions: SecretVersion[] = [];
    for (const [versionId, stages] of Object.entries(
      response.VersionIdsToStages || {}
    )) {
      versions.push({
        versionId,
        stage: stages?.[0] as any,
        createdAt: new Date(response.CreatedDate || 0),
      });
    }

    return versions.sort(
      (a, b) => b.createdAt.getTime() - a.createdAt.getTime()
    );
  }

  async rollbackToStable(secretId: string): Promise<void> {
    const versions = await this.listVersions(secretId);
    const stableVersion = versions.find((v) => v.stage === 'STABLE');

    if (!stableVersion) {
      throw new Error('No stable version available for rollback');
    }

    const current = versions.find((v) => v.stage === 'AWSCURRENT');
    await this.secretsClient.updateSecretVersionStage({
      SecretId: secretId,
      VersionStage: 'AWSCURRENT',
      MoveToVersionId: stableVersion.versionId,
      RemoveFromVersionId: current?.versionId,
    });

    console.log(`Rolled back to stable version ${stableVersion.versionId}`);
  }
}

// Automatic rollback on authentication errors
const pool = new Pool();
let consecutiveAuthErrors = 0;
const MAX_AUTH_ERRORS = 3;

pool.on('error', async (err) => {
  if (
    err.message.includes('password') ||
    err.message.includes('authentication')
  ) {
    consecutiveAuthErrors++;

    if (consecutiveAuthErrors >= MAX_AUTH_ERRORS) {
      console.error('Multiple auth errors, attempting rollback...');
      const manager = new BlueGreenSecretsManager();
      try {
        await manager.rollbackToStable('prod/rds/postgres');
      } catch (rollbackErr) {
        console.error('Rollback failed:', rollbackErr);
      }
    }
  }
});

API Key Rotation with Overlap Period

Dual-key setup allows seamless rotation without breaking clients:

interface ApiKey {
  keyId: string;
  secret: string;
  status: 'active' | 'pending' | 'deprecated';
  createdAt: Date;
  rotatedAt: Date;
}

class ApiKeyRotationManager {
  async rotateApiKey(userId: string): Promise<{ old: ApiKey; new: ApiKey }> {
    const oldKey = await db.apiKeys.findActive(userId);
    const newKey = this.generateKey(userId);

    // Mark old key as pending deprecation
    await db.apiKeys.update(oldKey.keyId, {
      status: 'pending',
      rotatedAt: new Date(),
    });

    // Create new key in active state
    await db.apiKeys.create(newKey);

    // After 24 hours, mark old key as deprecated
    setTimeout(async () => {
      await db.apiKeys.update(oldKey.keyId, { status: 'deprecated' });
    }, 24 * 60 * 60 * 1000);

    return { old: oldKey, new: newKey };
  }

  async validateApiKey(
    keyId: string,
    secret: string
  ): Promise<{ userId: string; valid: boolean }> {
    const key = await db.apiKeys.findById(keyId);

    if (!key) {
      return { userId: '', valid: false };
    }

    // Accept both active and pending (grace period)
    if (!['active', 'pending'].includes(key.status)) {
      return { userId: key.userId, valid: false };
    }

    const isValid = await this.verifySecret(secret, key.secret);
    return { userId: key.userId, valid: isValid };
  }

  private generateKey(userId: string): ApiKey {
    const secret = require('crypto').randomBytes(32).toString('hex');
    return {
      keyId: `key_${Math.random().toString(36).substr(2, 9)}`,
      secret,
      status: 'active',
      createdAt: new Date(),
      rotatedAt: new Date(),
    };
  }

  private async verifySecret(provided: string, stored: string): Promise<boolean> {
    return require('crypto').timingSafeEqual(
      Buffer.from(provided),
      Buffer.from(stored)
    );
  }
}

// Endpoint to rotate user's API key
app.post(
  '/api/keys/rotate',
  authenticate,
  async (req: express.Request, res: express.Response) => {
    const manager = new ApiKeyRotationManager();
    const result = await manager.rotateApiKey(req.user!.id);

    res.json({
      message: 'Key rotated successfully',
      newKey: result.new.keyId,
      oldKeyDeprecationIn: '24 hours',
    });
  }
);

// Middleware to validate API keys with overlap
const validateApiKeyMiddleware = async (
  req: express.Request,
  res: express.Response,
  next: express.NextFunction
): Promise<void> => {
  const authHeader = req.headers.authorization;
  if (!authHeader?.startsWith('Bearer ')) {
    res.status(401).json({ error: 'Missing API key' });
    return;
  }

  const [keyId, secret] = authHeader.substring(7).split(':');
  const manager = new ApiKeyRotationManager();
  const validation = await manager.validateApiKey(keyId, secret);

  if (!validation.valid) {
    res.status(403).json({ error: 'Invalid API key' });
    return;
  }

  req.user = { id: validation.userId };
  next();
};

app.get('/api/protected', validateApiKeyMiddleware, (req, res) => {
  res.json({ success: true });
});

Audit Trail and Alerting

Track all secret access and rotation:

interface AuditLog {
  timestamp: Date;
  action: 'access' | 'rotate' | 'create' | 'delete';
  secretId: string;
  userId: string;
  status: 'success' | 'failure';
  reason?: string;
}

class SecretsAuditLogger {
  async logAccess(
    secretId: string,
    userId: string,
    status: 'success' | 'failure'
  ): Promise<void> {
    const log: AuditLog = {
      timestamp: new Date(),
      action: 'access',
      secretId,
      userId,
      status,
    };

    await db.auditLogs.insert(log);

    // Alert on repeated failures
    const recentFailures = await db.auditLogs.countFailures(secretId, 5);
    if (recentFailures > 3) {
      await this.sendAlert(
        `Repeated secret access failures for ${secretId}`,
        'CRITICAL'
      );
    }
  }

  private async sendAlert(message: string, severity: string): Promise<void> {
    // Send to Datadog, CloudWatch, or PagerDuty
    console.error(`[${severity}] ${message}`);
  }
}

Checklist

  • Enable AWS Secrets Manager automatic rotation with Lambda
  • Implement PostgreSQL password rotation without application restart
  • Use connection pool lifecycle hooks to detect secret changes
  • Maintain blue/green secret versions for instant rollback
  • Implement API key rotation with 24-hour overlap period
  • Test rotation in staging before production
  • Monitor all secret access with audit logging
  • Alert on rotation failures and repeated auth errors
  • Document rollback procedures for each secret type
  • Verify zero-downtime rotation with production traffic

Conclusion

Automated secrets rotation removes the most painful aspect of secrets management—manual rotation. With AWS Secrets Manager, blue/green versioning, and connection pool awareness, your production systems rotate credentials without downtime, detect theft, and roll back instantly when needed. This is the foundation of secrets hygiene at scale.