> ## Documentation Index
> Fetch the complete documentation index at: https://handbook.polar.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Respond to a KMS Key Compromise

> Revoke the sessions, close the path, read what was decrypted, reissue it.

<Warning>
  Production. This key unwraps every secret we store encrypted.
</Warning>

The key material never leaves KMS. What leaks is access to it: a stolen session, a worker role,
an AWS account. This procedure cuts that access and replaces what it read.

The key wraps seven columns. CloudTrail reports each by the `{table, column}` in its encryption
context, and the ciphertext sits in the matching `*_encrypted` column:

* `oauth_accounts` — `access_token`, `refresh_token`
* `oauth2_clients` — `client_secret`, `registration_access_token`
* `slack_apps` — `client_secret`, `signing_secret`, `bot_token`

The token hashing secret is not one of them: Secrets Manager encrypts it under the AWS-managed
key.

Two roles hold `kms:Decrypt`, each through an inline policy of the same name:

* `polar-production-secrets`, assumed by the Render backend through OIDC
* one per worker Lambda, named after the function

## Prerequisites

* AWS access to the environment's account, with `iam:PutRolePolicy`
* CloudTrail history for that account
* A psql session on the production database
* [Render Dashboard](https://dashboard.render.com/) to redeploy

## 1. Revoke the sessions

Set `ROLE` to the one whose credentials leaked, and only that one.

```bash theme={null}
ROLE=polar-production-secrets   # or the worker Lambda's role

cat > /tmp/revoke.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Deny",
    "Action": "*",
    "Resource": "*",
    "Condition": {
      "DateLessThan": {"aws:TokenIssueTime": "$(date -u +%Y-%m-%dT%H:%M:%SZ)"}
    }
  }
}
EOF

aws iam put-role-policy \
  --role-name "$ROLE" \
  --policy-name AWSRevokeOlderSessions \
  --policy-document file:///tmp/revoke.json
```

Sessions assumed before that instant die, later ones are untouched, so our services re-assume
through OIDC and keep serving. The console does the same from **Revoke sessions**.

## 2. Close the path

Step 1 only helps if what leaked was a session. If the thief can assume the role again — they
hold the Render environment, or the OIDC trust is being abused — they are back within seconds.
Then:

```bash theme={null}
KEY=$(aws kms describe-key \
  --key-id alias/polar-production-secrets \
  --query KeyMetadata.KeyId --output text)

aws kms disable-key --key-id "$KEY"
```

Key management takes an id, not an alias, so resolve it first.

Every `Decrypt` and `GenerateDataKey` fails, theirs and ours. Slack stops sending and revealing
an OAuth2 client secret errors. Login and registration read hashes, so they keep working.

`aws kms enable-key --key-id "$KEY"` undoes it. Then cut the trust that let them in: the OIDC
provider, the role's trust policy, or the role.

## 3. Read what was decrypted

Every `Decrypt` carries its encryption context, and ours is `{table, column, id}`. CloudTrail
names the rows that were opened, not a guess at what else is gone.

```bash theme={null}
SINCE="$(date -u -v-1d +%Y-%m-%dT%H:%M:%SZ)"  # GNU: date -u -d '1 day ago' +%Y-%m-%dT%H:%M:%SZ

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=Decrypt \
  --start-time "$SINCE" \
  --max-items 500 \
  --output json > /tmp/decrypts.json

jq -r '.Events[].CloudTrailEvent | fromjson
  | [.eventTime, .userIdentity.arn,
     .requestParameters.encryptionContext.table,
     .requestParameters.encryptionContext.column,
     .requestParameters.encryptionContext.id] | @tsv' /tmp/decrypts.json

jq -r 'if .NextToken then "TRUNCATED: more than 500 matched" else empty end' /tmp/decrypts.json
```

The cap keeps the query responsive, and the last line says when it hid something. On a
truncation, narrow `SINCE` and run again until it stays quiet.

Most of what comes back is ours; only the rows the stolen credentials opened are disclosed.
`lookup-events` reaches back 90 days, no further.

## 4. Restore service

Delete `AWSRevokeOlderSessions`, restore whatever step 2 cut, re-enable the key, then redeploy
the API and the workers.

```bash theme={null}
ROLE=polar-production-secrets   # the one you revoked in step 1
KEY=$(aws kms describe-key \
  --key-id alias/polar-production-secrets \
  --query KeyMetadata.KeyId --output text)

aws iam delete-role-policy --role-name "$ROLE" --policy-name AWSRevokeOlderSessions
aws kms enable-key --key-id "$KEY"
```

## 5. Reissue what was read

A disclosed secret stays disclosed. Re-encrypting changes nothing — replace the value.

* **Slack.** These belong to the merchant's own app, so we cannot regenerate them. The merchant
  rotates them in Slack and enters them again.
* **OAuth2 clients.** We issue a new `client_secret` and registration access token. Warn the
  integrator before the old one stops working.
* **OAuth account tokens.** Delete the pair; the user signs in with GitHub or Google again.
  Revoke at the provider too: deleting our row does not stop the copy they took.

## Rotation is not the remedy

Key rotation looks like the answer and fixes nothing. AWS is explicit:

> Key rotation has no effect on the data that the KMS key protects. It does not rotate the data
> keys that the KMS key generated or re-encrypt any data protected by the KMS key.

Old key material is kept until the key is deleted, so a ciphertext read yesterday stays readable
tomorrow. Rotation protects the next write, never the last one. Run one if an auditor asks; it
changes nothing here.

Deleting the key does change what they can read later: every ciphertext under it becomes
unreadable for good, including a dump kept for the day they get back in. It becomes unreadable
to us on the same terms, so the option opens only once **every** row has been reissued under a
new key — seven columns, every merchant.