Available Procedures
Merge Accounts
Merge multiple merchant accounts when orders are split across accounts
Deploy Infra Changes
Sync infrastructure changes via Render blueprint
Trigger Webhooks
Manually trigger missed or failed webhooks
Run One-Off Jobs
Trigger backend scripts in production via Render One-Off Jobs
Kill Blocking DB Queries
Detect and terminate queries blocking migrations or other operations
Rotate Database Credentials
Rotate Postgres default user/password with zero downtime
Rotate the JWKS Signing Key
Publish a new signing key generation and retire the old one
Rotate the Token Hashing Secret
Add a hashing secret, let credentials migrate to it, retire the old one
Respond to a KMS Key Compromise
Revoke the principal, read what was decrypted, reissue those secrets
Vendor Security Monitoring
Triage security advisories and status incidents from our subservice organizations
Safety Guidelines
- Double-check account IDs, transaction IDs, and other identifiers before executing queries
- Test queries on staging environment when possible
- Document all manual operations in the on-call log
- Notify the team after performing manual operations
Where can I find the schedule?
On BetterStack: https://uptime.betterstack.com/team/t118853/oncalls/157936. This is also where you edit/update the schedule. You can find people on the rota and their phone numbers here: https://betterstack.com/settings/teams/118853/team-membersWhat triggers an on-call event?
Only BetterStack itself makes the phone call as it knows who and what to call. BetterStack will generally make a normal phone call, but can also send a iOS/Android notification to the BetterStack app on your phone. BetterStack reacts to these sources of data:- BetterStack itself (uptime monitoring of the API and checking that the “Pay” button is visible in Checkout)
- Sentry alerts
- Repeated errors “anomaly” – not the more common ones that are sent to Slack
What is the schedule?
- 1 developer is on-call for 1 full week
- The week goes from Monday 9:00 CET to the following Monday 9:00 CET
- We cycle between the devs every week
- People are free to swap between each other, but that should be reflected in the BetterStack on-call schedule.
- All hours of the day and night, both weekdays and weekend days during that week. It doesn’t mean that you can’t participate in family events (or not sleeping 😅). But it means having BetterStack’s phone numbers added as VIPs to your phone so that those calls go through and having your laptop with you plus the ability to connect to the internet so that you can investigate and respond to the incident. This sounds scary, but we generally don’t expect to have a lot of downtime.
How and when to escalate?
Every incident gets a severity level. Set it in the incident channel with@incident sev 1, sev 2 or sev 3.
When?
- SEV1 – Critical: all of Polar is down, checkout is down or badly degraded, money moved incorrectly, merchant data was lost, or there’s a suspected security incident. Always escalate and tell leadership right away – don’t try to be the hero.
- SEV2 – Major: an important feature is broken or degraded for some merchants, for example webhooks, payouts or the dashboard. If you’re unable to resolve it, escalate. Tell leadership if it lasts more than an hour.
- SEV3 – Minor: limited impact, like an error spike, a few merchants or internal tools only. Do try and resolve it yourself. Outside working hours, the fix can wait until the morning.
- To bring in help, page the person you need from the incident channel with
@incident betterstack page @person. The incident handbook’s “People of interest” list shows who knows what. - If paging doesn’t work, use the BetterStack team page and call the phone numbers listed there. BetterStack’s own paging is preferred, as those calls and notifications are more likely to come through.
Support
- The week you’re on call, you’re also the go-to person for support / Rishi to reach out to for deeper, more technical questions or things that can only be solved by a dev.