Primary DB exhausted connections
resolved
Back office website
Websales
Calendar websales
Accounts API
Inventory API
Trip Search API
Loyalty API
Notifications API
Operations API
Payments API
Reports API
Reports worker
Sales API
Webhooks worker
major-outage
The db cluster primary server exhausted all possible connections and services started to fail.
Jun 6, 2019 7:29 PM
Investigating
We notice several services failling to stablish new connections to the db or started to fail on existing connections.
Jun 6, 2019 7:39 PM
Identified
We decided to stop all connections to the cluster. Restarted primary but it was elected as a secondary. Cluster is lacking a primary server. Forcing an election manually while replacing some of the old servers with new ones.
Jun 6, 2019 7:45 PM
Identified
New primary elected. Configuring internal DNS and restarting services to connect to the new configuration. Verifying data integrity.
Jun 6, 2019 7:55 PM
Monitoring
Services are connecting normally.
Jun 6, 2019 8:05 PM
Identified
Some services are losing connections again. Cluster started a new election. Identified a problem in the new configuration. Fixing the problem.
Jun 6, 2019 8:14 PM
Resolved
All services are up and operational. Data integrity verified.