Everything is running stable again for the past ~2.5 hrs already since we isolated the faulty servers from the cluster. But we're continuing to monitor the situation closely.
Posted Apr 25, 2019 - 23:01 EEST
Identified
We suspect the fault of a rack's cooling system, which caused some of the servers to randomly freeze. We have isolated and removed those servers we think are impacted from the database cluster.
Posted Apr 25, 2019 - 21:06 EEST
Update
API is fully available again. We are still investigating into the root cause...
Posted Apr 25, 2019 - 19:58 EEST
Investigating
Our database cluster switched into read-only mode, which resulted in unavailability of most endpoints of our REST API. We are investigating...
Posted Apr 25, 2019 - 19:49 EEST
This incident affected: Transcoder Service, CDN Service, Website/Dashboard, and REST API.