The network disruption at our main cloud provider appears to have been solved meanwhile. So everything appears to be working normally again - however, we unfortunately still have a very large backlog of tasks to process that accumulated. Therefore processing speed is still not back to normal.
Posted Mar 30, 2020 - 20:12 EEST
Identified
Our upstream cloud provider suffered a complete network failure. This did not only lead to a failure of their API preventing us to start new instances and scale the transcoder service but also already running instances lost connection to our storage, which prevented them from performing any further work. We have meanwhile been able to obtain additional instances on our secondary cloud provider and overload is decreasing now.
Posted Mar 30, 2020 - 19:30 EEST
Investigating
We are currently unable to scale up our transcoder service and obtain additional CPU resources due to a failure of our upstream cloud provider. Because of this, speed of the service is severely impacted. We are working on the issue and are trying to obtain more capacity.