Outage of internal storage impacting Transcoder and CDN services

Incident Report for Xvid MediaHub

Resolved

This incident has been resolved.
Posted Jan 11, 2023 - 07:30 EET

Monitoring

The incident at our upstream provider has been resolved. However, there is still a backlog of transcoder jobs which currently still causes longer than usual processing times. We keep monitoring the situation.
Posted Jan 11, 2023 - 03:17 EET

Update

Our upstream provider has acknowledged the issue and is working on a solution but unfortunately the incident is still ongoing. We are still seeing high latencies and frequent timeouts accessing the storages which is causing a degradation of the CDN service and an elevated rate of failing transcoder jobs.
Posted Jan 11, 2023 - 02:16 EET

Identified

The incident at our upstream provider is still ongoing and our master storages remain inaccessible in all regions. We have partially restored CDN service but performance is still degraded, unfortunately. The transcoder service is impacted as well due to the inaccessibility of the storages which results in an elevated rate of job failures. We continue to work on mitigating the issue.
Posted Jan 11, 2023 - 00:59 EET

Update

We are continuing to investigate this issue.
Posted Jan 11, 2023 - 00:20 EET

Investigating

We are experiencing a multi-region outage of our master storage with our upstream cloud provider. This is impacting the CDN service. We are investigating the issue and are trying to mitigate it.
Posted Jan 11, 2023 - 00:05 EET
This incident affected: Transcoder Service and CDN Service.