Our increased capacity has continued to prevent new jobs from queueing. This incident has now been resolved.
Posted Apr 03, 2023 - 19:27 UTC
Monitoring
All queued jobs have now started processing. We expect our increased capacity to prevent further backups but we will continue to monitor the situation.
We are no longer seeing any user impact from this incident.
Posted Apr 03, 2023 - 16:13 UTC
Update
Our new capacity has come online. We are continuing to work through a backlog of queued jobs.
Posted Apr 03, 2023 - 15:49 UTC
Update
We've secured additional capacity that we expect to become available at 15:30 UTC (11:30 AM EDT).
So far we've seen that AI Training jobs are the most backed up. We expect all pending AI Training jobs to start running shortly after 15:30 UTC.
Posted Apr 03, 2023 - 14:07 UTC
Identified
We're experiencing limited capacity issues due to increased demand. Error rates are unaffected but some magic tool jobs submitted will take longer to start running.
AI training and Gen-1 video generation are both affected. Expect longer than normal wait times over the next several hours as we work to secure additional capacity.