Resolved -
This incident is resolved. All systems are operating normally.
Oct 21, 00:51 UTC
Update -
AWS has resolved their upstream incident. Our systems have caught up on their backlogs of pending tasks. We are continuing to monitor our systems to ensure a full recovery.
Oct 21, 00:18 UTC
Update -
AWS is reporting decreased error rates across their services. We have restored our virus scan, report alert, and scheduled dashboard delivery systems. AI Notetaker bots should now be joining interviews as expected.
We are continuing to monitor our system's health as it works through the backlogs of queued tasks for those components.
Oct 20, 22:07 UTC
Update -
We are seeing reduced errors in our Direct Booking system. Reports and Dashboards should no longer display stale data.
We are still experiencing delays when scanning incoming files for viruses, and when sending report alerts and scheduled dashboard deliveries. We will continue to monitor AWS's recovery and will provide further updates as we have them.
Oct 20, 20:00 UTC
Update -
We are seeing issues with Direct Booking links, where candidates may see an error when trying to schedule an interview. We are also seeing delays when scanning incoming files for viruses. Reports and Dashboards may also display stale data.
We will continue to monitor AWS status, and will provide further updates as soon as we have them.
Oct 20, 15:44 UTC
Update -
We are aware of an issue with the AI Notetaker, where the Meeting Bot fails to show up to the interview.
We have determined this to be caused by the AWS incident, and will continue to monitor for recovery.
Oct 20, 14:56 UTC
Update -
All of our systems have fully recovered and are running at nominal levels. We have secured enough capacity to ensure we can run our services as normal throughout the day.
AWS has shared a recent update where they're seeing increased error rates in some of their services, which are not currently affecting Ashby. We will continue to monitor further announcements from AWS and will provide an update once we have confirmation of full recovery.
We continue to work on replaying actions that failed during the incident.
Oct 20, 14:49 UTC
Update -
We have processed the backlog of queued actions and are currently working on replaying actions that failed during the incident
Oct 20, 12:00 UTC
Monitoring -
All services have been restored. We are currently processing the backlog of queued tasks and will provide an update once this is complete.
Oct 20, 10:16 UTC
Update -
We are gradually restoring services. Multiple components are being brought back online in a controlled manner. Performance may be intermittent as we complete the recovery process.
Oct 20, 09:54 UTC
Update -
We are currently experiencing widespread availability issues. Multiple components of Ashby are unavailable at this time. We have implemented partial service restrictions to minimize impact and are actively working to restore full functionality.
Oct 20, 08:36 UTC
Identified -
Our upstream service provider AWS acknowledged a widespread incident affecting multiple services. We'll continue to monitor and provide updates as we learn more.
Oct 20, 07:25 UTC
Investigating -
We are currently investigating a spike in error rates impacting various parts of the Ashby platform. The issue appears to be widespread and may be related to upstream service provider. Some users may experience difficulty logging in or intermittent errors. We will provide updates as we learn more.
Oct 20, 07:06 UTC