Summary

On Monday, June 15, from approximately 4:00 AM PDT to 10:26 AM PDT, Safari and iOS users encountered an error with embedded and Ashby-hosted job boards (jobs.ashbyhq.com) and could not view or apply to jobs. 

We became aware of the problem at 9:43 AM PDT and recovered at 10:26 AM PDT.

Why did this happen?

Ashby's internal code build process processes the code that powers your hosted and embedded job boards. As part of our security practices, each file is published with verification data that browsers use to confirm the file they download is exactly what we intend to serve. We shipped a change to this build process that modified the files after their verification data had been generated, so the files no longer matched the verification data and failed to load in Safari and iOS mobile browsers. The same files did not fail in the same way in other browsers, such as Chrome and Firefox, which is why the failure appeared inconsistent amongst visitors.

How did we resolve this situation?

After confirming the failure was tied to a recent deploy, we rolled back to a known-good state to restore service. We confirmed that job boards loaded correctly in Safari, on mobile devices, and in other browsers before resuming normal deployments.

How did we get to this point?

The change altered our shipped code in a way we did not anticipate, and two gaps let it reach production undetected: 

1. Our automated job board tests ran only against Chrome.
2. We didn’t have a check verifying file integrity at the point of publishing, where this particular failure surfaced.

What have we put in place to prevent this from happening in the future?

1. We added an automated check that verifies file integrity immediately before publishing; any mismatch now blocks the deployment of updates to Ashby. We confirmed it catches this exact failure.
2. We corrected how integrity verification is applied to our job board code, so that all browsers enforce it consistently.
3. We expanded our automated job board tests to run against Safari and Firefox, not just Chrome.
4. We have added additional monitoring, including on mobile browsers, to alert us directly when a job board fails to load in this way.