WHAT HAPPENED?

Approximately 1,400 unique applications submitted through the Greenhouse Job Board API between 11:00AM EDT on Monday, October 9th and 12:00PM EDT on Tuesday, October 10th received 401 errors and were not processed by Greenhouse Recruiting. By 2:00PM EDT on Wednesday, October 11th, we were able to manually reprocess the applications that encountered this error, and nearly all of the candidates appeared normally in Greenhouse Recruiting.

WHAT WAS THE EFFECT?

Candidates applying to a Job Board that sends applications to Greenhouse using the Job Board API may have received an error message when they tried to submit their applications, depending upon how the Job Board was configured. We recommend looping in the team that configured your external Job Board to confirm whether an error was present or not. If the "Thank you for Applying" email was configured on the Job Post candidates used to apply, they received those messages when their applications were reprocessed on Wednesday, October 11th.

When the candidate profiles were added back to Greenhouse Recruiting, attachments uploaded as resumes, cover letters, or custom questions were not able to be recreated. Only candidates who pasted resume or cover letter text would appear with these attachments intact. Additionally, approximately 30 applications could not be reprocessed at all. We will be reaching out to all affected customers shortly to confirm which boards were impacted by this outage, and can pass along candidate information for the small number of unprocessable candidates.

WHO WAS AFFECTED?

This disruption affected approximately 150 career sites that submit new candidates to Greenhouse Recruiting via the Job Board API. Any Job Board that uses the Greenhouse-hosted application form or embedded iframe application form was not affected by this incident.

WHAT WAS THE CAUSE?

On Monday, we released a change to an internal tool that ensures Greenhouse Job Boards are in-sync with Greenhouse Recruiting. A database update meant to run during this release encountered an error and failed to finish, leaving a number of Job Boards unable to process new applications. Because Greenhouse Job Boards regularly experience a small but consistent number of errors because of incorrect permissions, user error, or a number of other causes, our team did not immediately realize the extent of the issues that were occurring. The Greenhouse engineering team prepared a fix late Monday night, but chose to wait until Tuesday morning to deploy it so it could be better tested and deployed more safely.

WHAT ARE WE DOING TO PREVENT THIS FROM OCCURRING AGAIN?

We have prioritized a change that will make our Job Board Infrastructure more resilient to a failed database update like this one in the future. Additionally, we will be modifying our testing and deployment processes to mitigate the chance of a failure in the first place. We will also investigate improvements to our monitoring tools to help prevent a similar occurrence.

We take the reliability of our software very seriously, and are committed to making changes to prevent similar issues from occurring again. Please accept our apologies for any inconvenience caused. If you have any questions, please reach out to your Account Manager or support@greenhouse.io.