Greenhouse Job Application Delays
Incident Report for Greenhouse

WHAT HAPPENED?

Around 50% of the applications submitted through Greenhouse Job Boards between 4:30PM EDT on Wednesday, June 7th and 11AM EDT on Thursday, June 8th were not immediately processed by Greenhouse Recruiting. By 2PM EDT on Friday, June 9th, all application appeared normally in Greenhouse Recruiting.

WHAT WAS THE EFFECT?

No application data was lost, but candidates who submitted delayed applications received emails informing them that Greenhouse was unable to process their applications. If the "Thank you for Applying" email was configured on the Job Post candidates used to apply, they received those messages when their applications were finally processed on Friday, June 9th.

WHO WAS AFFECTED?

This disruption affected all customers using hosted or embedded job boards. Any career sites built using the Job Board API were not affected.

WHAT WAS THE CAUSE?

On Wednesday, we rolled out infrastructure changes to improve the speed and reliability of Greenhouse Job Boards. Unfortunately, due to a testing configuration oversight, we did not correctly validate the entire application flow. As a result, the Greenhouse Recruiting application was unable to process any of the files created within the new job board infrastructure. While attempting to access these files, we uncovered a separate issue that required moving them from one AWS region to another, which took some time to remedy.

WHAT ARE WE DOING TO PREVENT THIS FROM OCCURRING AGAIN?

We will be modifying our new service rollout strategy in order to mitigate the chances of testing configuration issues. We will also be improving our monitoring over application processing failures to provide us with an earlier indication that something is wrong.

We take the reliability of our software very seriously, and are committed to making changes to prevent similar issues from occurring again. Please accept our apologies for any inconvenience caused. If you have any questions, please reach out to your Account Manager or support@greenhouse.io.

Posted about 1 month ago. Jun 12, 2017 - 18:09 EDT

Resolved
Our team was able to recover and reprocess all applications that received errors during this period, and candidates are now appearing normally in Greenhouse Recruiting. If the "Thank you for Applying" email was configured on the Job Post a candidate used to apply, those messages have been successfully sent as well.

We will publish a postmortem on Monday with more information about the cause of this issue and the steps we're taking to protect against it in the future. We apologize for any inconvenience this has caused for you and your teams.
Posted about 2 months ago. Jun 09, 2017 - 13:58 EDT
Identified
Due to changes with our underlying job board infrastructure, 50% of job applications that were submitted from 4:30PM EDT on Wednesday, June 7th to 11AM EDT on Thursday, June 8th have been delayed in appearing in Greenhouse Recruiting. No data has been lost, and our team is currently working on processing the backlog of candidates.

Additionally, some of these candidates received an email from Greenhouse notifying them of the failure in processing their application. Once the applications are processed successfully, candidates will receive the "Thank you for Applying" email if it has been configured for the job post they applied to.

We apologize for any inconvenience this may have caused.
Posted about 2 months ago. Jun 08, 2017 - 12:32 EDT