2021-03-03 Greenhouse Recruiting Delays Processing Applications Post-Mortem

WHAT HAPPENED?

Some applications submitted between 14:47 UTC and 22:36 UTC on March 3, 2021 were delayed in arriving in Greenhouse Recruiting due to an update that was released. The applications were all reprocessed by 17:00 UTC on March 4, 2021.

TIMELINE (UTC)

March 3, 2021

14:47 - An update is released that unintentionally begins the incident.

17:47 - The incident is identified by the Engineering team.

20:32 - A fix is deployed to all Greenhouse Recruiting instances.

March 4, 2021

17:00 - All applications are reprocessed and tagged in impacted Greenhouse Recruiting accounts.

WHAT WAS THE EFFECT?

Some applications were delayed in arriving in Greenhouse Recruiting between 14:47 UTC on March 3, 2021 and 17:00 UTC on March 4, 2021.

WHO WAS AFFECTED?

Some Greenhouse Recruiting customers. 

All impacted customers were notified via email and impacted applications were tagged with “march2021_reprocessed_attachment” in Greenhouse Recruiting.

WHAT WAS THE CAUSE?

We released an update to application attachments that interacted with auto-merge for new applications in an unexpected way.

WHAT ARE WE DOING TO PREVENT THIS FROM OCCURRING AGAIN?

• We are revising our internal alert systems to ensure that if applications fail to process, we will know sooner
• We are adding more auto-merge workflows to our automated tests

‌

If you have any questions or concerns, please reach out via our Support center.