July 18th to July 25th 2018 – Various issues with 5% of Hired Candidate related system tasks
Incident Report for Greenhouse
Resolved
During the period between July 18th and July 25th 2018, a bug in a background task caused a number of issues related to actions that are normally performed by the system when a candidate is hired. These issues occurred to approximately 5% of candidates hired during this time. The affected system tasks were:

• Hired Candidate web hooks
• Exports to Greenhouse Onboarding
• Exports to HRIS integrations using the above Web Hooks
• New hire summary emails
• Agency status change emails

We were able to successfully re-run the affected background jobs which generated the correct web hooks and emails.

The cause of this issue was a bug related to a race-condition that became apparent when we moved some of the post-hire processing to a background job which was executed asynchronously. This race-condition was the reason that only a small number of hired candidates were affected, and why it took a while to understand the root cause. The issue has been resolved, and the engineering team has made changes to the code to prevent this kind of bug from occurring again.
Posted about 2 months ago. Jul 25, 2018 - 12:00 EDT