During the period between July 18th and July 25th 2018, a bug in a background task caused a number of issues related to actions that are normally performed by the system when a candidate is hired. These issues occurred to approximately 5% of candidates hired during this time. The affected system tasks were:
• Hired Candidate web hooks
• Exports to Greenhouse Onboarding
• Exports to HRIS integrations using the above Web Hooks
• New hire summary emails
• Agency status change emails
We were able to successfully re-run the affected background jobs which generated the correct web hooks and emails.
The cause of this issue was a bug related to a race-condition that became apparent when we moved some of the post-hire processing to a background job which was executed asynchronously. This race-condition was the reason that only a small number of hired candidates were affected, and why it took a while to understand the root cause. The issue has been resolved, and the engineering team has made changes to the code to prevent this kind of bug from occurring again.