Greenhouse Recruiting Outage
Incident Report for Greenhouse
Postmortem

WHAT HAPPENED?

At 5:00pm EST today, one of Greenhouse Recruiting's silos became unavailable during routine maintenance. The cause was immediately identified and resolved at 5:03pm EST. The scope of the outage was only the app.greenhouse.io endpoint. Other silos, our Job Boards infrastructure, and our APIs remained unaffected.

WHAT WAS THE EFFECT?

All users trying to access Greenhouse Recruiting at https://app.greenhouse.io could not access the site.

WHO WAS AFFECTED?

This disruption affected all customers trying to access Greenhouse Recruiting at https://app.greenhouse.io. Customers using SSO, customers on other silos, candidates applying via job boards, and our APIs were unaffected.

WHAT WAS THE CAUSE?

During maintenance, we removed an instance of infrastructure supporting app.greenhouse.io. While it was expected that app.greenhouse.io had redundant resources in place, a rare configuration of their allocation meant that it did not in this circumstance. Normal service was restored as soon as these resources came back online on other instances.

WHAT ARE WE DOING TO PREVENT THIS FROM OCCURRING AGAIN?

We have prioritized a change that will guarantee that the resources are allocated in such a way that guarantees redundancy, ensuring that maintenance of this nature will not impact site availability.

We take the reliability of our software very seriously, and are committed to making changes to prevent similar issues from occurring again. Please accept our apologies for any inconvenience caused. If you have any questions, please reach out to your Account Manager or support@greenhouse.io.

Posted Nov 28, 2017 - 19:52 EST

Resolved
These issues have been resolved, and we will issue a post-mortem with more details tomorrow. We apologize for any inconvenience this incident has caused for you and your team.
Posted Nov 28, 2017 - 17:23 EST
Monitoring
Greenhouse Recruiting is now available for all users. During this brief outage, Job Boards and the Job Board API were unaffected. We will continue monitoring for additional issues.
Posted Nov 28, 2017 - 17:05 EST
Identified
Greenhouse Recruiting is currently unavailable for all users. We have identified the fix and are working to bring the site back up now.
Posted Nov 28, 2017 - 17:02 EST