Increased Errors in Greenhouse Recruiting and E-Signature Errors in Greenhouse Onboarding
Incident Report for Greenhouse
Postmortem

WHAT HAPPENED?

Beginning at 6:00 AM UTC (2:00 AM EDT) on October 9, 2020, page load times in Greenhouse Recruiting increased for some customers. At 10:00 AM UTC (6:00 AM EDT), Greenhouse Recruiting began serving errors at an increased rate for some customers.

Beginning at 12:57 PM UTC (8:57 AM EDT), Greenhouse Onboarding began serving errors stemming from E-Signatures in the Welcome Experience.

By 1:00 PM UTC (9:00 AM EDT), error rates in Greenhouse Recruiting had decreased to a normal rate and by 4:15 PM UTC (12:15 PM EDT), delays in a small number of Greenhouse Recruiting accounts decreased to a normal rate.

At 7:56 PM UTC (3:56 PM EDT), after further monitoring, the incident was marked as resolved.

WHAT WAS THE EFFECT?

Some users received error pages or experienced slow page loads while accessing Greenhouse Recruiting.

Users received error pages while accessing E-Signatures in Greenhouse Onboarding.

WHO WAS AFFECTED?

A portion of Greenhouse Recruiting customers.

Greenhouse Onboarding customers accessing E-Signatures.

WHAT WAS THE CAUSE?

This incident was caused by an issue affecting network connectivity through our Greenhouse Recruiting infrastructure provider (AWS). The AWS issue caused a problem for our E-Signature provider for Greenhouse Onboarding (HelloSign).

WHAT ARE WE DOING TO PREVENT THIS FROM OCCURRING AGAIN?

We have identified additional monitoring and alerting over several parts of our systems, which could have helped us to identify the incident sooner and narrow down the cause more easily.

We have also identified additional documentation and processes, which will allow us to mitigate this type of incident faster.

We apologize for the inconvenience this incident has caused. We take the reliability of our application seriously and are actively working to prevent similar incidents like this one from occurring in the future. If you have any questions or concerns, please reach out via: https://support.greenhouse.io/hc/en-us/requests/new.

Posted Oct 13, 2020 - 16:33 EDT

Resolved
This incident has been resolved
Posted Oct 09, 2020 - 15:56 EDT
Update
We are no longer seeing elevated errors from E-Signatures in Greenhouse Onboarding and are continuing to monitor the situation.
Posted Oct 09, 2020 - 15:17 EDT
Update
We are no longer seeing latency issues in Greenhouse Recruiting and are continuing to monitor the situation. As well, we are continuing to monitor elevated errors affecting E-signatures in Greenhouse Onboarding.
Posted Oct 09, 2020 - 12:47 EDT
Update
We are seeing elevated errors affecting E-signatures in Greenhouse Onboarding. We are continuing to monitor the situation.
Posted Oct 09, 2020 - 12:08 EDT
Update
We are seeing increased latency in Greenhouse Recruiting and Greenhouse Onboarding and are continuing to monitor the situation.
Posted Oct 09, 2020 - 11:51 EDT
Update
Error rates and latency have returned to normal levels. We are continuing to monitor for any further issues.
Posted Oct 09, 2020 - 11:16 EDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Oct 09, 2020 - 10:31 EDT
Identified
The issue has been identified and a fix is being implemented.
Posted Oct 09, 2020 - 10:21 EDT
Investigating
We have identified an issue causing some users to experience errors or slow load times in Greenhouse Recruiting. We are monitoring the situation and identifying a fix.
Posted Oct 09, 2020 - 10:05 EDT
This incident affected: Greenhouse Products (Greenhouse Recruiting, Greenhouse Onboarding).