WHAT HAPPENED?

Beginning at 6:00 AM UTC (2:00 AM EDT) on October 9, 2020, page load times in Greenhouse Recruiting increased for some customers. At 10:00 AM UTC (6:00 AM EDT), Greenhouse Recruiting began serving errors at an increased rate for some customers.

Beginning at 12:57 PM UTC (8:57 AM EDT), Greenhouse Onboarding began serving errors stemming from E-Signatures in the Welcome Experience.

By 1:00 PM UTC (9:00 AM EDT), error rates in Greenhouse Recruiting had decreased to a normal rate and by 4:15 PM UTC (12:15 PM EDT), delays in a small number of Greenhouse Recruiting accounts decreased to a normal rate.

At 7:56 PM UTC (3:56 PM EDT), after further monitoring, the incident was marked as resolved.

WHAT WAS THE EFFECT?

Some users received error pages or experienced slow page loads while accessing Greenhouse Recruiting.

Users received error pages while accessing E-Signatures in Greenhouse Onboarding.

WHO WAS AFFECTED?

A portion of Greenhouse Recruiting customers.

Greenhouse Onboarding customers accessing E-Signatures.

WHAT WAS THE CAUSE?

This incident was caused by an issue affecting network connectivity through our Greenhouse Recruiting infrastructure provider (AWS). The AWS issue caused a problem for our E-Signature provider for Greenhouse Onboarding (HelloSign).

WHAT ARE WE DOING TO PREVENT THIS FROM OCCURRING AGAIN?

We have identified additional monitoring and alerting over several parts of our systems, which could have helped us to identify the incident sooner and narrow down the cause more easily.

We have also identified additional documentation and processes, which will allow us to mitigate this type of incident faster.

We apologize for the inconvenience this incident has caused. We take the reliability of our application seriously and are actively working to prevent similar incidents like this one from occurring in the future. If you have any questions or concerns, please reach out via: https://support.greenhouse.io/hc/en-us/requests/new.