Issues with emails being blocked
Incident Report for Greenhouse
Postmortem

The Greenhouse Recruiting platform suffered an email delivery issue that began early on March 15 and was resolved on March 18, 2019. This issue affected emails sent to Microsoft email addresses (for example, hotmail.com and outlook.com).

Microsoft’s email service provider flagged one of the Greenhouse Recruiting IP addresses as a potential spammer and blacklisted this IP address. Greenhouse Recruiting has a pool of IP addresses through which it can send emails. As a result, a portion of emails sent from Greenhouse Recruiting to a Microsoft mailbox failed to deliver. When emails fail to be delivered, Greenhouse users are notified via an automated email receipt.

Greenhouse engineers diagnosed the problem late Friday afternoon and began to mitigate the issue. Additional IP addresses were added to the Greenhouse Recruiting email pool. These new IP addresses were placed in a "warm-up mode" to prevent them from being marked as spam senders immediately.

After consulting with our mail provider, Mailgun, we decided not to remove the blacklisted IP address from the pool on Friday. We had reason to believe that the block on the IP address by Microsoft was temporary and would be removed without any action after 24 hours. However, the block did not appear to be removed by Monday morning, and some emails from Greenhouse Recruiting were still failing to be delivered. Greenhouse engineers responded by moving more traffic onto the newly-added IP addresses. After further consultation with Mailgun, we decided to move forward with removing the blacklisted IP address from the pool, which resolved the issue.

On March 19, 2019, Mailgun confirmed that Microsoft mitigated the IP address that caused the issue and removed the IP address from their block list. We are continuing to investigate why the IP was blacklisted, and Greenhouse engineers will be working with Mailgun to ensure that we are following best practices for email deliverability to help prevent issues like this moving forward. We’ll also create an action plan for how to handle this type of issue should Microsoft block another of our IP addresses in the future.

Posted 2 months ago. Mar 19, 2019 - 16:26 EDT

Resolved
This incident has been resolved.
Posted 2 months ago. Mar 18, 2019 - 14:59 EDT
Monitoring
This issue looks to have been resolved, but we will continue to monitor the situation.
Posted 2 months ago. Mar 18, 2019 - 14:44 EDT
Update
We have narrowed the problem down to a single IP address which is being blocked by Microsoft. This is resulting in a very small portion (< 1%) of traffic failing to deliver emails. We are working with our mail provider and Microsoft to resolve this issue. We will update you shortly.
Posted 2 months ago. Mar 18, 2019 - 12:48 EDT
Update
We are continuing to work on a fix for this issue.
Posted 2 months ago. Mar 18, 2019 - 10:50 EDT
Update
We are continuing to work on a fix for this issue.
Posted 2 months ago. Mar 18, 2019 - 10:43 EDT
Update
We are continuing to work on a fix for this issue.
Posted 2 months ago. Mar 18, 2019 - 10:42 EDT
Identified
We are currently experiencing issues resulting in emails being blocked for a large percentage of our customers. We are working on the issue and will provide an update shortly.
Posted 2 months ago. Mar 18, 2019 - 10:40 EDT
This incident affected: Greenhouse Products (Greenhouse Recruiting) and Third-Party Dependencies (Mailgun — Inbound, Mailgun — Outbound).