Part of owning your mistakes is making it better. As Dharmesh said at INBOUND this week, own your mistakes, say sorry, be sorry, and make it better.
We’re sorry.
Own your mistakes. That’s a core tenet of the Customer Code that HubSpot announced on stage at INBOUND this week. We just didn’t know we’d have to put it to use so quickly. On the morning of September 6, an internal issue caused some Marketing Hub enterprise customers to lose features and caused their websites to fail. Any outage is bad, but an outage that affects your website — your company’s digital front door to the world — and key tools like workflows is worse. For those affected, we are so sorry our mistake disrupted your business.
What Happened
At 04:38 EDT our team identified the issue, and at 5:45 EDT we identified the cause. The cause was a bug in the product configuration code that had ripple effects across the product. The most straightforward way to say it is that the bug caused affected HubSpot portals to temporarily forget they were enterprise and lose access to those features.
At 7:06 EDT we fixed the bug, and resolved the underlying issue. We are currently working to resolve issues that were a direct result of this outage. Here's a link to our status page.
This was a mistake and it was ours alone. We know what it feels like to be waiting for a resolution to a problem, and we’re sorry we put you in that situation. Thank you for hanging with us as we worked to fix the error and bring features and websites back up. We are lucky to have you as HubSpot customers and on this morning, we didn’t live up to our end of the relationship.
Not all enterprise customers were impacted; we are working with those affected on an as-needed basis. If you are experiencing additional issues related to this outage, please call Support at 1-888-HUBSPOT x 3. Please note that you may experience delays in response time as we work through communication with all impacted customers.
Next Steps
To help make it better, we are — of course — doing a full analysis on this issue, the root cause, the impact and how to prevent this from happening again. In addition, we plan to be ultra-transparent and share that detail with all of you. In the coming days, we plan to publicly post all of our notes from that analysis — what happened, why, how, and what we are doing to make it better in the future. Thank you for hanging with us as we dig in and get back to helping you grow better.
