Skip to main content
Deno 2 is finally here 🎉️
Learn more

August 22nd, 2023 Deno Deploy Post-mortem

On August 22nd, 2023, beginning at 07:00 UTC and ending at 08:30 UTC, the website serving deno.com and deno.land experienced a distributed denial of service (DDoS) attack. This attack not only impacted deno.com and Deno Deploy dashboard users, but also Deno runtime users importing packages through HTTPS URLs on deno.land/x or deno.land/std.

We sincerely apologize for any disruption casued by this attack on our infrastructure. We have taken steps to protect against this style of attack in the future, and remain firm in our commitment to providing reliable service for Deno Deploy and any software that depends on our infrastructure for dependency management.

Impact

During a 90-minute period, the Deno Deploy admin console and the deno.com and deno.land websites were unreachable. This also impacted our third-party module registry on deno.land/x and the standard library at deno.land/std. Lack of access to the registry may have caused build time errors for some Deno runtime users.

Deno applications where dependencies were already cached or vendored were not impacted. Other applications aside from deno.com already deployed and running on Deno Deploy were not impacted.

Incident timeline

All times in UTC, on August 22nd, 2023.

  • 07:00 - DDoS attack begins
  • 07:02 - Team members alerted to elevated errors on deno.com
  • 07:10 - Team members discover a DDoS attack underway and begin investigation
  • 07:41 - Additional team members paged to assist mitigation efforts
  • 08:21 - Attack mitigated as offending IP addresses are blocked
  • 08:29 - Team confirms system recovery
  • 08:59 - Incident is formally resolved

We estimate a downtime of approximately 90 minutes from when our systems first started failing until full recovery was achieved.

Root cause

A DDoS attack was mounted against deno.com, which is hosted on Deno Deploy. The very large volume of requests exceeded the ability of Deno Deploy to scale up and allocate more resources to this application, causing server errors that made web pages and modules hosted on deno.com and deno.land unavailable.

Thankfully, resource isolation between “deployments” (versions of apps hosted on Deno Deploy across projects and customers) was effective in shielding other applications on Deno Deploy from significant impact.

What’s next?

We realize how disruptive this service interruption was to many Deno users, both on Deploy and using the Deno runtime. We are taking swift action to ensure that this type of incident is not possible in the future.

  • We are improving our ability to allocate additional resources to Deno Deploy applications that are experiencing an abnormally large spike in traffic.
  • We are improving our on-call process, so that incidents are escalated more quickly and appropriately.
  • We have also identified our need for more effective communication during incidents, and will ensure that outage reports are more promptly added to denostatus.com.
  • More broadly, we are looking at changing how we host Deno module code to prevent incidents on deno.com and Deno Deploy impacting dependency management for other Deno programs.

Once again, we deeply regret the impact this incident may have had on your work. Thanks to your support and collaboration, we will continue to evolve and harden Deno and Deno Deploy in the months and years to come.

If you have additional questions or concerns, please start a conversation with us here.