SecurityBrief Australia logo
Australia's leading source of cybersecurity and cyber-attack news
Story image

Top internet outages of 2019 - ThousandEyes

By Newsdesk
Wed 15 Jan 2020
FYI, this story is more than a year old

ThousandEyes has announced last year’s most disruptive outages and what the industry can learn from them.

When Internet outages happen, they can be extremely disruptive to a business.

By preventing users from reaching applications and services, outages can cause major revenue and reputation damage.

While application delivery is dependent on many Internet Service Providers (ISPs), it also increasingly relies on a large and complex ecosystem of Internet-facing services — such CDN, DNS, DDoS mitigation, and public cloud.

These services work together to provide exceptional digital experiences to users and even brief disruptions can have a significant impact.

“At the same time, enterprises are increasingly relying on Internet transport to connect their sites and reach business critical applications and services. Gone are the days in which applications are solely hosted in private data centers and office locations are connected primarily by MPLS circuits,” says ThousandEyes product marketing director Angelique Medina.

“The Internet is replacing or supplementing services like MPLS as enterprises embrace SD-WAN technologies. As a result, the Internet is now effectively the enterprise backbone, which as a “best effort” transport can have significant yet unforeseen consequences for businesses.”

Over the last year, ThousandEyes has reported on a number of large-scale outages that had ripple effects across the global Internet, impacting enterprises and consumers alike.

The most significant of these outages took place over the northern hemisphere summer and disrupted nearly every top tech company in some fashion.

May 13, 2019 — China telecom outage reveals its global reach

While not the most disruptive outage of 2019, a global and fairly long lasting outage in China Telecom’s network proved to be a harbinger of incidents to come, while also revealing a lesson in how China Telecom’s reach extends far beyond mainland China.

For nearly five hours on May 13, 2019, China Telecom experience substantial packet loss across its backbone, primarily impacting network infrastructure in mainland China, but also affecting China Telecom’s network in Singapore and multiple points in the US, including Los Angeles. Over one hundred services were disrupted. Though not exclusively impacting western sites and services, many users of major US brands such as Apple, Amazon, Microsoft, Slack, Workday, SAP, would have experienced disruptions over the course of the outage window.

This incident illustrated some important realities about China and its impact on the global Internet that many folks aren’t aware of. Specifically, it highlighted that many of the censorship policies that apply to Chinese Internet users may actually be implemented far beyond China’s borders and in countries that have very different attitudes and policies related to Internet use.

June 2, 2019 — Summer of outages begins with Google Cloud

On June 2, 2019, Google Cloud Platform experienced a significant network outage that impacted services hosted in parts of us-west, us-east and us-central regions. This outage impacted Google’s own applications, including GSuite and YouTube. The outage lasted more than four hours, which becomes notable given the criticality of its service to business customers.

Google issued an official report of the incident several days later. ThousandEyes vantage points were able to see the outage as it unfolded in real time, revealing its characteristics and scale ahead of more detailed information becoming publicly available.

Beginning at approximately 9am ET in the US, ThousandEyes observed 100% packet loss from global monitors attempting to connect to a service hosted in GCP us-west2-a. Similar losses were seen for sites hosted in several portions of GCP US East, including us-east4-c.

The complete unavailability of parts of Google’s network, as seen by ThousandEyes, turned out to be due to Google’s network control plane inadvertently getting taken offline. Google later revealed that during the outage period, a set of automated policies determined which services were or were not reachable through the unaffected parts of its network.

One of the most important takeaways from cloud outages is that it’s vitally important to ensure your cloud architecture has sufficient resiliency measures, whether on a multi-region basis or even multi-cloud basis, to protect from future recurrence of outages. It’s reasonable to expect that IT infrastructure and services will sometimes have outages, even in the cloud.

June 6, 2019 — An unfortunate series of events takes down WhatsApp for many users

On June 6, 2019, a large number of users around the globe attempting to access the WhatsApp service experienced connectivity issues. ThousandEyes was able to immediately see that 100% packet loss was preventing the service’s reachability.

Upon further analysis, ThousandEyes determined the root cause of this packet loss was a massive route leak that steered traffic to China Telecom — a service provider that does not forward any Facebook-related traffic.

The incident was triggered when a Swiss colocation company called Safe Host announced to the Internet that the best way to reach WhatsApp and thousands of IP prefixes was through its network, AS 21217. When Safe Host advertised these routes, they were accepted by China Telecom and further propagated through other ISPs such as Cogent.

Users whose traffic was routed to Cogent and ultimately handed off to China Telecom would have been completely unable to reach the service.

It’s unclear why China Telecom would accept routes to a service that it censors, but what is clear is the lesson of this outage. BGP route leaks are not uncommon on the Internet. When you rely on the Internet, an ecosystem that is deeply interconnected and vulnerable, you must understand how it works and expect that a glitch in one service provider can have cascading effects on another.

The unfortunate reality is that business risks associated with BGP route leaks and other Internet flaws are greater given the modern enterprise and service delivery landscape.

June 24, 2019 — Cloudflare users fall victim to routing mishap

Just a couple of weeks after the massive route leak that impacted WhatsApp users, the Internet experienced yet another route-related incident - this one far more damaging.

On June 24, 2019, for nearly two hours, a significant BGP routing error impacted users trying to access services fronted by CDN provider Cloudflare, including gaming platforms Discord and Nintendo Life.

ThousandEyes analysis found that a significant BGP route leak affected a variety of prefixes from multiple providers. DQE, a transit provider, was the original source of the route leak, which was propagated through Allegheny Technologies, a customer of both DQE and Verizon. Unfortunately, Verizon further propagated the route leak, magnifying the impact.

Sites served through the CloudFlare CDN were impacted for nearly two hours.

This major Internet disruption affected about 15% of Cloudflare’s global traffic and impacted services like Discord, Facebook and Reddit. The route leak also affected access to some AWS services.

The root cause of the incident was eventually traced to DQE’s use of a BGP optimiser software that created routes to Cloudflare services that were only meant to be used within DQE’s internal network. When these routes were accidentally leaked to one of its customers, mayhem ensued.

This incident was yet another reminder of how incredibly easy it is to dramatically alter the Internet service delivery landscape. In a cloud-centric world, enterprises must have visibility into the Internet if they’re going to be successful in delivering services to their users.

July 4, 2019 — Apple Services impacted on fourth of July

On July 4, 2019, just before 9am PT, users connecting to Apple’s website and some of its services, such as Apple Pay, began experiencing significant packet loss for a period of over 90 minutes. This issue prevented many users from successfully connecting to Apple. ThousandEyes route visibility demonstrated that the packet loss was caused by a BGP route flap. A BGP route flap is caused when a routing announcement is made and withdrawn in quick succession, often repeatedly.

While Apple services are certainly important for many Internet users, the fact that the incident occurred early on a holiday seems to have prevented the incident from sparking more than a few user complaints. The lesson from this incident is that outages don’t happen in a vacuum. Sometimes even significant outages may go unnoticed (or conversely create significant uproar) simply based on their timing and context.

September 6, 2019 — DDoS attackers target the Internet’s knowledge base

On September 6, 2019, access to Wikipedia sites from around the world was disrupted for close to nine hours, the result of a massive and sustained Distributed Denial of Service (DDoS) attack. DDoS attacks can overwhelm their target’s web infrastructure and also create congestion within service provider networks that can lead to packet loss. These effects are exactly what ThousandEyes observed when Wikipedia came under attack.

During the course of the incident, ThousandEyes saw a significant drop in HTTP server availability from around the world, as well as a dramatic increase in HTTP response times. As a result, users across many regions were unable to establish an Internet connection for ongoing communication with Wikipedia servers. ThousandEyes also measured packet loss of up to 60% from our global vantage points, a condition that would have further prevented access to Wikipedia sites.

While DDoS events are an unfortunate reality of operating on the Internet, organisations should have visibility into the scope, impact and behaviour of these events and be able to validate that DDoS mitigation steps are effective.

Related stories
Top stories
Story image
Aussie businesses buckle down on digital tech investments
Australian businesses are planning major investments in digital technologies to support ambitious expansion plans.
Story image
ABI Research
75% of 5G networks to advance to 5G-Advanced by 2030
5G-advanced is set to launch in 2025, and 75% of 5G networks will upgrade by 2030, according to ABI Research. 
Story image
Examining the future of ransomware threats with Vectra’s CTO
As customers' valuable data move to the cloud, so will ransomware. What is the current landscape and what do we need to know?
Story image
Your tools, your choice: why allow employees to choose their own devices?
Jamf Australia says giving your team the freedom to work with their digital device of choice could help to attract and retain top talent in a tight labour market.
Story image
Oracle Cloud
Commvault, Oracle to deliver Metallic Data Management as a Service
"We are excited to partner with Commvault and enable our customers to restore and recover their most mission-critical cloud data."
Story image
Digital Transformation
What CISOs think about cyber security, visibility and cloud
Seeking to uncover the minds of CISOs and CIOs across Asia Pacific, my company recently asked Frost & Sullivan to take a snapshot of cloud adoption behaviour in the region.
Story image
Security driving customer identity & access management adoption
"CIAM allows businesses to embed a secure identity layer into their consumer and SaaS applications, facilitating secure, seamless end user experiences."
Story image
Palo Alto Networks' cloud security platform receives IRAP assessment
"We provide help protect all forms of compute, cloud native services and access to data within public and private sectors."
Story image
Tech and data’s role in the changing face of compliance
Accenture's study found that 93% of respondents agree or strongly agree new technologies such as AI and cloud make compliance easier.
Story image
Delinea’s Joseph Carson recognised with OnCon Icon Award
Delinea chief security scientist and advisory CISO Joseph Carson has been recognised as a Top 50 Information Security Professional in the 2022 OnCon Icon Awards.
Story image
Security Information and Event Management (SIEM)
LogRhythm updates SIEM Platform with latest innovations
LogRhythm has announced the launch of version 7.9 of the LogRhythm SIEM Platform and updates to LogRhythm NDR and LogRhythm UEBA.
Story image
Digital Fingerprint
Decline in counterfeit cherries after digital fingerprinting
Reid Fruits says there’s been a dramatic decline in counterfeit products for its cherries over the past three export seasons to Asia because of digital fingerprinting.
Story image
Remote Working
RDP attacks on the rise, Kaspersky experts offer advice
"Given that remote work is here to stay, we urge companies to seriously look into securing their remote and hybrid workforce to protect their data."
WSLHD and PwC’s Consulting Business came together to solve through the challenges of COVID-19. A model of care was developed to the NSW Health Agency for Clinical Innovation guidelines with new technology platforms and an entirely new workforce.
Link image
Story image
Jamf introduces new content filtering solution for education providers
Jamf has announced the launch of Jamf Safe Internet, a new offering that looks to deliver a safe online experience to students while offering better management options for admins.
Story image
Artificial Intelligence
Juniper study reveals top AI trends in APAC region
Juniper's research shows an increase in enterprise artificial intelligence adoption over the last 12 months is yielding tangible benefits to organisations.
Story image
Artificial Intelligence
Eight top DevSecOps trends to support IT innovation in 2022
The use of DevSecOps practices is growing, as it is increasingly seen as the best way to produce high-quality and secure code. So what are the current trends?
Story image
Document Management
NZ's FileInvite raises $10M in latest investment round
FileInvite has raised $10 million in Series A investment to fast-forward the extinction of email for requesting and collecting documents online.
Story image
Video: 10 Minute IT Jams - An update from CyberArk
Olly Stimpson joins us today to discuss the importance of MSP programmes and how MSP partners are experiencing success with CyberArk.
Story image
Cloud Security
Palo Alto Networks bolsters cloud native security offerings
Latest Prisma Cloud platform updates help organisations continuously monitor and secure web applications with maximum flexibility.
Story image
New study reveals 51% of employees using unauthorised apps
The research shows that 92% of employees and managers in large enterprises want full control over applications, but they don't have it.
PwC's Consulting Business and PwC's Indigenous Consulting are proud to play an important role in helping Australian Indigenous Mentoring Experience build IMAGI-NATION, a free online university for marginalised communities around the world.
Link image
Story image
Secureworks reveals new information on BRONZE STARLIGHT threat group
New research from Secureworks has uncovered new information on the Chinese threat group BRONZE STARLIGHT and how they are using targeted ransomware to initiate complicated attacks.
Story image
Enterprise Resource Planning / ERP
Five ways your ERP is letting you down and why it's time for a change
Wiise explains while moving to a new system may seem daunting, the truth is that legacy systems could be holding your business back.
Story image
Document Management
Regaining digital trust and enhancing digitisation in Australian Government agencies
Having a digitised ecosystem of documents, tools and data can help bolster security, improve workflow and ultimately create better services.
Story image
Businesses unprepared to defend against ransomware attacks
Ransomware attacks continue to impact organisations worldwide with high costs, but businesses are still largely unprepared.
Story image
Forescout reveals top vulnerabilities impacting OT vendors
Forescout’s Vedere Labs has disclosed OT: ICEFALL, naming 56 vulnerabilities affecting devices from 10 operational technology vendors.
Story image
Email threats spike 101%, remains a top attack vector
"Each year we see innovation in the threat landscape, but each year email remains a major threat to organisations."
Story image
HP Inc
Firmware attacks significant threat in age of hybrid work
Changing workforce dynamics are creating new challenges for IT teams around firmware security, according to new research.
Story image
Artificial Intelligence
Vectra AI named as AWS security competency partner
Threat detection and response company Vectra AI has announced that it has become an Amazon Web Services Security Competency Partner.
Story image
FIDO Alliance releases guidelines for optimising UX with FIDO Security Keys
The new guidelines aim to accelerate multi-factor authentication deployment and adoption with FIDO security keys.
Story image
Adobe study finds lack of digital trust and utilisation in Australian Government agencies
New research commissioned by Adobe has revealed a significant lack of digital trust within Australian Government departments, along with the continued underutilisation of key digital processes.
Story image
Internet of Things
ManageEngine wins big in IDC MarketScape assessment
ManageEngine's Endpoint Central service has been recognised as a leader by IDC MarketScape in several categories including Internet of Things device deployments and UEM software for SMEs.
Story image
State Library of Victoria
State Library of Victoria entrusts Oracle support and security to Rimini Street
“Our finance team are very happy with the support and security that Rimini Street provides, which keeps our assets and our customers secure."
Story image
Registrations for the W.Media Sydney Cloud and Datacenter Convention 2022 now open
Are you a C-Level executive looking to enhance your knowledge in the cloud and data center space in order to get the best results for your company?
Story image
Without trust, your security team is dead in the water
The rise of cyberattacks has increased the need for sound security that works across any type of business, but with any change, buy-in is essential. Airwallex explains why.
Story image
How New South Wales state departments achieved cloud migration success
State departments in New South Wales are heading to the cloud to achieve better workflow solutions, and one company is paving the way for their success.
Story image
How organisations can mitigate IoT and IIoT security risks
IoT and IIoT come with inherent risks because they are often deployed faster than they can be secured, putting organisations in danger of cyber threats. Here are tips on how to mitigate those risks.
Story image
Colt launches new SASE Gateway solution with Versa
Colt Technology Services’ customers now have access to an integrated full SASE solution that brings together SD WAN and SSE features.
Story image
Barracuda customises training to fit ACSC Essential Eight
Barracuda has announced that its Security Awareness Training now provides a customised training curriculum in line with the Australian Cyber Security Centre’s (ACSC) Essential Eight.
Story image
Tech job moves
Tech job moves - Bitdefender, Cohesity, Fortinet & MODIFI
We round up all job appointments from June 27-30, 2022, in one place to keep you updated with the latest from across the tech industries.
Story image
Overcoming hybrid and multi-cloud challenges to drive innovation
Driven by improvements in technology, financial services companies have advanced both internal and external systems and processes, with the likes of digitisation, personalisation and risk management redefining the industry.
Story image
Cyber Criminal
Identity and access: the fight is on
Blue team defenders are used to protecting our data, applications, and users with access controls and other security mechanisms, which is why attacks like this are especially challenging when they target identity and access control systems.
Story image
Ping Identity appoints Deloitte Australia as a partner
Ping Identity has appointed Deloitte Australia as a Consulting Technology Partner, uniting its offerings with the company's consulting services.