[LINK] Optus Outage Origins (OOO, nasty)

[Tom Worthington]

On 16/11/23 15:49, Roger Clarke wrote:

> It might be worth running to ground who started the story (yesterday?) 
> about it being Singtel's fault...

I couldn't find out any more, but from what Brendan Arrow was asking me 
on ABC New Radio this morning, Singtel is not happy about Optus blaming 
them.

More may come out in today's Senate hearing. So far there is only a 
submission from Optus, which has a reasonable level of technical detail, 
but seeming to want to blame Cisco, & Singtel:

	"19. It is now understood that the outage occurred due to approximately 
90 PE routers automatically self-isolating in order to protect 
themselves from an overload of IP routing information. These 
self-protection limits are default settings provided by the relevant 
global equipment vendor (Cisco).

	20. This unexpected overload of IP routing information occurred after a 
software upgrade at one of the Singtel internet exchanges (known as 
STiX) in North America, one of Optus’ international networks. During the 
upgrade, the Optus network received changes in routing information from 
an alternate Singtel peering router. These routing changes were 
propagated through multiple layers of our IP Core network. As a result, 
at around 4:05am (AEDT), the pre-set safety limits on a significant 
number of Optus network routers were exceeded. Although the software 
upgrade resulted in the change in routing information, it was not the 
cause of the incident.

	21.Restoration required a large-scale effort across more than 100 
devices in 14 sites nationwide to facilitate the recovery (site by 
site). This recovery was performed remotely and also required physical 
access to several sites."

https://www.aph.gov.au/sitecore/content/Home/Parliamentary_Business/Committees/Senate/Environment_and_Communications/OptusNetworkOutage/Submissions


-- 
Tom Worthington http://www.tomw.net.au