[LINK] Optus Outage

Thu Nov 9 12:56:54 AEDT 2023

Wow -

For the record: Inner West Council(NSW) went out and this from NSW police

>
> UPDATE: Optus network disruption
>
> /Wednesday, 08 November 2023 03:41:31 PM/
>
> The Optus Network outage has caused a major communications disruption 
> to millions around the country today (Wednesday 8 November 2023).
>
> In NSW, mobile and landline services were impacted, including an 
> impact on calling Triple Zero (000), especially from fixed landlines.
>
> Service restoration is progressing across all domains - Regional and 
> remote service restoration will be concurrent with capital city services.
>
> Community members who are unable to contact emergency services due to 
> the outage are urged to try another device, or Telstra payphones until 
> their service is restored.
>
> Family, friends and neighbours of vulnerable community members are 
> urged to check in on them and alert emergency services if they need 
> assistance.
>
> There are no major disruptions to the operations of NSW Police Force, 
> Rural Fire Service, Fire and Rescue NSW, NSW Ambulance, Transport for 
> NSW or the Department of Education.
>
> Optus customers will not be able to reach the NSW SES on 132500 on 
> mobile phones or landlines until the outage is resolved.
>
> Mobile phones will connect to Triple Zero 000, which people should 
> call for life-threatening situations, provided there is an alternative 
> telecommunications network available.
>
> Service NSW can be reached on 13 77 88. There are likely to be longer 
> customer wait times.
>
> A statement from Optus is included below.
>
> OPTUS STATEMENT:
>
> https://www.optus.com.au/content/optus/en/notices/outage.html
>
> We're aware of an issue impacting Optus Mobile, Fixed Internet and 
> Phone services
>
> Latest updates:
>
> Some internet and phone services are gradually being restored.
>
> This may take a few hours for all services to come back online and 
> different services may restore at different sites over that time. No 
> action is required, and services will be automatically restored.
>
> We are aware of some mobile phones having issues connecting to 000. If 
> Optus customers need to call emergency services, we suggest finding a 
> family member or neighbour with an alternative device.
>
> We apologise to customers for the nationwide service outage that has 
> occurred this morning.
>
> We will continue to provide updates as we have information available.
>
https://www.police.nsw.gov.au/news/news_article?sq_content_src=%2BdXJsPWh0dHBzJTNBJTJGJTJGZWJpenByZC5wb2xpY2UubnN3Lmdvdi5hdSUyRm1lZGlhJTJGMTA5MDkxLmh0bWwmYWxsPTE%3D

>
> Media Statement - Triple Zero unaffected by Optus outage
>
> /Wednesday, 08 November 2023 10:05:31 AM/
>
> */Due to the ongoing Optus Network outage, the NSW Police would like 
> to reassure the community that there are no issues affecting Triple 
> Zero (‘000’) phone calls./*
>
> */Police urge anyone requiring urgent assistance to contract Triple 
> Zero. /*
>
> */For all other non-emergencies please continue to contact Crime 
> Stoppers on 1800 333 000, the Police Assistance line 131 444 or your 
> local police station./*
>
https://www.police.nsw.gov.au/news/news_article?sq_content_src=%2BdXJsPWh0dHBzJTNBJTJGJTJGZWJpenByZC5wb2xpY2UubnN3Lmdvdi5hdSUyRm1lZGlhJTJGMTA5MDg2Lmh0bWwmYWxsPTE%3D

On 9/11/23 11:33, Kate Lance wrote:
> Hi Narelle,
>
> An interesting post on Mastodon from Rob Thomas, supporting the idea 
> it was
> a route reflector overload -
> https://mastodon.au/@xrobau/111376847362633903
>
> The problem yesterday started at about 4am, when Optus told the world 
> 'I no
> longer have any internet connectivity', and 'Do not send any internet 
> traffic
> to me, at all'. The technical description is that they withdrew ALL of 
> their
> routes from the #DFZ (Which is "The Internet", as seen by all the core 
> routers
> that ACTUALLY control the internet).
>
> However, as a precursor at about 3am there was a hint that things weren't
> perfect, as there was a flurry of changes from Optus to the outside world
> saying, roughly, 'Something has changed inside my network, but you can 
> still
> keep sending me stuff'.
>
> Now, as two final bits of possibly relevant information, the default for
> maximum-prefix on #Cisco #ASR9000 is 1048576 (this number is 'the 
> number of
> routes that can be accepted by this router'), and MOST IMPORTANTLY the DFZ
> ("the internet") has about 980,000 routes in it at the moment. That's 
> only 90k
> odd routes LESS than the default maximum.
>
> I'd be amazed if Optus has less than 100k internal routes that aren't 
> visible
> to the internet, but are visible internally.
>
> So here's what I think happened. The at 3am, the first core #router was
> upgraded, and a new config was put in place. This did not join the network
> correctly, and things were half broken. What SHOULD have happened is 
> that all
> the changes should have stopped, and either rolled back, or waited for 
> further
> investigation (the cause being that more than 1mil routes were 
> visible, causing
> it to shut down)
>
> However, someone decided 'Well, maybe if we upgrade the SECOND one, 
> that'll fix
> the first one' at 4am. That broke the SECOND one, and took Optus 
> completely off
> the internet.
>
> (Continued, see next for why this is far worse than it should have been)
> .....
>
>
> Regards, Kate
>
>
> On Wed, Nov 08, 2023 at 05:33:43PM +1100, Narelle Clark wrote:
>> Rumour has it was a BGP update from an external source that wasn't 
>> filtered
>> properly with which the BGP route reflectors then overloaded the internal
>> routers. Persistently.
>>
>> It was clearly an internal transport problem arising from an 
>> underlying IP
>> protocol. BGP fits that bill completely as it would be redistributed, and
>> clearly their management network isn't sufficiently out of band. Once a
>> network of that scale goes down like that, you can't just turn it back on
>> and expect it to all work fine - millions of devices all want to
>> re-register at once, and all those state changes across the network 
>> have to
>> converge...
>>
>> Narelle
>>
>> On Wed, 8 Nov 2023 at 10:30, Alex (Maxious) Sadleir <maxious at gmail.com>
>> wrote:
>>
>>> Around 4am, Optus networks re-announced all their BGP routes at once
>>> https://radar.cloudflare.com/routing/as7474
>>> https://radar.cloudflare.com/routing/as4804
>>> This is indicative of a change management malfunction, akin to IBM's
>>> 2016 eCensus routers restarting with no routes
>>>
>>> https://www.itnews.com.au/news/ibm-treasury-in-settlement-talks-over-census-failure-440066
>>>
>>> The VoWifi infrastructure seems to be online but unable to connect any
>>> calls
>>> https://goughlui.com/2023/11/08/breaking-optus-nationwide-outage-08-11-2023/
>>> More alarmingly 000 doesn't work on landlines
>>> https://twitter.com/lucethoughts/status/1722029287727825124 contrary
>>> to advice from emergency services
>>> https://twitter.com/nswpolice/status/1722028862161449151
>>>
>>> On Wed, Nov 8, 2023 at 10:08 AM Tom Worthington
>>> <tom.worthington at tomw.net.au> wrote:
>>>> Any more news on what caused the Optus network outage? On ABC Canberra
>>>> Radio this morning I suggested it was most likely a software upgrade
>>>> which went wrong, and would be fixed by 6pm.
>>>>
>>>> Is VoWiFi working?
>>>>
>>>> I use Telstra, but when COVID-19 struck, I purchased an Optus 4G modem,
>>>> with an Optus SIM. This was in case Telstra went down.
>>>>
>>>>
>>>> --
>>>> Tom Worthington http://www.tomw.net.au
>>>> _______________________________________________
>>>> Link mailing list
>>>> Link at anu.edu.au
>>>> https://mailman.anu.edu.au/mailman/listinfo/link
>>> _______________________________________________
>>> Link mailing list
>>> Link at anu.edu.au
>>> https://mailman.anu.edu.au/mailman/listinfo/link
>>>
>>
>> -- 
>>
>> Narelle
>> narellec at gmail.com
>> _______________________________________________
>> Link mailing list
>> Link at anu.edu.au
>> https://mailman.anu.edu.au/mailman/listinfo/link
> _______________________________________________
> Link mailing list
> Link at anu.edu.au
> https://mailman.anu.edu.au/mailman/listinfo/link

-- 
Marghanita da Cruz
Telephone: 0414-869202
Email:  marghanita at ramin.com.au
Website: http://ramin.com.au