[LINK] Optus Outage Origins (OOO, nasty)

[Tom Worthington]

On 13/11/23 17:06, Roger Clarke wrote:
> This seems to indicate that the interpolations made early by linkers and 
> others were pretty close to what actually happened ...

Yes, no surprises there. Similar to the problem which took out the 
Australian Population Census in 2016: 
https://www.abc.net.au/news/2016-08-09/abs-website-inaccessible-on-census-night/7711652

One question which needs to be asked is if Optus is implementing the two 
man rule. That is, can one person make a change to the system on their 
own? There needs to be one person input the change and another check it.

On Sky News I was asked if Optus should have redundancy. It would be 
possible to replicate all the hardware, but that would double to cost of 
services to customers and would not stop a systematic failure of this sort.

One surprising outcome is that, in this case, mobile phones were more 
reliable than land lines for emergency calls. The mobile phone standards 
have provision for using any company's network to make an emergency 
call. So phones automatically switched from Optus to Telstra, or Vodaphone.

The Australian Government is already working on mobile roaming between 
carriers during natural disasters. 
https://minister.infrastructure.gov.au/rowland/media-release/government-scope-emergency-mobile-roaming-capability-during-natural-disasters

This could be extended to cover other network outages. On a trip to 
India, I used one telco in Goa, and when they were not available in 
Bangalore, the phone automatically switched to another network. This is 
a commercial arrangement between carriers. It would require some 
difficult commercial and regulatory negotiations to implement in Australia.

For government, business, and domestic users of internet and phone 
services there are some clear lessons from the Optus outage. Don't have 
all your phones and Internet provided by the one company. If you are 
providing safety critical services, have connections to multiple networks.


-- 
Tom Worthington http://www.tomw.net.au