[LINK] Optus Outage Origins (OOO, nasty)

[Roger Clarke]

> On 13/11/23 17:06, Roger Clarke wrote:
>> This seems to indicate that the interpolations made early by linkers 
>> and others were pretty close to what actually happened ...
 >> 
https://www.abc.net.au/news/2023-11-13/optus-identifies-cause-of-nationwide-outage-software-upgrade/103099902

On 13/11/23 6:11 pm, Tom Worthington wrote:
> Yes, no surprises there. Similar to the problem which took out the 
> Australian Population Census in 2016: 
> https://www.abc.net.au/news/2016-08-09/abs-website-inaccessible-on-census-night/7711652 
>  
> One question which needs to be asked is if Optus is implementing the two 
> man rule. That is, can one person make a change to the system on their 
> own? There needs to be one person input the change and another check it.

One?  Onion rings / defence in depth used to mean plurality.
(Okay, accountancy debased that to dual-signatory approaches, but ...).


> On Sky News I was asked if Optus should have redundancy. It would be 
> possible to replicate all the hardware, but that would double to cost of 
> services to customers and would not stop a systematic failure of this sort.

Redundancy as a general principle is fine.  But if one limits their 
thinking to just one of the elements, even "hardware", they're done.


> One surprising outcome is that, in this case, mobile phones were more 
> reliable than land lines for emergency calls. The mobile phone standards 
> have provision for using any company's network to make an emergency 
> call. So phones automatically switched from Optus to Telstra, or Vodaphone.

Shhhhh!  Corporate strategists don't know that.  If they realise that's 
the case, they'll strangle the people they sent to Standards Committees, 
and debase the Standards.  (I'm joking of course.  Corporate strategists 
don't read things like link, and wouldn't understand if they did).


> The Australian Government is already working on mobile roaming between 
> carriers during natural disasters. 
> https://minister.infrastructure.gov.au/rowland/media-release/government-scope-emergency-mobile-roaming-capability-during-natural-disasters 
> 
> This could be extended to cover other network outages. On a trip to 
> India, I used one telco in Goa, and when they were not available in 
> Bangalore, the phone automatically switched to another network. This is 
> a commercial arrangement between carriers. It would require some 
> difficult commercial and regulatory negotiations to implement in Australia.

ACCC is authorised to, and does, grant requests for approvals of 
apparent complicity between competitors.  But I don't think they receive 
all that many requests ...


> For government, business, and domestic users of internet and phone 
> services there are some clear lessons from the Optus outage. Don't have 
> all your phones and Internet provided by the one company. If you are 
> providing safety critical services, have connections to multiple networks.

Two-SIM handhelds?  Rolling, cheap, year-long contracts with someone you 
can switch to if/when needed?  But watch out for unexpected 
pointc-of-failure / dependencies, non-scalability (Vodefone cope with 
large-scale and quick switching from a Telstra meltdown??), and the 
takeover game.


-- 
Roger Clarke                            mailto:Roger.Clarke at xamax.com.au
T: +61 2 6288 6916   http://www.xamax.com.au  http://www.rogerclarke.com

Xamax Consultancy Pty Ltd      78 Sidaway St, Chapman ACT 2611 AUSTRALIA 

Visiting Professor in the Faculty of Law            University of N.S.W.
Visiting Professor in Computer Science    Australian National University

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL: <https://mailman.anu.edu.au/pipermail/link/attachments/20231113/cd8aba59/attachment.sig>