[LINK] Optus Outage Origins (OOO, nasty)

[Narelle Clark]

On Mon, 13 Nov 2023 at 21:04, Roger Clarke <Roger.Clarke at xamax.com.au> wrote:
> On 13/11/23 6:11 pm, Tom Worthington wrote:
> > On Sky News I was asked if Optus should have redundancy. It would be
> > possible to replicate all the hardware, but that would double to cost of
> > services to customers and would not stop a systematic failure of this sort.
>
> Redundancy as a general principle is fine.  But if one limits their
> thinking to just one of the elements, even "hardware", they're done.

Redundancy doesn't help when it is a signalling failure. Optus would
have had redundancy for the boxes and connections concerned, but the
issue was with the routing protocol that joins everything together.

The issue here is with segmentation and containment of the problem,
and with having (or rather not having) people on the ground able to
take the appropriate containment and remediation actions.

> > For government, business, and domestic users of internet and phone
> > services there are some clear lessons from the Optus outage. Don't have
> > all your phones and Internet provided by the one company. If you are
> > providing safety critical services, have connections to multiple networks.
>
> Two-SIM handhelds?  Rolling, cheap, year-long contracts with someone you
> can switch to if/when needed?  But watch out for unexpected
> pointc-of-failure / dependencies, non-scalability (Vodefone cope with
> large-scale and quick switching from a Telstra meltdown??), and the
> takeover game.

That Victorian trains went out is a serious design problem on their
part. That's not on Optus. Networks fail and that is what you design
for.

Same with other essential services - you design for the level of
failure you can withstand. If it's okay to be out for a day, then have
one network/service provider. If you can't then you need to design two
or more into place.

Regards

Narelle


-- 


Narelle
narellec at gmail.com