[LINK] itNews: 'Dodo cops blame ...'
Roger.Clarke at xamax.com.au
Thu Feb 23 17:16:12 AEDT 2012
Dodo cops blame for national internet outages
Ry Crozier, James Hutchinson
Feb 23, 2012 3:05 PM (2 hours ago)
Telstra routers downed for 35 minutes.
Dodo has revealed a "minor hardware issue" was behind a Telstra
outage that impacted multiple service providers and internet services
The outage, which lasted approximately 35 minutes this afternoon,
impacted an international link used by major service providers
Telstra, Optus and iiNet for ADSL, cable and 3G data services.
Telstra said it had solved the issue but was still investigating what
Network engineers took to web forums with suspicions that a routing
issue originating in Dodo's network had caused the issue.
Industry sources said the network issue came as a result of Dodo
mistakenly issuing new IP route addresses from its system that
confused Telstra's systems and causing blackouts on the AS1221
A memo purportedly from Optus, and posted to Whirlpool, indicated
Dodo had "decided to advertise all the global routes it knows to
Telstra and for some unknown reason Telstra then accepted these as
'best path' which in effect meant ALL traffic originating from the
Telstra network would try and route traffic via Dodo".
Dodo chief executive Larry Kestelman confirmed the outage could be
sourced to Dodo as the result of a "minor hardware issue".
"Telstra and us are investigating why this had such an overstated
effect," he told iTnews.
Though the problem has been fixed in the meantime, it is expected
engineers at both telcos will spend the next 48 hours investigating
the outage before concluding the actual cause in Dodo's routers.
Users of multiple ISPs flooded Twitter and broadband forum Whirlpool
to seek information on the outage.
A post on the Whirlpool forums of a partial notification by wholesale
service provider Vocus appeared to point to a routing problem.
"Vocus network operations staff have been made aware of a routing
incident that has had significant impact on some domestic networks,"
the post read.
"This incident has impacted most of the Telstra domestic peering
points and caused a large amount of route churn and some sub-optimal
"Engineers from the affected network are working to restore traffic
back to its original paths but this may take some time."
Other network operators and ISPs reported downstream issues that
appeared to point back to the issue at a specific Telstra router.
"Looks like a major outage in Telstra network. Appears nation wide,"
Spectrum Networks tweeted, saying its network was operating as
ISP iiNet also pointed to an "upstream routing issue" as the cause of
a nationwide fault it described as having a "major" impact on
services. The fault had since been resolved, engineers said.
An iiNet representative said it was "intentionally avoiding Telstras
[sic] network where possible at the moment".
"You may see higher than usual latency to hosts in Australia (other
than via the IX [internet exchange] points)," Gavin Tweedie posted to
"We'll slowly bring back up the domestic Telstra capacity during the
course of this afternoon if performance is acceptable."
TPG also noted issues it attributed to a "3rd party carrier fault".
AusWeb said the outage was "causing major routing table changes
throughout the network."
The outage affected the websites of banks including Westpac. "There
is a @Telstra issue affecting westpac.com.au & other sites," the bank
"We're looking into to ensure it's resolved as soon as possible."
NAB reported similar issues on their Twitter feed.
Roger Clarke http://www.rogerclarke.com/
Xamax Consultancy Pty Ltd 78 Sidaway St, Chapman ACT 2611 AUSTRALIA
Tel: +61 2 6288 1472, and 6288 6916
mailto:Roger.Clarke at xamax.com.au http://www.xamax.com.au/
Visiting Professor in the Faculty of Law University of NSW
Visiting Professor in Computer Science Australian National University
More information about the Link