[LINK] itNews: 'Dodo cops blame ...'

[Roger Clarke]

Dodo cops blame for national internet outages
  Ry Crozier, James Hutchinson
Feb 23, 2012 3:05 PM (2 hours ago)
http://www.itnews.com.au/News/291364,dodo-cops-blame-for-national-internet-outages.aspx

Telstra routers downed for 35 minutes.

Dodo has revealed a "minor hardware issue" was behind a Telstra 
outage that impacted multiple service providers and internet services 
nationwide.

The outage, which lasted approximately 35 minutes this afternoon, 
impacted an international link used by major service providers 
Telstra, Optus and iiNet for ADSL, cable and 3G data services.

Telstra said it had solved the issue but was still investigating what 
caused it.

Network engineers took to web forums with suspicions that a routing 
issue originating in Dodo's network had caused the issue.

Industry sources said the network issue came as a result of Dodo 
mistakenly issuing new IP route addresses from its system that 
confused Telstra's systems and causing blackouts on the AS1221 
upstream router.

A memo purportedly from Optus, and posted to Whirlpool, indicated 
Dodo had "decided to advertise all the global routes it knows to 
Telstra and for some unknown reason Telstra then accepted these as 
'best path' which in effect meant ALL traffic originating from the 
Telstra network would try and route traffic via Dodo".

Dodo chief executive Larry Kestelman confirmed the outage could be 
sourced to Dodo as the result of a "minor hardware issue".

"Telstra and us are investigating why this had such an overstated 
effect," he told iTnews.

Though the problem has been fixed in the meantime, it is expected 
engineers at both telcos will spend the next 48 hours investigating 
the outage before concluding the actual cause in Dodo's routers.

Widespread effect

Users of multiple ISPs flooded Twitter and broadband forum Whirlpool 
to seek information on the outage.

A post on the Whirlpool forums of a partial notification by wholesale 
service provider Vocus appeared to point to a routing problem.

"Vocus network operations staff have been made aware of a routing 
incident that has had significant impact on some domestic networks," 
the post read.

"This incident has impacted most of the Telstra domestic peering 
points and caused a large amount of route churn and some sub-optimal 
routing. 

"Engineers from the affected network are working to restore traffic 
back to its original paths but this may take some time."

Other network operators and ISPs reported downstream issues that 
appeared to point back to the issue at a specific Telstra router.

"Looks like a major outage in Telstra network. Appears nation wide," 
Spectrum Networks tweeted, saying its network was operating as 
normal. 

ISP iiNet also pointed to an "upstream routing issue" as the cause of 
a nationwide fault it described as having a "major" impact on 
services. The fault had since been resolved, engineers said.

An iiNet representative said it was "intentionally avoiding Telstras 
[sic] network where possible at the moment".

"You may see higher than usual latency to hosts in Australia (other 
than via the IX [internet exchange] points)," Gavin Tweedie posted to 
Whirlpool.

"We'll slowly bring back up the domestic Telstra capacity during the 
course of this afternoon if performance is acceptable."

TPG also noted issues it attributed to a "3rd party carrier fault".

AusWeb said the outage was "causing major routing table changes 
throughout the network."

The outage affected the websites of banks including Westpac. "There 
is a @Telstra issue affecting westpac.com.au & other sites," the bank 
tweeted.

"We're looking into to ensure it's resolved as soon as possible."

NAB reported similar issues on their Twitter feed.


-- 
Roger Clarke                                 http://www.rogerclarke.com/
			            
Xamax Consultancy Pty Ltd      78 Sidaway St, Chapman ACT 2611 AUSTRALIA
                    Tel: +61 2 6288 1472, and 6288 6916
mailto:Roger.Clarke at xamax.com.au                http://www.xamax.com.au/

Visiting Professor in the Faculty of Law               University of NSW
Visiting Professor in Computer Science    Australian National University