[LINK] Server With Full Hard Drive Shuts Down Every Toyota Plant in Japan

[Stephen Loosley]

Server With Full Hard Drive Shuts Down Every Toyota Plant in Japan

With fourteen factories offline at the same time, the stoppage could 
have cost Toyota many millions in lost production.

BY LEWIN DAY  SEP 6, 2023 
https://www.thedrive.com/news/server-with-full-hard-drive-shuts-down-every-toyota-plant-in-japan


Modern automotive factories typically run 24 hours a day, 7 days a week, 
with the loss from even a short disruption typically measured in 
thousands or even millions of dollars.

Last week, Toyota faced a huge and uncharacteristic shutdown across 
multiple Japanese factories, all thanks to an embarrassingly simple IT 
problem.

As covered by The Guardian, all 14 Toyota assembly plants in Japan faced 
an unexpected stoppage on August 29. It was all thanks to a breakdown of 
a key computer system responsible for ordering vehicle parts for Toyota 
factories.

The cause of the problem? A full hard drive.

Toyota's just-in-time production system sees parts and components only 
delivered to the production line as needed. It's a boon to efficiency, 
but it can quickly lead to problems when things aren't operating 
perfectly. Thanks to this, the factories were unable to operate when the 
crucial system went down.

Toyota stated that during routine database maintenance the day before 
the incident, "data that had accumulated in the database was deleted and 
organized, and an error occurred due to insufficient disk space."

Like any diligent organization, Toyota had a backup in place, but as it 
was running "on the same system," a similar failure occurred and it 
could not be switched into operation. This failure brought all of 
Toyota's Japanese factories screeching to a halt in short order.

Eventually, the system was transferred to a server with excess storage 
capacity, and the assembly plants were able to resume operations.

Toyota's Japanese plants are responsible for around a third of its total 
vehicle output. Back-of-the-envelope maths from Reuters suggests the 14 
factories' average output is around 13,500 vehicles a day in total.

Based on current average Toyota sales prices and production rates, the 
loss of a full day's production could be as steep as $356 million in 
revenue. Take that with a grain of salt, but it's a guide to how high 
the stakes are when a country's worth of factories all go offline at the 
same time.

It's not the first time Toyota has seen a surprise problem knock its 
Japanese operations offline, either.

Last year, plants were stopped for a day when one of Toyota's suppliers 
reported its own servers had been infected with a virus. The issue 
raised questions about the cybersecurity of the production chain, though 
it bears noting that Toyota has clearly stated there was no cyberattack 
involved in this recent incident.


It's likely that some of Toyota's IT staff were in very deep trouble for 
this incident. If there's one thing to be said in their defense, it's 
that the vast majority of people stopped worrying about hard drive space 
shortly after multi-terabyte hard drives hit the market.

Regardless, if you're in charge of a crucial server that supports 14 
factories, it pays to make sure that it's not about to choke on its own 
data. Oh, and make sure the backup server actually works, too. That last 
one? That's kind of unforgivable.

--