[LINK] itN: 'Salesforce users in 16 hour outage outrage'

Thu May 12 09:49:05 AEST 2016

[The article at the bottom of this posting suggests that, ever so slowly, there's an awakening to the fragility of cloud services.

[I've tackled this a few times:

   The Cloudy Future of Consumer Computing (2011)
   http://www.rogerclarke.com/EC/CCC.html

   How Reliable is Cloudsourcing? (2012)
   http://www.rogerclarke.com/EC/CCEF-CO.html

   Data Risks in the Cloud (2013)
   http://www.rogerclarke.com/II/DRC.html

   Backup Strategies for Users Dependent on Service-Providers (2015-16)
   http://www.rogerclarke.com/EC/PBAR-SP.html#BS

Here's an extract from the conclusion to the current paper:
>The analysis of five mainstream SaaS providers concluded that the backup requirements of small organisations and individuals are generally not currently capable of being satisfied either by trusting the SaaS provider, by backing up to the entity's own site, or by using a BaaS provider. Given the vital economic, social and personal importance of assuring ongoing access to data, this is a fairly remarkable finding. The adoption of cloud solutions may have been enthusiastic, but is has been blind, and that blindness will have serious negative consequences for some users.

And here are the points I made about Salesforce as at late 2015:
>Salesforce is quite blunt about its shortcomings. Its web-site declares that "Salesforce Support doesn't offer a comprehensive data restoration service", and warns the user not to rely on it for recovery purposes: "Although Salesforce does maintain back up data and can recover it, it's important to regularly back up your data locally so that you have the ability [to] restore it to avoid relying on Salesforce backups to recover your data. The recovery process is time consuming and resource intensive and typically involves an additional fee". It also explains that it is wildly expensive: "Data Recovery is a last resort process ... The price is a flat rate of $US 10,000" (emphases added). This would be very expensive for many micro-organisations and individuals, and prohibitive for some.
...
>Salesforce enables data to be extracted, but according to the company's site:
-   extraction can only be performed weekly or monthly
-   there may be frequent and long delays in a scheduled extraction 
    being performed
-   it can only be performed manually by the user and cannot be automated
-   the data is delivered in comma-separated values (CSV) files, with the
    only other separator available being a space. This is seriously 
    problematic, because:
    -   many of the fields already legitimately contain commas and/or spaces
    -   each CSV is an unstructured, flat file, whereas the database is
        structured
    -   there may be scores of record-types and hence scores of flat files 
>
>Further, the company acknowledges that recreating a Saleforce database from a backup may be very difficult, and it cannot provide assistance: "Overall planning, development and implementation of a strategy to manage record restoration falls outside the scope of Support's offerings". In short, the service does not appear to be capable of being effectively backed up to the user's site.
...
>Salesforce publishes APIs to BaaS providers, and a range of third parties claim to provide backup services. These include Backupify, Cloudally, Cloudfinder, Ownbackup, SesameSoftware, Skyvia and Spanning. All such products are oriented towards 'enterprises', i.e. large and medium-sized organisations, and none provide straightforward explanations of the kinds that would be comprehensible to small and micro-organisations. 

__________

Salesforce users in 16 hour outage outrage
By Allie Coyne on May 11, 2016 2:41PM
Benioff offers personal apology to angry customers.
http://www.itnews.com.au/news/salesforce-users-in-16-hour-outage-outrage-419400

Salesforce chief Marc Benioff has been forced to apologise to the company's customers after a 16 hour outage that is still ongoing [due?] to a North American instance [which?] downed operations around the country.

[I'm having trouble parsing that sentence.]

The Salesforce NA14 instance fell over late last night Australian time and has been unavailable to users since.

The outage stemmed from a database failure which introduced a file integrity issue, Salesforce advised customers.

Salesforce had moved the NA14 instance to a new site in Washington DC around eight hours before the outage, after a circuit breaker failure caused two hours of downtime at its former primary data centre in Herndon, Virginia.

[Maybe I'm naive, but I thought instances were managed by software rather than by people, and that failure of instances was normal, and that recovery from failed instances was too.  Whatever happened to rollback and recovery techniques?]

Other instances around America also suffered service issues at the same time as the initial failure, according to Salesforce's status page.

The company had been trying repair the file integrity issue, but revealed about two hours ago it would likely not be able to fix the problem.

"To bring NA14 back to full health, we have shifted our focus to recovering from a prior backup, which was not impacted by the file integrity issues," Salesforce said.

It is yet to provide an estimated time of return to service.

[So they *do* maintain occasional frozen mirrors and actionable logs - which would enable restart from a prior state and re-run forwards.  But they *don't* exercise the recovery routines, and hence are making it up as they go along?!] 

Benioff took to Twitter to personally respond to disgruntled customers, who rained [railed?] on social media in force to complain about the elongated outage.

The CEO offered an apology and his personal email address, copying in co-founder Parker Harris, in an attempt to placate irate users.

"I am sorry for our service disruption on NA14 please email me ceo at salesforce.com so we can call you," Benioff tweeted.

[And he's stampeding.  The logical thing to do is invent a one-time but proper-sounding email-address, e.g. BenioffApologises at saleforce.com.au, so that the generic (*not* personal) address ceo@ has a chance of avoiding being DOS'd to the point of having to be closed.]

-- 
Roger Clarke                                 http://www.rogerclarke.com/

Xamax Consultancy Pty Ltd      78 Sidaway St, Chapman ACT 2611 AUSTRALIA
Tel: +61 2 6288 6916                        http://about.me/roger.clarke
mailto:Roger.Clarke at xamax.com.au                http://www.xamax.com.au/ 

Visiting Professor in the Faculty of Law            University of N.S.W.
Visiting Professor in Computer Science    Australian National University