[LINK] Preservation of Data

[Markus Buchhorn]

At 10:14 AM 12/04/2007, Tom Worthington wrote:
>At 02:53 PM 5/04/2007, Markus Buchhorn wrote:
>>At 10:49 AM 5/04/2007, Tom Worthington wrote:
>>>... Researchers at government organisations need to keep in mind that unauthorized deletion of data is not just unfortunate: it is a crime.
>>
>>If you actually get into those laws (state, territory and federal, depending on the organisation) you'll quickly find that it's not at all clear that research data is a record in the legal sense. ...
>
>A court would need to decide if specific data was a "record". But it would seem reasonable that if it is worth paying a government employee to research something, then their data is worth keeping. At the very least, their research data is evidence they actually did some work.

I agree with you, but it is not part of our legal frameworks at this stage. So, courts do not even get involved. I'd love to see them have a go though...

>As an example, government money is used to track bushfires across Australia, via satellite <http://www.tomw.net.au/technology/it/emergency/sentinel/index.shtml>. 

Stop trying to make money off me with your ads ;-)

>Those running the system clearly have an obligation to keep the data. A court investigating deaths in a bushfire is likely to take a dim view of a researcher who says: "we were running out of disk space so I deleted the old records".

Be careful here, you're mixing several things:

- A government employee is not necessarily a researcher, and vice-versa. Where data is being collected under a clear business mandate, there will be a policy framework within that organisation, but there is rarely a legal framework that binds it. So 'obligation' to do anything is very context sensitive.

- Most folks in the data world have an innate feeling of ownership (rightly or wrongly) of the data they collected and will try to keep everything, or delete things on a value judgement, within hopefully a policy framework, rarely a legal one. Some policy frameworks that do exist even demand deletion of some data (data involving people, for example), even though it might have value down the track.

- When Sentinel moved to GA it became an 'operational' service, but that didn't necessarily include an obligation to keep bushfire-related data forever. The spatial data underlying it, and some of the remote sensing data is being kept, depending on which layers you are looking at. In some cases ownership of the layers does not even reside with GA, so they couldn't keep it if they wanted to.

- A court may take a dim view (or express disappointment), but it still has no power to do anything about it in this context, unless an entity is legally responsible for the job, in which case it isn't research data by most definitions. There have been plenty of cases, in this country and elsewhere, where data has been lost that could have been useful in trials, and feedback from the court says "we should do better". But that doesn't mean it creates large-scale policy changes. Sadly. Occassionally you do get some results, but only in the organisations where you'd expect it was part of their job (surveillance companies and government departments, for example).

I totally agree with you that data once collected should ideally be kept (for ever) in preference to being deleted (admittedly some stuff is a waste of space, almost immediately, or after a while). 

It gets hairy when people have to make value judgements. Why do they have to make those judgements? Resource constraints. Who pays for the long-term storage and properly curated archival of 'that' data? If you look at the current funding streams and processes around them, they do not address this, nor do they feel they should (do we fund fewer projects and expect them to do more with the money? Do we give more dollars up front to archival facilities and processes? (and fund fewer projects) etc). You saw my talk, so you know why this is hard :-)

It's something that's fairly high on various agendas at the moment (see for example the PMSEIC and PC reviews recently, and our report Evan referenced earlier). Whether that changes anything for the better remains to be seen....

Cheers,
        Markus