[LINK] Guidelines for Digital Repositories, Canberra, 27 July

[Markus Buchhorn]

Hi Gordon

At 10:11 AM 21/07/2006 +1000, Gordon Keith wrote:
>Related to this topic, does anyone know if any guidelines exist in terms of 
>codecs suitable for archival of video data?

Presuming you mean digital preservation of the content, rather than the physical media, this is a topic near and dear to my heart :-). There are several groups that have published discussion papers on the problem, in AU, the US, the UK, and some in Asia (Taiwan, Korea), but none really go into guidelines specific to video (and other moving image formats). There are plenty of guidelines on general archival issues for digital materials, see e.g. the NLA.

This thread's already covered the major issues regarding compression, to which I'd add the minor point that most a/v compressions are perception-based, not information-based. i.e. they target delivery to human senses, not to computers for analysis. Not always what you want. (Did you know most speech compression codecs are tuned for American English? Don't work at all for Chinese etc.)

There's also the issue that basically anything that goes through an algorithm of any kind has to also preserve the algorithm itself - which can be fun :-/ (can get into issues of preserving languages, compilers and operating systems).

As a specific note for video, I know a colleague at the NLA is looking hard at mjpg2000, i.e. motion-jpeg, using wavelet-based compression, which can be lossless but still provide significant space-savings. Not totally sure on any patents/licenses yet.

Regarding bit-rot, you can avoid that by reducing your compression ratio a bit, by including full error-correction information (basically checksums). That's a pretty well-established technology nowadays (see the streams coming from the various remote spacecraft - you don't get a second chance). I don't know if there's a profile of mjpg2000 that includes EC data though, might need to be worked on.

And if you want some light bedtime reading :-) (ignore the marketing blurb):

LOSSLESS COMPRESSION handbook. 
Author/Editor:    Sayood,K. 
Publisher:    Academic Press 
Pages:    550 
ISBN:   0126208611 

Cheers,
        Markus