|
|
This topic comprises 2 pages: 1 2
|
|
Author
|
Topic: Digital Archiving Artical in NY Times
|
Leo Enticknap
Film God
Posts: 7474
From: Loma Linda, CA
Registered: Jul 2000
|
posted 12-26-2007 10:05 AM
The basic, underlying problem is that there's no known way to archive digital data 'offline' with anything approaching the same longevity and reliability as an analogue image on film.
We now know enough about the chemistry of film bases and emulsions, and their behaviour over time, to be reasonably sure that in the right atmospheric conditions, it'll last for centuries (and ironically, nitrate is proving to be the most stable of all the bases that were ever in mainstream use). There is simply no equivalent for data. The options are:
1. Magnetic tape. It also has exacting temperature and humidity requirements, can be subject to chemical decomposition (e.g. sticky shed on some types of polyester mag, or binder layer breakdown on evaporated metal oxide), and presents a major hardware and software obsolescence problem. So major, in fact, that going down this route more or less commits you to a programme of continuous format migration. The bigger your collection, the bigger the cost and complexity of this.
2. Optical discs. Lots of research going on, but the bottom line is that we simply don't know how chemically stable the dyes and substrates are in the long term. There is still a potential obsolescence issue, although as a common form factor appears to be establishing itself, this may be less of a problem than with tape.
3. Hard discs archived offline. Some in the archival world are seriously trying to push this as a solution, while others have serious misgivings. It's certainly true that the cost per gigabyte is falling like a stone and that HDDs don't have any particularly expensive atmospheric requirements; but even in as little as 20 years time, how easy will it be to connect them to the current generations of computers? And what are the mechanical implications of spinning one up after many years of storage?
4. 'Always on' HDD storage, i.e. RAID or MAID arrays. Probably quite reliable, but very expensive and, due to the power input needed, not particularly environmentally friendly. Certainly not a viable option for smaller, public archives on a shoestring budget.
5. Digital data on film emulsion. Some think that this is the great white hope: the proven stability of b/w polyester film, very simple optics and mechanics to read it back but a potentially large storage capacity if you use the whole film width. There was a panel on it at the last but one AMIA conference, but from the presentations there I quickly concluded that the sums didn't add up. They were talking less than a terabyte (four hours of digibeta) on a 2,000 foot roll of 35mm: the cost of preserving film-resolution material this way doesn't bear thinking about, and given that most of the cost to the customer is in stock and processing, this is going to get more expensive, not less, as overall film use continues to reduce. Besides, there is not even a viable system on the market as yet.
So, sadly, I fear that a lot of 'born digital' material is not going to survive in the long term, because the cost and complexity of preserving it is going to be beyond the reach of all but the best equipped archives.
| IP: Logged
|
|
|
|
|
|
Leo Enticknap
Film God
Posts: 7474
From: Loma Linda, CA
Registered: Jul 2000
|
posted 12-27-2007 03:24 AM
Exactly. For that reason, some people are advocating using no compression whatsoever for archival storage. The problem there is that an hour of NTSC comes to about 250gb, so you can imagine the volume of data storage involved with even a substantial TV collection, let alone 2k or 4k resolution film. Others believe that lossless compression is the way to go - for example this company uses MJPEG2000 in its automated, robotic videotape archiving system. Most are agreed that lossy compression (e.g. MPEG-2) is an absolute no-no, both for the reason you suggest (will the codecs still be available in the 23rd century?), and because you've introduced permanent generational loss relative to the original. Furthermore, the only way to copy the data thereafter involves decompressing it (some codecs do this differently from others, hence introducing even more digital noise, potentially) and then recompressing it into whatever output format you want.
As for solid state storage, the general consensus at the moment seems to be that the volume of data storage is the major problem. Flash memory degrades over time (i.e. the stuff on your USB stick will still be there in 2-3 months, but it may well not be in 2-3 years and it certainly won't be in 20-30 years), and the more permanent alternatives are so expensive per GB archived that for the amount of data required by a major collection, not even the most lavishly funded archives in the world could contemplate it.
| IP: Logged
|
|
Stephen Furley
Film God
Posts: 3059
From: Coulsdon, Croydon, England
Registered: May 2002
|
posted 12-27-2007 04:39 AM
quote: Leo Enticknap
4. 'Always on' HDD storage, i.e. RAID or MAID arrays. Probably quite reliable, but very expensive and, due to the power input needed, not particularly environmentally friendly. Certainly not a viable option for smaller, public archives on a shoestring budget.
But you can't just forget about it; in our server room we have fifty odd servers, with something like 250 hard discs between them. Most of these are replaced on a five year cycle, but even so e typically have 4-5 discs fail per year; we actually had one go a few hours before we closed for Christmas. 95% of these are SCSI, or on the later servers SAS, drives, with the system usually on a pair of mirrored drives, and the data on a raid 5 array. sometimes with a running hot spare. All of this requires monitoring and maintenance, takes something like 13kVA from the mains, and requires a similar amount of power for the air conditioning to remove the heat produced. I can't remember the total amount of available storage, but it's somewhere around 10 TB.
quote: Leo Enticknap
We now know enough about the chemistry of film bases and emulsions, and their behaviour over time, to be reasonably sure that in the right atmospheric conditions, it'll last for centuries (and ironically, nitrate is proving to be the most stable of all the bases that were ever in mainstream use).
How is polyester in this respect; or is nobody prepared to commit themselves to predicting a figure yet. I have a few reels of Ferrania and Prinz Super-8 film from the late '60s somewhere; the last time I looked, a few years ago, the colour fas faded, but the film looked physically ok. The Kodachrome on Triacetate also seemed to be ok; no vinegar.
I think that, in the longer term, safety colour material will prove to be a bigger preservation problem than B/W nitrate. Look at the restoration that was necessary on 'Vertigo', and I've heard that there are other major films that are equally badly preserved.
I suspect that the issue of obsolete hardware and software will not be too much of a problem for the major formats; if somebody finds a DVD of something important in the future that needs to be transferred then there will probably be a few DVD players somewhere in the world, simply because of the vast number that were produced. I would be more concerned about some of the less successful formats; how about the material which Thames Television produced on MII for example? How many servicable machines for that format still exist. Then you've got to know whether something you've found is worth preserving; If you find a reel of film somewhere you hold it up to the light, and you can see what's on it. If it turns out to be something important you can show people what you've got, and try to get funding to copy it. Even if it's some totally unknown format, say 51mm horizontally running, with seven perfs on on side and five on the other, it wouldn't be too difficult to machine up an optical printer gate for it, even if you had to pull down and register each frame manually. With some 'lost' digital format you can't even tell what's on the media without spending large amounts on building equipment the retrieve the data, and then working out how to decode them.
| IP: Logged
|
|
|
|
Leo Enticknap
Film God
Posts: 7474
From: Loma Linda, CA
Registered: Jul 2000
|
posted 12-27-2007 12:14 PM
quote: Stephen Furley
All of this requires monitoring and maintenance, takes something like 13kVA from the mains, and requires a similar amount of power for the air conditioning to remove the heat produced. I can't remember the total amount of available storage, but it's somewhere around 10 TB.
Or in other words, about two feature films at 4k resolution.
quote: Stephen Furley
How is polyester in this respect; or is nobody prepared to commit themselves to predicting a figure yet.
As it's an inorganic polymer, the base itself is said to be good indefinitely with no special atmospheric requirements. The problem, with both photographic emulsion and magnetic oxide on polyester, is the binder layer which sticks one to the other. There is some anecdotal evidence of the emulsion peeling off the base of b/w polyester film which is only 10-15 years old. Unlike nitrate and acetate decomposition and colour dye fading, all of which have 20-30 years of research by three separate institutes to tell us how it happens, we really know very little about what polyester film does over time.
quote: Stephen Furley
The Kodachrome on Triacetate also seemed to be ok; no vinegar.
Because it's non-substantive (i.e. the dyes are introduced to the couplers during processing rather than linked to the emulsion in the film before it's exposed), the colour dyes on Kodachrome are very stable over time - to all intents and purposes, as good as an IB print.
quote: Stephen Furley
I think that, in the longer term, safety colour material will prove to be a bigger preservation problem than B/W nitrate. Look at the restoration that was necessary on 'Vertigo', and I've heard that there are other major films that are equally badly preserved.
Digital intermediate technology is now able to help a lot with dye fading. Vertigo was one of the last major all analogue feature film restorations. The basic technology used to do it (printing through filtered light in order to extract three separate colour records from the best surviving elements) works, but is very difficult and expensive. The hardware and software needed to scan at up to 4k resolution, rebalance the colour and then burn out separations is coming down in price and increasing in capability all the time. Agreed entirely that stored in the right atmosphere, we now know that b/w nitrate is incredibly stable stuff. But substantive coupler dye fading can also be slowed down almost to the point of arrest by by cold storage, and digital restoration technologies are coming along in leaps and bounds. So I'm a bit more optimistic about the ability of archives to deal with this stuff in the future. A bigger problem, IMHO, will be films that are inappropriately 'restored' by amateurs who have the kit but not the expertise to use it.
quote: Wayne Keyser
And as Archive.org and its abundance of new projects is showing (as if we didn't know from our own laments about vanished films) the answer is: "everything is worth preserving."
Agreed in principle, but often the issue is not as simple as that. All properly run archives will have an acquisition policy - a statement which says what they do and don't collect, and why. There's no point in collecting material which is already preserved by another archive, and if you have limited preservation resources you may well have to decide that you're not in a position to take certain collections on which are offered to you.
| IP: Logged
|
|
Stephen Furley
Film God
Posts: 3059
From: Coulsdon, Croydon, England
Registered: May 2002
|
posted 12-27-2007 12:19 PM
quote: Wayne Keyser
And as Archive.org and its abundance of new projects is showing (as if we didn't know from our own laments about vanished films) the answer is: "everything is worth preserving."
If you look a a reel of film, and you see that it's a poor quality print, possibly incomplete, with significant damage, of something that you already have much better elements for, then it's probably not worth preserving. There may be exceptions, for example, if it's a Technicolor dye print of something that you have the original negatives for then you wouldn't want to duplicate from it, but it might be useful as a colour reference when making new colour prints. The trouble with digital media, is that you can't tell just by looking at them what they might contain. It might contain an episode of Big Brother, and be labeled as such, but also contain, in the last few minutes, a recording of some historically important event, not known to exist elsewhere, but you don't know that, and if you don't have the correct hardware and software available to play it, you can't easily find out without spending lots of money just on the offchance that there might be something interesting on it.
| IP: Logged
|
|
|
|
All times are Central (GMT -6:00)
|
This topic comprises 2 pages: 1 2
|
Powered by Infopop Corporation
UBB.classicTM
6.3.1.2
The Film-Tech Forums are designed for various members related to the cinema industry to express their opinions, viewpoints and testimonials on various products, services and events based upon speculation, personal knowledge and factual information through use, therefore all views represented here allow no liability upon the publishers of this web site and the owners of said views assume no liability for any ill will resulting from these postings. The posts made here are for educational as well as entertainment purposes and as such anyone viewing this portion of the website must accept these views as statements of the author of that opinion
and agrees to release the authors from any and all liability.
|
Home
Printer-friendly view of this topic