Film-Tech Forum ARCHIVE: Academy Warning on Digital Preservation

This is topic Academy Warning on Digital Preservation in forum Film-Yak at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=8;t=006604

Posted by Martin McCaffery (Member # 37) on 01-21-2012, 12:16 PM:

From VARIETY (18 January 2012) at http://www.variety.com/article/VR1118048861?refcatid=1009&printerfriendly=true. You can download the AMPAS report at http://www.oscars.org/science-technology/council/projects/digitaldilemma2/index.html.

Acad sounds alarm about fragility of digital prod'n
Sci-Tech report spotlights short lifespan of non-film formats
By DAVID S. COHEN
On the day that Sundance kicked off and Eastman Kodak filed for bankruptcy, the Motion Picture Academy threw a bucket of ice water on the digital filmmaking revolution.
Preserving movies is an ongoing issue for the entire industry, but a new report from the Acad warns that movies shot or finished digitally face a lifespan so short they can be lost even before they get distribution. Worse, indie and docu filmmakers, whose work is most vulnerable to this risk, seem oblivious to the danger.Those grim conclusions, found in the long-awaited Part Two of the Acad's Science & Technology Council "Digital Dilemma" report on the problems of digital preservation, will likely make for some somber chatter in Park City.

Where part one (released in 2007) focused on the studios, the second installment looks at indies and docs and finds "the technology that makes it easy to make the picture also underlies the lack of guaranteed long-term access to it." And while the Acad found those communities still ignorant of the fragility of digital files, it may not matter -- those sectors lack the resources to attack the problem anyway.

"The bottom line is we're running out of time," Sci-Tech Council member Milt Shefter, co-author of the report, told Variety. "The time for studies is past. We have to find some solutions or we're going to lose a lot of material."

In short, digital storage, be it on hard drives, DVDs or solid-state memory, simply isn't on a par for anything close to the 100-plus-year lifespan of film. The life of digital media is measured in years, not decades, and file formats can go obsolete in months, not years. As the report explains, that affects movies still looking for distribution, not merely library titles. "In general," the report says, "independent films that beat the odds and secure some form of distribution do so after a much longer time period than movies produced by the major studios. This time period quite likely exceeds the 'shelf life' of any digital work; that is, by the time distribution is secured, the digital data may become inaccessible.

"Most of the filmmakers surveyed and interviewed for this report were not aware of the perishable nature of digital content or how short its unmanaged lifespan is compared to the 95-plus years that U.S. copyright laws allow filmmakers to benefit from their work."

Much indie content, the report says, is in danger of being lost before it can receive the full benefits of those 95 years of protection.

Shefter called filmmakers' ignorance of the issue "probably our biggest surprise."

"They were concentrating on the benefits of digital workflow," he said, "but weren't thinking about what happens to their (digital) masters. They're structured to make their movie, get it in front of an audience, and then move onto the next one."

Also a surprise to the Acad's researchers: Documentarians also were unaware of the vulnerability of digital files. On the contrary, documakers were generally excited about the easy access to footage in the digital age. When Acad interviewers raised the idea that there might be "a black hole for the last 25-30 years" because digital files aren't being preserved, said Shefter, "they really didn't get that."

"The main difference between analog and digital is, analog was store-and-ignore," said Shefter. "Digital has to be actively managed."

Such active management is expensive, however, vastly more expensive than putting film in a vault. Even when they take such steps, however, filmmakers and producers are up against an insurmountable problem: The only reliable method for archiving digital images is to go analog. The best archiving solution today is to print out to film, ideally with a three-color separation printed onto black-and-white archival film. That's a very expensive solution.

The Academy is doing what it can to help address the problem, said Andy Maltz, director of the Sci-Tech Council. "One of the keys to preservation is to have file-format standards, so if you can recover the zeros and ones, you'll know what they mean and know what they're supposed to look like on the screen." The Acad's Image Interchange Framework project is helping create such standards. SMPTE will be publishing the first of them later this year.

The Acad is coordinating Hollywood's efforts to work with the Library of Congress and with other industries to find a method for archiving digital data. But, said Maltz, "It's up to the manufacturers to incorporate archival lifetimes into their products." Fortunately for the entertainment industry, it's not alone in facing this issue. Banking, medicine, energy and other fields all need to preserve digital data for more than a few years, and they're all looking for the same elusive breakthrough.

The report says that unless preservation becomes a requirement for planning, budgeting and marketing strategies, it will remain a problem for indie filmmakers, documentarians and archives alike. "These communities, and the nation's artistic and cultural heritage, would greatly benefit from a comprehensive, coordinated digital preservation plan for the future."

The report includes proposals for more education, sharing of information and collaboration among archives and other orgs.

Shefter was careful to say that the Council and the report are not attacking digital, which offers "tremendous benefits" in some areas. Said Shefter, "The broader issue is (that) as we embrace the benefits of the newer technology, one thing is missing: long-term guaranteed access. That's what the analog world had and we think any replacement should have at least as much."

Contact David S. Cohen at david.cohen@variety.com

Posted by Jenn Jennings (Member # 6455) on 01-21-2012, 06:40 PM:

Thanks for posting this! Also, thank you for allowing us to "film" you at the Art House Convergence. This is an important issue and your input on the issue is valuable to us. I will be notifying those that make the final cut.

Thanks again!

Jenn Jennings

Posted by Leo Enticknap (Member # 534) on 02-05-2012, 11:24 AM:

quote: Martin McCaffery
The main difference between analog and digital is, analog was store-and-ignore," said Shefter. "Digital has to be actively managed."

I don't think it's as simple as that. For one thing, storing was an active decision: a lot of early cinema has been lost because its creators regarded it as ephemeral, first and foremost a business asset, and thus not worth storing in the first place after its initial commercial lifetime. Secondly, if you ignored it, it could decompose in storage (film), or become obsolete and impossible to play back (some analogue tape formats).

So we have been here before. The real difference is that the migration cycles are so much shorter, and metadata replaces the information about the digital asset that with analogue films and tapes, was embodied in the physical object. I suspect that Moore's Law will inflate away a significant amount of the preservation problem, because it'll become cheaper and easier to preserve a given file as storage media capacities increase over time.

As an example, I remember when I bought my first CD burner in 1999: I was able to preserve the contents of all 300 or so floppy discs in my possession, some of which dated back to the first time I used a computer in the mid-80s, on the single disc that was thrown in with the drive. The contents of that disc have since been migrated to the hard drive backup system I use to keep my personal data safe now (copy everything to a USB hard drive every Sunday, which is then stored in my office 26 miles away during the week. All the files I created between the mid-80s and 1999 now represent less than 1% of the total contents of my 'My Documents' folder on my home PC.

But the essential point of the report is dead right - preservation and migration has to be done, done actively and managed.

Posted by Randy Stankey (Member # 64) on 02-05-2012, 12:38 PM:

I think the main problem is that digital files are intangible. You can hold the medium on which the information is written but it is not human readable. Unless you have the machine on which the data was created or a machine that is backward-compatible with the machine that created the data, you have no idea whether or not the data has been corrupted or even if you have any data at all.

Even if you photo-engraved the data onto a permanent, stable medium like a glass disk or a platinum/palladium platter you still can't read the data. Without some sort of indexing and cataloging system, you couldn't tell one file from another.

Way back in 2001, I posted a frame scan on Film-Tech which was from a piece of a movie film I found in an old theater and I asked if anybody could identify the movie. John Pytlak posted the answer in 31 minutes. There is no way that could have happened if the movie was digital. In fact, if I walked around in an old theater in the future, I probably wouldn't be able to tell that there were even any old movies lying around in the first place, let alone identify them.

Posted by Mitchell Dvoskin (Member # 751) on 02-05-2012, 01:44 PM:

Taking Randy's post a step further, unless the digital media is marked as too it's format, and is a complete file, it will be hit or miss as to whether or not you will ever find the appropriate software to read it. That is especially true with propriety commercial formats and encrypted release files.

Posted by Louis Bornwasser (Member # 3063) on 02-05-2012, 03:02 PM:

Don't ask if there is anything made today that will be anything other than "an artifact of 2011 or 2012." I am not being a wise a$$, Looking backward at any title we respect (still selling on dvd, etc.) is there anything like this now or is it ALL just disposable? To be sure, the past was like this too, but there were a few keepers. Louis

Posted by Randy Stankey (Member # 64) on 02-05-2012, 03:54 PM:

But, how can anybody know what movies will become culturally significant?

For instance, "Rocky Horror"
When that movie was made it was considered a flop. Worse than a flop. Pretty much everybody at the studio wanted to bury that movie where it could never be found again. But, for some strange reason that nobody knows, "Rocky Horror" found a midnight cult following.

Further, if you simply look at the studio's money making angle, "Rocky Horror" is STILL making money almost 40 years after it was made. Who would have guessed it? Who COULD have guessed it?

If "Rocky Horror" had been stored on a hard drive, there's a good chance it might have never seen the light of day if it still existed at all.

Posted by Philip Jones (Member # 6677) on 02-05-2012, 04:46 PM:

is this so much of a problem these days (for mainstream films at least) since there are hundreds of thousands of copies of most things produced now for the home market.

Posted by Louis Bornwasser (Member # 3063) on 02-05-2012, 07:41 PM:

Good point! With such ubiquity who will care? Louis

Posted by Bobby Henderson (Member # 840) on 02-05-2012, 09:33 PM:

The problem is consumer copies (movies on DVD, Blu-ray disc, etc.) are SHIT compared to the original master source material.

For instance, while 1080p/24 material on Blu-ray can be impressive it is wholly inferior to the HD master material used to encode the Blu-ray source. Blu-ray at best has a 50 million bit per second maximum bandwidth. Uncompressed 1080p/24 HD at a basic level of 8-bits color per channel uses roughly 1.4 billion bits per second in terms of bandwidth. If you want to go farther into "deep color" modes like 10-bit, 12-bit or even 16-bit double buffered color depths then the bandwidth needs increase geometrically.

Posted by Mark J. Marshall (Member # 1409) on 02-05-2012, 11:05 PM:

Now is a great time to figure this out, Hollywood.

Congrats on that!

Posted by Joe Redifer (Member # 3) on 02-05-2012, 11:06 PM:

I remember when Mac OS 7.1 came on a bunch of floppy discs, maybe 6 or 7 of them. Everyone was like "Damn, this OS is HUGE!" That really wasn't that long ago in the grand scheme of things, so I imagine that digital storage ability will only increase. I would certainly advise against storing compromised digital versions of movies in the mean time. The only reason file formats become outdated is because new compression schemes will be created. But uncompressed is always uncompressed. It doesn't need to be a particular flavor (like MPEG 4). As long as the industry keeps shit around that can read uncompressed files, I really don't see what the big deal is.

Posted by Manny Knowles (Member # 1171) on 02-06-2012, 12:12 AM:

As an independent filmmaker who came close to losing segments of a film I made in 2001, I can confirm the need to archive actively.

In my case, it wasn't an issue of whether there was software that could read the file. As it happened, the master tape (Mini-DV) had dropouts near the start of the tape... and the editorial file was missing several shots, so we couldn't simply output to a new master tape.

Fortunately, the original camera tapes had been properly filed and we were able to recapture the missing shots and reconstruct the whole film.

Yes, there were at least a few other copies of the entire film intact. But these were lesser-quality versions including: a QuickTime file (some compression), a few DVD's (noticeably compressed) and a backup output to Mini-DV (in LP mode). Luckily, the camera tapes were available.

That was a wake-up call, proving the old adage -- "There are two kinds of people: those who have lost data, and those who will."

If money was no object, I'd be in favor of originating on film and doing everything else digitally (including distribution/exhibition) -- preservation would include black-and-white separations on film. Again... if money was no object.

In the real world... periodic backup drives and tapes will have to suffice.

Posted by Randy Stankey (Member # 64) on 02-06-2012, 12:50 AM:

quote: Joe Redifer
As long as the industry keeps shit around that can read uncompressed files, I really don't see what the big deal is.

That is the problem, right there. How can you read any digital file if you don't have a machine that can read it and interpret it?

If I handed you a file on a floppy disk and told you to read it, you'd probably laugh at me. Now, if I handed you a box of Hollerith cards and told you that some vital piece of information is contained on the media in that box and, if you couldn't read it, your business would go bankrupt you'd be stuck holding your ass.

In 50 years, how do you know that computer data isn't going to be stored on some kind of holographic crystal or something like that? A magnetic hard drive will become the Hollerith cards of the future.

This begs the question that, even with a machine that can read the file, you'll even know what that file or what digital format it is encoded in? Even if you know that, how are you going to interface that old machine and that old data format with current computers?

What if you had to interface an ASR-32 teletype to your computer?
Can you even get your computer to understand 5-bit Baudot code?
(BTW: I know a guy who did this... just because he could.)

It is possible to play a 50 year old movie on 35mm film on a projector that was made today (recently) but there's no way of knowing whether you'll be able to play a movie made today and stored on digital video 50 years from now.

Posted by Manny Knowles (Member # 1171) on 02-06-2012, 01:06 AM:

Oh, Randy!

We can project film prints older than 50 years!

Why...just earlier today... I ran a print of a 1922 film. [Big Grin]

No question about film's track-record when it comes to accessibility. But I still maintain that exhibition on film is inferior to digital, from a practicality standpoint.

I know, I know... Everyone wants to believe they're "doing film right." But even if you are, you're still at the mercy of the print you get.

Posted by Frank Cox (Member # 6258) on 02-06-2012, 01:22 AM:

I know of a guy who makes a rather good living out of reading obscure and obsolete magnetic media. He has a huge collection of everything from early word processors to tape drives and you-name-it. It's my understanding that he does work mostly for forensic accountants, law enforcement and private investigators.

It's not particularly difficult to read a card deck "manually", just by looking at it and doing some plotting. The data density is obviously very low, and if you know what kind of a computer it came off of (if you're lucky it's noted on the deck somewhere) you can probably figure out if it's 9-left, Chinese binary or something else. A bit of amateur cryptography will probably get you there if nothing else -- if 9-left doesn't provide sensible output, it's probably not encoded that way.

Many years ago, when dinosaurs roamed the earth, the police in the town I lived in at the time seized a lawyer's office and discovered that he had all of his records on a computer. Nobody knew anything about it, in fact they were afraid to touch it for fear of losing evidence. They called me to look at it, and I spent two days there figuring out how he did what he did and printing out records on paper.

It is possible but obviously not cheap to read hard drives and tapes with an electron microscope. There are also several outfits around who do data restoration; again, none of them are cheap. (And if they are, don't deal with them.)

Posted by Joe Redifer (Member # 3) on 02-06-2012, 01:50 AM:

quote: Randy Stankey
That is the problem, right there. How can you read any digital file if you don't have a machine that can read it and interpret it?

That was my whole point. They need to make sure that they have stuff that can read it. The machine is irrelevant, the software is what is needed. They need to have software that can read it and when they change hardware they need to update their software so that their new machines have the software that can read it. If that sounds too complex for the studios to handle, I weep for humanity's idiocy. Likewise, they need to make lots and lots of copies of their files. When new storage media comes out, copy that shit over! Someday it will be all solid state and the only way it won't work is if it gets physically damaged (fire, flushed, etc) thus taking the media storage format out of the equation.

Posted by Manny Knowles (Member # 1171) on 02-06-2012, 02:06 AM:

Tom Holman often used the term "bit slinging" to describe the work of migrating digital files.

As far as the major studios... I recall seeing job postings for this kind of work all the time. I suspect they are investing in the preservation of their assets. Especially given that we now understand that there's money to be made by repackaging previously-released material.

It's the independent filmmakers that are at risk of losing their stuff. Not so much because of expense but, as the article points out, because archiving hasn't typically been integrated into the "independent" workflow.

Posted by Randy Stankey (Member # 64) on 02-06-2012, 02:45 AM:

It isn't just software that is needed to read a computer file from a disk drive or other storage medium.

What if you had to read a hard drive with a SCSI interface or a PATA interface? Your computer has SATA. How are you going to get that data?

A teletype doesn't even have software. It's entirely electromechanical and it transmits its data with current detector loops. To interface that with your computer, you need to build a circuit that connects to the current loops on the teletype and puts out some sort of bit code that the computer can understand. Then, assuming you got that far, how do you even figure out what software to use to interpret the information you get?

As Frank said, if you are reading Hollerith cards, how will you know whether the data is in 9-Left format, Chinese binary or something else? Then, once you have that data, what do you do with it?

It it a text file? Is it some kind of binary data file? Is it a program? What? Is it a digital video file? Is it MPEG? Is it raw binary data taken from a camera sensor? Even if you know that, how are you going to translate that data from that format to something you can use?

The only way to ensure that your data can be read in the future is to store a complete, working machine which can read that data back along with the data you want to archive. Then you'll have to include printed instructions on how to use it.

Again, the whole reason for this problem is because digital data isn't tangible. You can't see it. Even if you can see it how can you understand it? You can see a box of Hollerith cards and, in theory, you could manually read each card, one by one, and translate the punched code into symbols to type into a computer but that is mind-numbingly boring, tedious and fraught with possibility for error.

With film, you can see the image. You could still project it years from now but, if you couldn't find a projector, you could probably scan it and digitize it using current technology.

BTW, Manny, 50 years was a was nice round number. I bet you could play a 100 year old movie. Even so, you'll often have a hard time reading a computer file that's only ten years old.

While using film is not as convenient, I don't think "practical" is the correct way to characterize the difference. A thing can be practical for one purpose but not another. Digital video might be more practical for lazy movie executives but not necessarily so for theater owners who have to replace all the equipment in the complex.

Posted by Mark Ogden (Member # 43) on 02-06-2012, 04:16 AM:

quote:
Now is a great time to figure this out, Hollywood.

Congrats on that!

Go back and re-read the original article, which is addressed to smaller independent and documentary film-ma . . . oh, uh, excuse me, data stream makers, who probably haven't given the issue much thought.

Hollywood and the big players figured this out years ago,and the proof is on every screen that is showing Beauty and the Beast right now. This was a film that was produced on the old, long obsolete Disney CAPS system. They got rid of all that stuff years ago, but kept a few systems around to read archival material and were thus able to resurrect the original files and transfer them.

I tried to make the point years ago that anyone with important digital assets is obliged by common sense to either keep equipment around that can read them, or to migrate them to another format. The pros know this and are doing it. The key is the whole "active management" thing. Everything needs to be actively managed, film included. How many horror stories have been heard about film-makers going back to the vaults and finding their negatives in shit shape, too tightly wound, falling apart, mislabeled, mishandled? Look at the horrors that befell My Fair Lady, which we recently discussed in another thread. Here is a movie shot on 65mm that was in such shit condition that everything had to digitized in order to save it. Even the original b/w separations were screwed.

Anything film or digital that is handled with care, forethought, and common sense will outlive all of us.

Posted by Joe Redifer (Member # 3) on 02-06-2012, 04:53 AM:

quote: Randy Stankey
What if you had to read a hard drive with a SCSI interface or a PATA interface? Your computer has SATA. How are you going to get that data?

If you'd take care to read my ENTIRE posts, you'd know that I had already covered that.

Exhibit A:

quote: Joe Redifer
Likewise, they need to make lots and lots of copies of their files. When new storage media comes out, copy that shit over!

The bottom line is that they can't sit on their thumbs. If they do, they have nobody to blame but themselves. Technology isn't the enemy here, stupidity is. I still have files from 20 years ago on formats that weren't even comprehended back then. Doesn't seem to be a problem for me. But then again I just don't sit around passively and then later wonder why my stuff doesn't work.

Also, they make IDE/PATA/whatever adapters. I can read an EIDE drive (or PATA as you call it) on my SATA computer just fine. I can run SATA drives in my IDE PC. Crazy thing, technology.

Posted by Louis Bornwasser (Member # 3063) on 02-06-2012, 06:47 AM:

I guess they will just have to do what they have been doing: 35mm separations. Works good, lasts a long time. Louis

Posted by Mark J. Marshall (Member # 1409) on 02-06-2012, 07:09 AM:

quote: Mark Ogden
Hollywood and the big players figured this out years ago

I'm not convinced. Come see me in 10 years and we'll see where we're at. If you're right, I'll buy you a nice steak dinner.

Posted by Paul Gordon (Member # 3320) on 02-06-2012, 07:25 AM:

Its the small players we have to worry about... not so much the big studio films. Thats what the report is all about. In the appx there is a interview with one of the post crew for David Fincher films. He went digital early on. But check out how the film is saved after post.

Q: The rendered DI, the digital master that is the result of all your hard work, must
obviously be protected. How do you insure that?

PM: When a movie is completed, assuming it is finished as a digital intermediate, it becomes
standard operating procedure for us to make the following:

LTOs of the final rendered DI files – at least two sets
Negative made directly from the DI files – sometimes as many as six
Interpositives – often as many as four
Dupe negatives – many
Digital Cinema Package master
High Defi nition video master
Digital YCM 35mm separations
From the above elements are made:
Hundreds, if not thousands, of 35mm prints
Hundreds, if not thousands, of Digital Cinema Package files, although these are erased
at the end of a theatrical run
Many thousands of DVDs and Blu-ray discs
Dozens, if not hundreds, of TV broadcast masters in Hi Def and Standard Def

The point is that in our modern media-addicted world, it becomes nearly impossible to wipe a
movie’s existence off the face of the earth.

Posted by Martin McCaffery (Member # 37) on 02-06-2012, 09:24 AM:

Yeah, the small guys are most threatened, but I would not bet my life on Hollywood's preserving its history as a indication of enlightened self-interest.

Studios as corporate institutions may have an interest in preservation, but the suits that run the corporations are only interested in making as much money as they can as quickly as they can. That's why they have the bean counters to say they can't afford to maintain dual inventories, or strike new prints or whatever profit trimming expense that gets in their way.

Hollywood's history of preservation - with the exception of Disney - has been dismal, as the above post re: My Fair Lady shows. If they can't be trusted to preserve the big films, they certainly will not be interested in their lesser performing catalog.

Obviously, they have vaults and libraries and some preservation plans, but how much of a priority they are is still questionable.

Posted by Manny Knowles (Member # 1171) on 02-06-2012, 12:19 PM:

Randy, by "practical" I was referring to what we actually get on-screen, as opposed to what is theoretically possible with perfect prints and projection, which the industry was willing to abandon long ago. That's all.

Posted by David Buckley (Member # 2600) on 02-06-2012, 09:29 PM:

Clifford Stoll (known as the author of The Cuckoo's Egg) in his subsequent book, Silicon Snake Oil, notes:

The list of media types we can no longer read gets longer by the year.

Like Randy, all my data has moved with me as time has changed, but we're beginning to store really large amounts of data now...

Posted by Joe Redifer (Member # 3) on 02-07-2012, 02:21 AM:

I think you mean "like Joe". Anyway, the size of the data is irrelevant. NASA was foolish. They have absolutely no one or nothing to blame but themselves. Storing digital movies on film will do no good.

Also, I still use paper tape. Stuff is awesome. I used it all of the time to write the titles and format on my 35mm prints and have found many other uses for it as well.

Posted by Paul Gordon (Member # 3320) on 02-07-2012, 05:52 AM:

Regarding David's Article.

Most of those formats he lists are still available and can still be played back. 78's! lots of turn tables play 78's. 8mm, there are loads of projectors out there, and Kodak still makes super 8 film. Reel to Reel 1/4" thousands of playback decks exist. 2 inch Quad is getting hard to service but they are still around, and BTW a broadcast image from a quad looks great.

NASA should have just saved the info to a standard LP record. Even if all the record players in the world suddenly disappear I'm sure someone on the planet is smart enough to make one.

Posted by Mark Hajducki (Member # 1732) on 02-07-2012, 08:05 AM:

The data most at risk of being unreadable will be that stored in obscure file formats on equipment made in limited quantities.

The 1986 BBC Domesday Project was an example of this. Stored on laserdisk, accessed using a modified (Acorn) BBC Master computer.

quote:
Domesday Reloaded producer Alex Mansfield explained that transferring the data from Laserdisc to the web was problematic because none of the data was stored in recognisable file formats.

"It was pre-digital photography, so all those pictures are analogue, even though they are on a Laserdisc. There is no compression," said Mr Mansfield.

Each individual photograph, satellite image or map page was stored as a single frame of video on the disc.

While the system appears to function like a modern-day website, with pictures loading when they are clicked on, the playback head is constantly jumping between 50,000 video stills.

Each of these images had to be digitised from the original one inch video tape.

Domesday's many written articles proved easier to recover because they were created in a digital format.

Although, with no data storage capability, the thousands of pages of text had to be encoded and stored on one of the Laserdisc's spare audio tracks.

source

Posted by Randy Stankey (Member # 64) on 02-07-2012, 08:46 AM:

Joe, I think we're talking about the same concept but from different sides of the fence.

You were saying, "If data is copied in a timely manner."
I was saying, "If data is NOT copied in a timely manner."

If a new type of storage comes out, if the data is valuable, transfer it ASAP but real life shows that people do not take the time to do so and the OP's article suggests that most people are have not even considered it.

In David's article, had the scientists of the Pioneer mission stored their images on 5-inch glass lantern slides, they would still be accessible. Just pick them up and look at them. If you want to view them on a computer, just scan them. Further, I wonder whether digital data could be stored on film or slides using some kind of bar code. It might not be a convenient format but it is something that can be recovered in the future.

This is what I think the main point of the article is getting at. We don't know what kinds of data technology will exist in the future. Who knows? Maybe metal platter hard drives will still exist. If so, great! Just back up your data onto hard drives and you're safe for the foreseeable future.

Back in the 1970s, Hollerith cards were among the best technologies we had. It made sense to store data on cards but most people had no idea what would happen in the future. A warehouse full of punch cards is useless today, only 30 years later. However, we still have the first permanent photograph ever, taken by Joseph Niépce in 1826. It looks like crap because it wasn't a very good photo in the first place but we still have it and we can digitize and display it on computer screens.

I don't see the question as being about technology, per se. Some people are just making the assumption that digital storage is better. Others are saying film is better. The truth lies somewhere in between. Maybe making backups in both analog and digital is the answer? I don't know.

The main thing that people need to do is think ahead. Punch cards and magnetic tape are obsolete and, in many cases, unreadable but 100+ year old film is still readable but not easily accessible. Where's the middle ground? How will we carry that solution into the future?

We still have ancient records on Sumerian clay tablets and Egyptian papyrus scrolls but the people who made them, thousands of years ago, never gave any thought to those documents' longevity. Nowadays, we struggle to preserve our data for 50 or 100 years. Go figure... [Shrug]

Posted by Scott Norwood (Member # 30) on 02-07-2012, 08:55 AM:

To summarize:

Film will last longer than any digital format if kept on a shelf, but information is lost when it is duplicated. It is also a self-documenting format and its storage characteristics are well understood.

Digital data can be copied losslessly (assuming no format conversion or compression is involved), but must be copied regularly in order to preserve it due to short media and format life. Information on storage format needs to be maintained with the data. Long-term storage of digital data is an emerging field, which can be expected to experience rapid change over time.

Which is better depends on how much we trust the owners of the material to maintain it. The best option is probably to store information in both formats, although converting from one to the other will involve loss of information.

Posted by Steve Guttag (Member # 268) on 02-07-2012, 09:58 AM:

I think Scott hit the nail on the head in that analog storage is typically self-explanatory. One does not really have to figure out film or paper formats. With digital, you must not only know the medium but how the data is stored and what that data is!

When it comes to migration (moving from one digital format that may be obsolete to a current one) the problem becomes geometric. That is, you have the ENTIRE history of one's digital work to deal with. So if next year a new format that looks like it has legs to it becomes the standard, we will have to migrate say the last 5-years of things mastered in a DCP format...but moving ahead 20-years, now we will have 20-years of history to once again migrate to the then current format. 50-years away...it repeats...etc. Your list of things to migrate is constantly growning exponentially with each migration. As for film, if motion pictures are stored on film (or stills), then it only need come off that medium to get it to the current medium of the day on a case-by-case basis unless the one detects deterioration.

The chief reason that films were lost in the 20th Century (except for Disney) was a film's value dropped pretty fast after it was released. There was no home market until TV came along to the masses and no real revenue stream for those old titles until home-video hit the masses. Disney has titles that they could reissue every 7-10 years for the next generation...thus they knew to keep them in good shape...they would be perpetually valuable.

The archiving methods in the past were very cursory. They might make a separation copy of the movie but NEVER check to see if it was a good copy. That is how cheap the studios, historically, have been with titles that have done their thing on 1st run. One would think that after making a protection copy in separation or even just a dupe negative they would then immediately strike a print off that to ensure it was a good copy. One thing I hope they start doing more is when they do separations, do it on one negative (three sequential frames) rather than three reels. This will avoid the lost color but more importantly, ensure that the shrinkage or other possible distortions are uniform across the three colors. But again, before putting the separations in the vault, strike a check print off of them to ensure that you have something worth keeping. Spending 100s of millions of dollars and not spending 10s of thousands to ensure you have it forever is short-sighted.

Like with film, LPs or even cylinders are almost self-explanatory on how to get the information off of them...it makes the a good storage medium for longevity if not for accuracy. Remember too...all of this encyription business will only go to thwart future archivists from retrieving "lost data."

-Steve

Posted by Brian Guckian (Member # 1678) on 02-07-2012, 11:01 AM:

quote: Randy Stankey
I wonder whether digital data could be stored on film or slides using some kind of bar code.

In recent years ARRI and more recently Point.360 have developed systems which facilitate encoding "born-digital" material onto single-strip 35mm B/W film.

Details for the Point.360 system:

Point.360 Visionary Archive Technology

Posted by Scott Norwood (Member # 30) on 02-07-2012, 01:08 PM:

Recording digital data on film will solve the media life problem, but not the problem of format documentation or of reader hardware.

Microfilm is a good choice for long-term preservation of human-readable material (newspapers, text, etc.) for this reason, as the reader hardware consists of little more than a lens. Storing non-textual data, however, is a much more difficult problem.

One point in favor of film is that the people who run labs and archives are generally familiar with the idea of storing material for decades. IT people (I am one) generally don't consider long-term storage issues, and ten years is usually the outside limit on the lifespan of most "normal" data. If you were to go into a typical company and ask them to restore material from a backup made more than a decade ago, they probably would not be able to do it.

Posted by David Buckley (Member # 2600) on 02-07-2012, 01:13 PM:

quote: Randy Stankey
Back in the 1970s, Hollerith cards were among the best technologies we had.

These cards are still handy for taking notes; they just fit nicely in a shirt pocket. Not that I wear shirts with pockets much these days.

At work I have a couple of boxes of original IBM Hollerith cards, which should last me until my box is summoned. These cards have outlasted the IBM 360/44, which was what these cards were originally purchased for, by half a century.

More scarily perhaps, some of my colleauges who date back to that era still have boxes of cards in their offices that ran production back in the day. And it was only last year I persuaded the tape librarian to dump all the old 9 track tapes from the safe, these dating back to the IBM's sucessor, a Burroughs...

Books and other physical stuff last (almost) forever, and almost in spite of our efforts not to care for it. Digital data only lasts as long as we want it to and we are willing to put in the effort to keep it.

Posted by Brian Guckian (Member # 1678) on 02-07-2012, 01:40 PM:

quote: Scott Norwood
Recording digital data on film will solve the media life problem, but not the problem of format documentation or of reader hardware.

Interestingly this issue is addressed in both systems mentioned earlier (apologies as I don't have any links to the ARRI one). Here's an extract from the Point.360 Visionary Archive information, linked to above:

All the information needed to understand and recover Visionary Archive films are recorded in human readable form right at the beginning of each reel of film. If digital technology becomes obsolete, Visionary Archive films will be readable and recoverable

Posted by Paul Gordon (Member # 3320) on 02-07-2012, 02:00 PM:

Of course archiving to film is a moot point if no one is making film emulsion or stock anymore. How long does Kodak have? not a lot of other players make black and white motion picture stocks.

Posted by Joe Redifer (Member # 3) on 02-07-2012, 04:35 PM:

The best way to store stuff seems to be posting it on Facebook. Even after you delete it, it's still there forever and ever.

EDIT: This is my 12,000th post. I demand $1 from each of you.

Posted by Scott Norwood (Member # 30) on 02-07-2012, 04:53 PM:

Joe--don't use Facebook. Use Mega Upload. It's way less likely to disappear in the future. Oh, wait....