Film-Tech Forum ARCHIVE: Digital Archiving article from IEEE

This is topic Digital Archiving article from IEEE in forum Film-Yak at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=8;t=007607

Posted by Bill Brandenstein (Member # 7758) on 05-01-2017, 02:49 PM:

Saw this posted elsewhere and thought this crowd would have some interesting opinions about the subject, which seems to have been reported with an unusually high degree of factuality:
The Lost Picture Show

TEXT (sorry, no graphics or images):

The Lost Picture Show: Hollywood Archivists Can’t Outpace Obsolescence
Studios invested heavily in magnetic-tape storage for film archiving but now struggle to keep up with the technology

By Marty Perlmutter Posted 28 Apr 2017 | 15:00 GMT

When the renowned cinematographer Emmanuel Lubezki began planning to shoot the wilderness drama The Revenant, he decided that to capture the stark, frozen beauty of a Canadian winter, he would use no artificial light, instead relying on sunlight, moonlight, and fire. He also planned to use traditional film cameras for most of the shooting, reserving digital cameras for low-light scenes. He quickly realized, though, that film “didn’t have the sensitivity to capture the scenes we were trying to shoot, especially the things we shot at dawn and dusk,” as he told an interviewer.

The digital footage, by contrast, had no noise or graininess, and the equipment held up much better in the extreme cold. The crew soon switched over to digital cameras exclusively. “I felt this was my divorce from film—finally,” Lubezki said. The film, released in December 2015, earned him an Academy Award for cinematography two months later.

Lubezki’s late-breaking discovery of digital is one that other filmmakers the world over have been making since the first digital cameras came to market in the late 1990s. Back then, digital moviemaking was virtually unheard of; according to the producer and popular film blogger Stephen Follows, none of the top-grossing U.S. films in 2000 were recorded digitally.

These days, nearly all of the films from all of the major studios are shot and edited digitally. Like Lubezki, filmmakers have switched to digital because it allows a far greater range of special effects, filming conditions, and editing techniques. Directors no longer have to wait for film stock to be chemically processed in order to view it, and digital can substantially bring down costs compared with traditional film. Distribution of films is likewise entirely digital, feeding not only the digital cinema projectors in movie theaters but also the streaming video services run by the likes of Netflix and Hulu. The industry’s embrace of digital has been astonishingly rapid.

Digital technology has also radically altered the way that movies are preserved for posterity, but here the effect has been far less salutary. These days, the major studios and film archives largely rely on a magnetic tape storage technology known as LTO, or linear tape-open, to preserve motion pictures. When the format first emerged in the late 1990s, it seemed like a great solution. The first generation of cartridges held an impressive 100 gigabytes of uncompressed data; the latest, LTO-7, can hold 6 terabytes uncompressed and 15 TB compressed. Housed properly, the tapes can have a shelf life of 30 to 50 years. While LTO is not as long-lived as polyester film stock, which can last for a century or more in a cold, dry environment, it’s still pretty good.

The problem with LTO is obsolescence. Since the beginning, the technology has been on a Moore’s Law–like march that has resulted in a doubling in tape storage densities every 18 to 24 months. As each new generation of LTO comes to market, an older generation of LTO becomes obsolete. LTO manufacturers guarantee at most two generations of backward compatibility. What that means for film archivists with perhaps tens of thousands of LTO tapes on hand is that every few years they must invest millions of dollars in the latest format of tapes and drives and then migrate all the data on their older tapes—or risk losing access to the information altogether.

That costly, self-perpetuating cycle of data migration is why Dino Everett, film archivist for the University of Southern California, calls LTO “archive heroin—the first taste doesn’t cost much, but once you start, you can’t stop. And the habit is expensive.” As a result, Everett adds, a great deal of film and TV content that was “born digital,” even work that is only a few years old, now faces rapid extinction and, in the worst case, oblivion.

To understand how the movie studios and archives got into this predicament, it helps to know a little about what came before LTO. Up until the early 1950s, filmmakers shot on nitrate film stock, which turned out to be not just unstable but highly flammable. Over the years, entire studio collections went up in flames, sometimes accidentally and sometimes on purpose, to avoid the costs of storage. According to the Film Foundation, a nonprofit founded by director Martin Scorsese to restore and preserve important films, about half of the U.S. films made before 1950 have been lost, including an astounding 90 percent of those made before 1929.

It wasn’t just that film was difficult to preserve, however. Studios didn’t see any revenue potential in their past work. They made money by selling movie tickets; absent the kind of follow-on markets that exist today, long-term archiving didn’t make sense economically.

In the 1950s, nitrate film was eclipsed by more stable cellulose acetate “safety film” and polyester film, and it became practical for studios to start storing film reels. And so they did. The proliferation of television around the same time created a new market for film. Soon the studios came to view their archives not as an afterthought or a luxury but as a lucrative investment—and as an essential part of our collective cultural heritage, of course.

The question then became: What’s the best way to store a film? For decades, the studios took a “store and ignore” approach: Put the film reels on shelves, placed horizontally rather than vertically, at a constant cool temperature and 30 to 50 percent humidity. Ideally, they’d have redundant copies of each work in two or more of these climate-controlled vaults. Remarkably, the industry still uses film archiving, even for works that are born digital. A master copy of the finished piece will be rendered as yellow-cyan-magenta separations on black-and-white film and then preserved as traditional celluloid, on polyester film stock.

“We know how long film lasts,” says the USC archivist Everett. “And archives were designed to store things. They’re cool, they’re dry, and they have shelves. Put the film on the shelf, and it will play in a hundred years.”

One big problem with this approach is that to preserve the work, you must disturb it as little as possible. Dust, fingerprints, and scratches will obviously compromise the integrity of the film. Archive staff periodically check the stored masters for signs of degradation; occasionally, a master will be used to make a duplicate for public release, such as a showing at a repertory cinema or film festival. But otherwise, the archive remains pristine and off-limits. It’s like having a museum where none of the art is ever on display.

Maintaining such a facility isn’t cheap. And as chemical film stock becomes obsolete, along with the techniques used to create and manipulate it, relying on a film-based archive will only grow more difficult and more costly.

“The sad truth is that film images are ephemeral in nature, kept alive only by intensive effort,” David Walsh, the head of digital collections at London’s Imperial War Museum, has written. “Apart from anything else, if you are storing film in air-conditioned vaults or running digital mass-storage systems, your carbon footprint will be massive and may one day prove to be politically or practically unsustainable.”

The movie industry executives I interviewed would argue that the current system for digital archiving is already unsustainable. And yet when LTO storage first came along 20 years ago, it seemed to offer so much more than traditional film. Magnetic tape storage for computer data had been around since the 1950s, so it was considered a mature technology. LTO, as an open-standard alternative to proprietary magnetic tape storage, meant that companies wouldn’t be locked into a single vendor’s format; instead they could buy tape cartridges and tape drives from a variety of vendors, and the competition would keep costs down. Digital works could be kept in digital format. Tapes could be easily duplicated, and the data quickly accessed.

And manufacturers promised that the cartridges would last for 30 years or more. In an interview, Janet Lafleur, a product manager at Quantum Corp., which makes LTO cartridges and drives, said that LTO tape may still be “perfect” after 50 years. LTO came to be widely used for data backup in the corporate world, the sciences, and the military.

But the frequency of LTO upgrades has film archivists over a barrel. Already there have been seven generations of LTO in the 18 years of the product’s existence, and the LTO Consortium, which includes Hewlett Packard Enterprise, IBM, and Quantum, has a road map that specifies generations 8, 9, and 10. Given the short period of backward compatibility—just two generations—an LTO-5 cartridge, which can still be read on an LTO-7 drive, won’t be readable on an LTO-8 drive. So even if that tape is still free from defects in 30 or 50 years, all those gigabytes or terabytes of data will be worthless if you don’t also have a drive upon which to play it.

Steven Anastasi, vice president of global media archives and preservation services at Warner Bros., therefore puts the practical lifetime of an LTO cartridge at approximately 7 years. Before that time elapses, you must migrate to a newer generation of LTO because, of course, it takes time to move the data from one format to the next. While LTO data capacities have been steadily doubling, tape speeds have not kept up. The first generation, LTO-1, had a maximum transfer rate of 20 megabytes per second; LTO-7’s top rate is 750 MB/s. Then you need technicians to operate and troubleshoot the equipment and ensure that the migrated data is error free. Migrating a petabyte (a thousand terabytes) of data can take several months, says Anastasi.

And how much does it cost to migrate from one LTO format to the next? USC’s Everett cited a recent project to restore the 1948 classic The Red Shoes. “It was archived on LTO-3,” Everett says. “When LTO-5 came out, the quote was US $20,000 to $40,000 just to migrate it.” Now that the film is on LTO-5, it will soon have to be migrated again, to LTO-7.

For a large film archive, data migration costs can easily run into the millions. A single LTO-7 cartridge goes for about $115, so an archive that needs 50,000 new cartridges will have to shell out $5.75 million, or perhaps a little less with volume discounts. LTO drives aren’t cheap either. An autoloader for LTO-6 can be had for less than $3,000; an equivalent for LTO-7 is double that. And archivists are compelled to maintain and service each new generation of LTO drive along with preserving the LTO cartridges.

Lee Kline, technical director at Janus Films’ Criterion Collection, regards data migration as an unavoidable hassle: “Nobody wants to do it, but you have to.” Archivists like Kline at least have the budgets to maintain their digital films. Independent filmmakers, documentarians, and small TV producers don’t. These days, an estimated 75 percent of the films shown in U.S. theaters are considered independent. From a preservation standpoint, those digital works might as well be stored on flammable nitrate film.

Meanwhile, the motion-picture studios are churning out content at an ever-increasing rate. The head of digital archiving at one major studio, who asked not to be identified, told me that it costs about $20,000 a year to digitally store one feature film and related assets such as deleted scenes and trailers. All told, the digital components of a big-budget feature can total 350 TB. Storing a single episode of a high-end hour-long TV program can cost $12,000 per year. Major studios like Disney, NBCUniversal, Sony, and Warner each have archives of tens of thousands of TV episodes and features, and they’re adding new titles all the time.

Meanwhile, the use of higher-resolution digital cameras and 3D cameras has caused the amount of potentially archivable material to skyrocket. “We went from standard definition to HD and then from HD to UHD,” Peter Schade, NBCUniversal’s vice president of content management, said in an interview. Pixel resolutions have gone from 2K to 4K and soon, 8K, he adds. Codecs—the software used to compress and decompress digital video files—keep changing, as do the hardware and software for playback. “And the rate of change has escalated,” Schade says.

Computer-animation studios like Pixar have their own archiving issues. Part of the creative process in a feature-length animated film is developing the algorithms and other digital tools to render the images. It’s impossible to preserve those software assets in a traditional film vault or even on LTO tape, and so animation and visual effects studios have had to develop their own archival methods. Even so, the sheer pace of technological advancement means those digital tools become obsolete quickly, too.

When Pixar wanted to release its 2003 film Finding Nemo for Blu-ray 3D in 2012, the studio had to rerender the film to produce the 3D effects. The studio by then was no longer using the same animation software system, and it found that certain aspects of the original could not be emulated in its new software. The movement of seagrass, for instance, had been controlled by a random number generator, but there was no way to retrieve the original seed value for that generator. So animators manually replicated the plants’ movements frame by frame, a laborious process. The fact that the studio had lost access to its own film after less than a decade is a sobering commentary on the challenges of archiving computer-generated work.

Another problem for archivists is that digital camera technology has allowed productions to shoot essentially everything. In the past, the ratio of what’s shot to what’s eventually used for a feature film was typically 10 to 1. These days, says Warner archive chief Anastasi, films can go as high as 200 to 1. “On some sets, they’re simply not turning the camera off,” he says.

All that material will typically get saved and stored for a while. But at some point, somebody will have to decide how much of that excess really needs to be preserved for posterity. Given the huge expense of film preservation, archivists are being ruthless about what they choose to store. “There’s no way we can store it all,” says USC archivist Everett. “We’re just going to store the bare minimum.”

At Warner, Anastasi has taken a triage approach. Four years ago, when he took over the studio’s archives, he faced two distinct challenges: First he had to “stop the bleeding” by figuring out how to save those assets that were most vulnerable to being lost. Those on two-inch videotape, the medium of choice for network TV shows in the 1970s and 1980s, “were the most at risk. We captured that material on digital as uncompressed JPEG 2000 files.” That part of the triage is now nearly complete.

The second challenge was finding a way to affordably maintain the studio’s archive for more than a generation. He set the goal at “50-plus years.” He also decided that rather than operating an in-house archive, the problem would be better handled by outsourcing it. And so in 2014, Warner signed a long-term contract with USC Libraries to maintain the studio’s archives.

Sam Gustman, associate dean of the USC Libraries, says that the Warner archives are now part of 50 petabytes of archived data at USC, which also includes nearly 54,000 video interviews with Holocaust survivors gathered by the USC Shoah Foundation. For 20 years of storage, including power, supervision, and data migration every 3 years, USC charges $1,000 per terabyte, or $1,000,000 per petabyte. That works out to a relatively affordable $2.5 million per year for its current 50-PB holdings. It’s not a money-making business, Gustman adds.

The USC archive maintains four copies of each tape: Two are held at USC, one at Clemson University in South Carolina, and one at Charles University in Prague. The aim is to “touch every tape” every six months, using an automated system, Gustman explains. A robotic arm selects a tape from a rack and loads it into a reader, which plays it back while a computer checks for aberrations. Any tape that isn’t perfect is immediately trashed, and the archive makes a replacement from one of its remaining copies of the tape. The archive migrates to the latest version of LTO as it becomes available, so no tape is more than three years old.

Warner also began classifying its 8,000 feature films and 5,000 TV shows into two categories: those it will “manage”—that is, preserve for the long term—and those it deems “perishable.” Managed assets include not just the finished work but also marketing materials and some deleted scenes. Perishable material may include dailies for features or unused footage; it will be stored for some time in the archive but may not be migrated. To decide what’s perishable and what’s not, the studio considers things like how successful the film has been, how popular its stars are, and whether the film could have enduring (or cult) appeal.

The manage-or-perish scheme is by no means perfect, Anastasi admits, but he sees it as buying the studio a little time until a truly long-term digital storage technology comes along. If one ever does.

For now, he says, “We’ll keep it, and there’ll be time to rethink the strategy. But after 10 years, we can’t guarantee access” to any of the material that hasn’t been migrated to managed storage.

Everett says Warner’s strategic thinking about digital archiving is pioneering. All of the studios, he notes, “are in a realm where there is no policy.” Meanwhile, they’re waiting for an archival technology that is better than LTO. “Originally, we went all digital because it’s so much cheaper,” Everett notes. “But is it? Really? We haven’t solved the storage problem.”

If technology companies don’t come through with a long-term solution, it’s possible that humanity could lose a generation’s worth of filmmaking, or more. Here’s what that would mean. Literally tens of thousands of motion pictures, TV shows, and other works would just quietly cease to exist at some point in the foreseeable future. The cultural loss would be incalculable because these works have significance beyond their aesthetics and entertainment value. They are major markers of the creative life of our time.

Most of the archivists I spoke with remain—officially at least—optimistic that a good, sound, post-LTO solution will eventually emerge. But not everyone shares that view. The most chilling prediction I heard came from a top technician at Technicolor.

“There’s going to be a large dead period,” he told me, “from the late ’90s through 2020, where most media will be lost.”

This article appears in the May 2017 print issue as “The Lost Picture Show.”
About the Author

Marty Perlmutter is based in Southern California and has worked in interactive video and new media for four decades, including early work designing immersive technology hardware, building exhibits, and exploring artistic and commercial uses of image control.

Posted by Frank Cox (Member # 6258) on 05-01-2017, 04:06 PM:

I really don't get this. I suppose it ultimately comes down to scale, but I have files on this computer that I'm typing on that date to the early 80's and maybe a few that are even older than that. I keep them along with all of my other files (including the stuff that I did last year, last week and this morning) on a series of backups to various devices including a portable hard drive and a fileserver.

And I'm not in the business of archiving.

If I can do this with my own personal stuff and not have any trouble with it or even have to think about it very often since my backups are all fully automated, what's wrong with these folks who are supposed to be the experts and who are actually in the business?

Give me ten million dollars and I'll set up a nice datacenter. Give me twenty million and I'll set up two of them. These guys are working with even more money than that and it seems like they just can't do it?

Posted by Leo Enticknap (Member # 534) on 05-01-2017, 07:12 PM:

This is a magnetic tape read by helical scan transport problem, not a specifically digital problem. The same migration problems apply to analog videotapes on small-scale and short-lived formats (e.g. EIAJ, M-5).

The sad truth of the matter is that during the last decade, a big debate and product marketing war took place for the emerging big data archiving market, between hard disc based approaches (most of them based on RAID or MAID architecture) and tape-based ones, and a lot of the world’s media archives placed their bet on the wrong horse. At the time, short-term economies of scale favored tape. The LTO organization had a slick marketing operation and deals with two big hardware makers, so they grabbed a big chunk of the market. Then Moore’s Law on steroids went to work on hard drive technology, and is working at a barely credible pace now: this time last year, $200 would buy you a 2TB server grade drive; now, the same money will buy you 4-5TB. Flash memory is plummeting in price, and several new solid state storage methods are about to come to market, that show a lot of promise for the possibility of making big data storage so cheap and easy that it’ll be a non-issue.

That pace of improvement and cost reduction per TB never happened with tape. Tape-based transports are difficult and expensive to make, with thousands of tiny moving parts and semiconductors. Even for the widespread formats (e.g. Betacam and its successors that used the same form factor), they were never made in huge numbers, and DIY solutions for reading helical scan tapes are next to impossible. In contrast, a hard drive contains only 10-15 moving parts (if you consider the motor and bearing assembly to be a single unit), that are made by the billion. A flash memory SSD contains no moving parts.

So the good news, which this article doesn’t really touch on, is that archives sitting on vaults full of data tapes have a migration problem, but they probably only have a one-time migration problem. After that it becomes a gradual process, of the sort that the managers of big data operations such as Google, IBM and most of the world’s governments are very used to taking care of (as Frank points out), and which shouldn’t be a significant problem for smaller archives either: compared to the cost of keeping a film vault cool, it’ll be a drop in the ocean.

quote: Frank Cox
...what's wrong with these folks who are supposed to be the experts and who are actually in the business?

The fact that most of them are not really experts: most of the decision makers at the world's moving image archives, and especially the big, national nonprofit ones, come from an arts and humanities background, are still fixated on film, and tend to be reluctant to hire IT experts. They therefore have a huge problem coming to terms with the fact that, as you realized immediately, their organizations are in the process of transforming into big data archiving operations, no different in terms of the technological function they perform, from Google or IBM.

The author of this article seems to think that $1,000 to archive 1TB of data is somehow a good deal. Tapes in four separate locations, which have to be physically shipped between them ... verification in a drive every six months, with the cost in head hours that implies ... I can believe that it costs that much, but no way would any major commercial data archiving operation tolerate costs that high nowadays. Server farms based on huge RAID arrays, with backup copies of data transferred between locations automatically and online would be their approach.

When you consider that Amazon drive will store your data for a retail price of $120 per TB per year (and that's for a consumer and including a humongous profit margin for Amazon - imagine how cheaply they'd probably be able to do petabytes and on a 20-year contract!), it becomes clear that people trying to archive big data long-term using tapes and offline storage have been left well and truly in the last century on this one.

Posted by Scott Norwood (Member # 30) on 05-01-2017, 09:25 PM:

LTO is linear (it's in the name), not helical scan. All modern videotape formats are helical scan, but LTO is a data storage format only.

Tape is not entirely bad, and should certainly be a part of any backup and archive strategy.

Tape has the advantage of being really good at offline storage. Tapes sitting on a shelf with the write-protect lever engaged are not subject to fat-fingering (witness the recent Amazon S3 outage), software bugs, bad RAID controllers, and the like.

Disks are really only good for online storage, which provides faster (and non-linear) access, but is more prone to failure. LTO7 tape is specified with an error rate of 1 in 10^19 bits. Enterprise-grade disks are specified with an error rate of 1 in 10^16 bits. Yes, some of this is a nonissue with proper filesystems, RAID controllers, self-error-correcting file formats, and storage practices, but there is value to having a lower bit error rate.

I pretty much agree with the article in principle, though, that long-term storage of high-volume data is not a well-tested field, and will become expontentially more expensive over time, even as information is lost. One issue not discussed at length in the article is that of file formats. Recovering raw data does no good if the format is not documented and supported by available software.

Posted by Leo Enticknap (Member # 534) on 05-02-2017, 09:43 AM:

My apologies, yes, LTO uses static heads. This makes it all the more puzzling to me as to why LTO drives and tapes are so effing expensive (around $1,500 for a basic LTO-6 drive, and $20 for a 2.5TB tape).

Agreed that in theory, offline storage on linear media makes sense if access to the data is not needed very often. But even if you don't need always on/instant access, the cost differential between using ginormous RAIDs and tape-based storage for archiving big data must be small and diminishing. Larger capacity hard drives means less physical space (server racks, etc.), power and cooling needed per TB stored, less labor in maintaining the facility (swapping out drives as they fail), and the ability to move data between facilities (to store your separate location redundant backups) online rather than having to eat the cost of physical transportation.

Added to which, 3-D optical discs, holographic storage and other emerging data storage technologies seem to be on the verge of becoming available commercially, all of which offer the promise of driving down the cost and driving up the long-term reliability of archival data storage.

I think the guy quoted in the article who speculates that looking back in, say, a century's time, the period of 2000 to 2020 might be one in which an unusually high proportion of archival documents (not just movies, but anything else born digital you care to mention) were lost may be right, but I don't think the problem is going to be as dramatic as the article implies.

In any case, the basic problem is one of human nature, not the technology itself. It's no coincidence that of all the movies originated on nitrate, the ones that the critics raved about tend to have survived, but B-movies that disappeared without trace culturally, also tended to do so literally. In the digital realm, I too have files that I created on Microsoft Word 2.0 and originally saved on 5.25" floppies in the late '80s. These were then copied to 3.5" floppies, then to Zip cartridges (incidentally, I still have a working IDE Zip drive in one of my PCs, and last year recovered some files from a Zip disc written in 1998 for a friend of my wife's - the disc just went in and its entire contents read without any errors), then to internal hard drives, and now sit in my NAS server. The point is that I cared about them enough to want to preserve them, and the same principle applies to mega-archives, too.

Posted by Bill Brandenstein (Member # 7758) on 05-03-2017, 12:08 PM:

You gentlemen never disappoint in a great technical conversation.

Leo, I'm assuming the price of LTO drives is high because they're literally the last magnetic tape hardware being manufactured on earth, and certainly not in high quantities. And because they can charge whatever they want.

$20 per tape sounds about right, considering what we were paying for DTRS-format digital recording tape when Fujifilm pulled the plug a decade ago, and the fact that Sony still makes 16-bit-only tape for about $11 each. And that's only for about 5-8GB worth of data! Good tape, analog or digital, has never been cheap, and now that there's almost nobody in the world making it, volume discounts don't apply, even if they ever did.

Posted by David Buckley (Member # 2600) on 05-03-2017, 03:52 PM:

quote: Frank Cox
Give me ten million dollars and I'll set up a nice datacenter. Give me twenty million and I'll set up two of them. These guys are working with even more money than that and it seems like they just can't do it?

It is true, $10m will set up a nice (but not very big) datacentre. But it wont get you close to 50PB of storage. Our nice datacentres are probably in the order of that cost, and we're still in the hundreds of TB range. OK, we have stuff in there besides raw storage, but even so.

With data, longevity of storage and of format are issues. With scale, keeping access and needing to migrate technologies is the issue. At the tail end of the 1990s, I was involved in a project that wanted to acquire a few hundred TB per year of scanned images and have it available for a century, just like they did with paper. Upshot is they are - to this day, I believe - sticking with paper.

Posted by Bobby Henderson (Member # 840) on 05-03-2017, 04:15 PM:

Back in the early to mid 1990's I used a couple different tape back-up systems to store large amounts of data. They were a pain in the ass to use. I stopped using the tape drives once CD-R & RW drives and blank media became economical. Iomega Zip Drives were pretty convenient too. These days I have a growing collection of large capacity hard discs.

quote: Frank Cox
I really don't get this. I suppose it ultimately comes down to scale, but I have files on this computer that I'm typing on that date to the early 80's and maybe a few that are even older than that. I keep them along with all of my other files (including the stuff that I did last year, last week and this morning) on a series of backups to various devices including a portable hard drive and a fileserver.

I have a bunch of computer files dating back to the 1980's as well, such as Word Perfect files of school papers or the manuscripts to a couple of my father's books. I've had early CorelDRAW graphics files that will not open on any newer version of CorelDRAW. For anything earlier than CorelDRAW version 5, I've had to fire up an old 1990's era PC loaded with CorelDRAW 9 to open those files and save them forward a few versions. The artwork version migration process is not pain-free either. If you're missing fonts that's a problem. Different versions handled certain effects differently. Even simple things like flowing columns of paragraph text could vary from one version to the next. I have some art files in Macromedia Freehand format. Adobe bought Macromedia, killed Freehand and even removed Freehand import capability from Adobe Illustrator. So, again, I have to fire up this old PC to launch Freehand 9 and save that artwork in another, not dead format. Newer versions of the MacOS and Windows are not friendly at all to old versions of software. I could not install an old copy of CorelDRAW or Freehand on a newer version of Windows.

With movies it's an even more complicated nightmare. If a movie studio wanted to do a massive restoration of a big visual effects movie, like Terminator 2 or Jurassic Park they would have to re-do ALL of the visual effects work from scratch if there are any live action plates to use as an image source. Very little if any of the original CGI data from those projects is usable at all. Alias Power Animator and SoftImage were two applications used on those projects, both of which are now dead applications. And they ran on the Silicon Graphics IRIX OS, which is also dead.

Posted by Frank Cox (Member # 6258) on 05-03-2017, 05:07 PM:

There are easier ways to get things done than keeping obsolete hardware around and hoping that it doesn't break.

Just a couple of months back I set up a new POS/bookkeeping system for a local business here who was using an old DOS-based program that was running on a 1992 Compaq all-in-one with a dot matrix printer.

Today he's running the same program in exactly the same way on a cute little mini pc and a flat-screen monitor with a laser printer for his invoices and reports. He still has his entire business history available and since it's the same program there's nothing for him to re-learn how to do. He can put his full backup onto a flash drive.

The setup runs under dosemu on Linux using enscript to package the invoices and reports to send to the printer. I just had to write a three-line bash and sed script to massage the data a bit before sending it to enscript.

There are lots of ways to emulate old hardware that runs common stuff like DOS, Unix and old versions of Windows and the like. Dosemu, dosbox and VirtualBox are good places to start looking when you're faced with a task of that nature.

Posted by Carsten Kurz (Member # 5396) on 05-03-2017, 05:17 PM:

Tape drives and cartridges are rather low quantity items, and full systems are expensive. Prices and development of disc drives/capacities benefit from consumer market quantities and the fact that there is a huge need for datacenters, cloud storage, etc. As such, yes, maybe tape systems were the wrong route. With the cost of LTO drives being so high, I'm wondering why there is no market to maintain production capabilities or stock of previous generation drives. How much sense does it make to guarantee shelf lives of 30 or 50 years for the tapes, if you are forced to migrate tape formats every 5 years? I understand it is necessary to develop larger capacities, because new backups need more space, but why can't they supply means to keep the previous media in operation? Of course the equipment manufacturers are earning more money with the migration need.

- Carsten

Posted by Rick Raskin (Member # 1561) on 05-03-2017, 05:29 PM:

With all the need for cloud storage evolving its no surprise they are building a new data center just down the road from where I live. There is already another nearby off of I-66.

The Library of Congress has the Packard Campus in Culpeper (my niece works there) and maintains a collection of obsolete tape drives for the purpose of retrieving data from those once prominent but now obsolete formats.

LOC also has the "meat locker" where they store the copyright collection of 35mm films.

Posted by Frank Cox (Member # 6258) on 05-03-2017, 05:46 PM:

Several years back I remember reading an article about a guy who had a business where he had all sorts of obsolete computer hardware that he used to copy data that came in from everywhere. He was copying data for scientific labs and for police investigations and you-name-it.

Posted by Leo Enticknap (Member # 534) on 05-04-2017, 08:56 AM:

There are lots of "we migrate your obsolete formats" vendors out there. They range from operations like mine (one guy or gal operations, often part-time, out of a spare bedroom used as a small shop to do a very limited range of audio and videotape formats, the players for which are still widely available on the used market, and whose customers are mainly private individuals and small nonprofits on a limited budget), to professional lab/post house type operations, which tend to work for professional archiving operations, and charge accordingly.

As a general rule, the more short-lived or small-scale a given media format was, the more difficult and expensive it will be to migrate it now. The complexity and availability of the electronic or mechanical components is a factor, too. For example, I would charge you a more to digitize an Edison hill-and-dale phonograph disc than I would an acetate-on-glass recording from the 1940s (assuming that both are in good physical condition), because the correct stylus is more difficult and expensive to get hold of.

In the data field, I'm guessing that it would be a lot easier and cheaper to get a 5.25" floppy from the late 1980s copied, than an 8" floppy from a decade earlier. This is why you should really plan to do the migration before the format is totally obsolete, capturing the moment when equipment is being discarded in large quantities (this is where we seem to be with, say, Beta SP and DVCAM - players with thousands of good head hours left on them can be bought on Ebay for a few hundred). The beauty of RAID-based datacenters is that this migration can happen gradually and in an automated way, without the need for an army of technicians to plow through tens of thousands of individual physical objects in a single, time-limited project.

Posted by Connor Wilson (Member # 6088) on 07-23-2017, 10:37 AM:

This was a very interesting, cultivating read. It's kind of funny since I first read it last night, after watching a film that came in the mail from the Warner Archive Collection: Cats Don't Dance. Not only it was a film I've wanted to see again after 15 years, it touches on several themes related to that article.

a.) The film is set in an alternate 1939, which is likely home to several of the 50% of films lost before 1950. Thematically, it's also about the adversity that is faced by marginilized (in this case, animals) groups common in the Golden Age of Hollywood. The Shirley Temples of the world are already popular enough to be preserved while it took perserverance for the Hattie McDaniels to even get a supporting role (and I know Gone with the Wind is protected as well as her Oscar, but did you remember her in Hypnotized [1932]?). The year in which we first saw The Wizard of Oz is also the first and last time we saw The Good Old Days.

b.) It was released in 1997, which is in that "dead period" the Technicolor guy spoke about. Cats Don't Dance has gained a cult following throughout the years, hence why it got an MOD DVD release in widescreen after the original pan-and-scan DVD went out-of-print. The end credits indicate that Cats Don't Dance was inked digitally, similar to Disney's CAPS system, but the transfer appears to have been sourced from a 35mm interpositive, and not the digital files (if they still exist, that is). Disney has made direct-from-digital transfers of their 1990s Renaissance period that used the CAPS system except for The Rescuers Down Under (which used CAPS but was transferred from film) and The Little Mermaid which didn't use it. It's also important to note that the direct-from-digital transfers are altered on DVD and Blu-ray from their appearance theatrically. It's almost like a Finding Nemo situation but slight revisionism is embraced, rather than trying to mimic its original look. Also, considering Cats Don't Dance was an archival release and hardly a restoration, it might have been too expensive to make a new master from the digital files.

Actually, I remember someone on the Blu-ray.com forum considering that a thorough restoration of Star Wars: Episode I - The Phantom Menace (1999), especially the original theatrical cut, would be laborous and expensive, moreso than restoring the unaltered original trilogy of Star Wars. Because so much of the film has been manipulated in the digital domain, between shooting 35mm, "digital directing," the visual effects, recording those digitally-touched shots to tape, and then back to 35mm internegative, so much is needed to be done to have it acceptable for even a 4K Blu-ray master. I suppose some of this was done for its Blu-ray release, but the film has been filtered through horrible DVNR that I would assume it's mostly a 35mm IP scan that had been filtered excessively to match the aesthetics of the digitally-shot, rest of the prequels, and comped with a CGI Yoda. Furthermore, I don't know if those original elements are on at latest, obsolete LTO tapes, knowing Georgie boy.

I also wanted to talk about another artifact from the theoretical "dead period." A logo. A computer-animated Feature Presentation ident from National Amusements (now Showcase Cinemas) that ran in their theaters from at least 2000 to 2008 at Showcase locations, and was continued until around 2010 for NA cinemas that didn't bear the Showcase name. The only known surviving copy of it is a camrip of a lost, illegal camrip of Step Up 2: The Streets that I noticed was online back in early 2008, then went missing a week later. The uploader of that YouTube video wouldn't tell us exactly where he found it, but the Step Up camrip had been found, in a moment in time, and its whereabouts are unknown yet. It might just end up with that low-quality phone cam rip being the only copy of the ident in existence, being that its last archived LTO tape is likely more than a decade obsolete. Likely, a 35mm IN or IP of that ident might still exist somewhere, but it isn't known. and the ident is too outdated to be put out in the public again, due to inconsistent and dated branding, and maybe legal issues. We've actually tried contacting people who could have made the ident, but nobody said they did. We might be already living in the dead period, in that respect.

Posted by Scott Norwood (Member # 30) on 07-23-2017, 10:53 AM:

SW Ep. 1 would not have originally existed on LTO. The LTO format did not exist in 1999. The common tape format in use at the time would have been DLT.

DLT hardware (used, not new) still exists and those tapes are generally readable now, although the underlying tape format would need to be something like GNU tar or some other common format in order to make use of them without extensive software archaeology.

Posted by William Kucharski (Member # 7246) on 08-03-2017, 08:53 PM:

quote: Frank Cox
There are easier ways to get things done than keeping obsolete hardware around and hoping that it doesn't break.

Auto garage in Poland run on Commodore 64
Commodore Amiga runs HVAC for Grand Rapids, MI School District

Posted by Leo Enticknap (Member # 534) on 08-03-2017, 09:10 PM:

[url= http://"http://www.npr.org/sections/thetwo-way/2016/05/26/479588478/report-u-s-nuclear-system-relies-on-outdated-technology-such-as-floppy-disks"]Nuclear missiles run on 8" floppy discs[/url].

Posted by System Notices (Member # 2357) on 09-04-2018, 04:41 AM:

It has been 396 days since the last post.

Posted by Leo Enticknap (Member # 534) on 09-04-2018, 07:57 AM:

quote: Connor Wilson
The year in which we first saw The Wizard of Oz is also the first and last time we saw The Good Old Days.

I couldn't have put it better. You've hit the nail squarely on the head as to the real reason why movies (and other cultural artifacts, for that matter) disappear. It is not, fundamentally, because the technology makes it difficult to preserve them. It is because they were not considered worth saving, either ever and at all, or until it was too late.

A great example of this is Beyond the Rocks. The movie is a mainstream Hollywood genre pic from the early 1920s. It got lukewarm reviews, and did OK, but not great, at the box office. It then vanished without trace, until a print was discovered in a barn in The Netherlands in 2003, which was used to restore and re-release the film in a blaze of global "film archiving event of the century!" publicity three years later.

It was a ready made project, that checked all the boxes needed to make the work of film restorers into a big media event:

1 - The film had two big stars in it (Swanson and Valentino)
2 - There was no evidence of it having been seen in any way, shape or form, for over 80 years after its original release.
3 - The rediscovered print was complete and most of it was in good condition
4 - It was rediscovered in a very unlikely location and under bizarre circumstances
5 - Scanning and digital film restoration was in its infancy, and this was one of the first projects, if not the first, in which a Hollywood feature was restored using this technology.
6 - The archive that did the restoration was very good at press relations and PR.

There was just one, tinsy little problem. The movie is fucking abysmal. It is truly, truly bad. I have probably seen around 100 Hollywood features made between the end of WWI and the conversion to sound, and this is easily the worst. The script is a crude, cliche-ridden hack job from a basically unfiilmable (at least, without spoken dialogue) novel. The acting is totally ham - exaggerated gestures of the sort you'd expect to see in movies made a decade earlier, before Griffith invented the cut to a close-up. The problem is that Sam Wood was using cuts to close-ups, but the actors (especially the supporting cast) didn't get it. The screen was filled with them making faces like clowns. Camera movement, or anything else even remotely visually interesting? Forget it. This film is so incompetently conceived and executed that it actually provides unintentional amusement value.

However, in the media event that accompanied this restoration, no-one was really allowed to point this out. When I did so, discussing it with a few people after a screening at an AMIA conference, I was rebuked for undermining the achievement of the film archiving movement, etc. etc. Actually, I'd consider it a bigger achievement than if they'd found the lost footage from Greed, or The Mountain Eagle, because this actually gives us a more accurate idea of what a typical American going to the movies in 1922 would have been found themselves watching than anything we had available previously. Most of the films that the professors idolize and rave about now did not reach a huge audience when they were first made, and the ones that are lost are only lamented now because those professors, and the archivists they taught, are making a big deal out of them.