Film-Tech Forum ARCHIVE: Recovering Deleted Content

This is topic Recovering Deleted Content in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=001492

Posted by Steve R Pike (Member # 6238) on 05-04-2013, 11:18 AM:

Hi, this is a question I probably know the answer to, but worth an ask!

I deleted a feature yesterday off our server not realising we have got it again in two days time (Doh!). Unfortunately as it was an art-house film the distributors collected the drive earlier in the week so I couldn't re-ingest it right away.

Thankfully another local cinema is showing the same film and so we have borrowed their drive!

My question is, if I accidently deleted a feature from the Cinelister, can it be recovered or is it a case of once you confirm deletion it has gone forever?

We have a Doremi Cinelister system with a Christie CP2000 projector.

Like I say, probably know the answer - but maybe someone knows otherwise? [thumbsup]

Posted by Randy Stankey (Member # 64) on 05-04-2013, 12:13 PM:

My guess is that it is like any other computer system.

Deleting files from a hard drive does not necessarily erase the data. It only releases the sectors on the drive that were allocated to those files for reuse. Therefore, the data isn't really gone. It is just "hidden," so to speak. There is a caveat, however...

Because those sectors on the drive are released, the operating system can reuse them at any time. When that happens, the data will be overwritten and lost forever. While it is possible to use some sort of software to trace the locations of all the data on the system, if another file was allocated to those sectors, the recovery software will hit a dead end.

Further, I am guessing that, because a digital cinema server uses a RAID system, file recovery is likely to be even more complicated and difficult than usual. The way I understand, RAID can reallocate data at virtually any time. Recovering files across multiple discrete drives would be like searching for a needle in a haystack.

Theoretically, if you accidentally deleted a file and you stopped everything you were doing then immediately went about trying to recover the data, it might be possible but it wouldn't be easy. The longer you wait before you try to recover, the smaller your chances become.

In theory, maybe. In practical reality, fat chance.

Posted by Bruce McGee (Member # 131) on 05-04-2013, 12:51 PM:

I had several occasions for the distributor to try to collect the hard drives before we were finished showing the feature.

The corporate policy on this was to hold the drive until the last showing played. Period.

I got some nasty emails from a distributor of orange drive cases more than once over their precious drives.

Posted by Steve R Pike (Member # 6238) on 05-06-2013, 10:56 AM:

Thanks Guys - I thought that might be the case.

99.9% of the time we have the drive on site, and keep it onsite for the run (in case of any unforseen issues that requires a re-ingest), but sods law this film had been requested back.

We get some distributors send the drive to us 2 months before the screening date and expect us to ingest and send the drive back. We only have a limited amount of server space and, as an independent arts centre, we have a high turnover of films each week.

Posted by Dennis Benjamin (Member # 1137) on 05-06-2013, 11:25 AM:

The amount and time and effort that would be involved in recovering a file that size off of a RAID server - would be way too much work.

You should always have the hard drive on hand for any film that you are playing. Period.

Posted by Gavin Lewarne (Member # 5397) on 05-06-2013, 11:27 AM:

Could you do this if the drive has been requested back......ingest then copy to a spare drive?

Posted by Marcel Birgelen (Member # 6801) on 05-06-2013, 11:57 AM:

While I would really advise against it, you could load a program like "extundelete" onto your Doremi server and try to recover files from there.

Still, I guess you need to move it to some kind of external harddisk and re-ingest it afterwards to be able to play it.

Posted by Carsten Kurz (Member # 5396) on 05-07-2013, 11:24 AM:

Even using extundelete, this DCP is not residing on the server as a simple file. You would need to know the UUIDs and spread locations of all files involved (would be dozends). Too much hassle, too much of a risc. Unless Doremi would have it's own higher level of undelete option that could cover all these issues. But on a system that is in constant use, I doubt that the free data chunks are still available and consistent.

A local backup is always the better choice to preprare for issues like this.

Setup a very basic machine with one or two 3GB drives, copy distribution drives on it before you send them back.

- Carsten

Posted by Randy Stankey (Member # 64) on 05-07-2013, 12:14 PM:

I was thinking along similar lines.

What if you got a hard drive duplicator and several blank drives?
Just ingest the movie then, before putting it back into the case, stick it in the duplicator.

Store a bunch of drives in a cabinet and build a library. If you ever have a situation where you need to restore a movie you'll always have a backup.

Do those drives really need to be enterprise class?
If not, you could buy hard 2TB hard drives for ~$100.

Posted by Carsten Kurz (Member # 5396) on 05-07-2013, 02:39 PM:

No, you can use the cheapest drives available. This is not archival grade work - the stuff is usually still available on your server, the distribution drive, and at various other locations.

- Carsten

Posted by Randy Stankey (Member # 64) on 05-07-2013, 03:02 PM:

So,then, one could buy a desktop duplicator that looks like an iPod dock for about $100 or $200 and a couple-three cheapo drives for $100 apiece. For about 500 clams you could have yourself backed up six ways until next Tuesday.

If you have a drive crash on your RAID array, could you use one of those drives to keep you going in a pinch until you can order an enterprise class drive to replace the bad one? That could potentially keep you going if you have a crash on a busy weekend.

Posted by Marcel Birgelen (Member # 6801) on 05-07-2013, 04:15 PM:

quote: Carsten Kurz
Even using extundelete, this DCP is not residing on the server as a simple file. You would need to know the UUIDs and spread locations of all files involved (would be dozends). Too much hassle, too much of a risc. Unless Doremi would have it's own higher level of undelete option that could cover all these issues. But on a system that is in constant use, I doubt that the free data chunks are still available and consistent.

I guess that, if you don't ingress another movie, the actual files will still be very much intact. But still, you will probably still have a hard time reconstructing the original DCP.

Posted by Monte L Fullmer (Member # 2797) on 05-09-2013, 05:09 PM:

There was an old program built for win95 called "Directory Snoop" and a shareware program that would rebuild the destroyed first character in the file string so the file then could be saved .. a form of "undelete" .. and worked quite well.

Be a gamble if this program can recover a file being saved in a RAID system, yet could be a miracle if it could.

And this program still works with Win 7 in recovering deleted files, along with having macros that one could do a HDD wipe in one pass or many.

Posted by Randy Stankey (Member # 64) on 05-09-2013, 06:34 PM:

Correct me if I am wrong but it is my understanding that a RAID system can move data around on the disk array at any time, without assistance from the operating system.

If my understanding is correct, that means that data blocks on the physical drive could be overwritten at any moment. The longer the system runs, the smaller the chances that any given block of data will remain intact.

Complicating this is the fact that you have huge, multi-terabyte files spread over many physical locations across multiple drives. Tracing the extent of those files is a daunting undertaking, at best, and probably a futile one under practical conditions.

In theory it could be done but, in reality, I imagine it would be like trying to isolate a thimble full of water in an ocean.

My advice would be to back up your data in the first place.

Posted by Marcel Birgelen (Member # 6801) on 05-10-2013, 01:15 AM:

There are multiple levels at work here:

1) RAID: It doesn't matter very much if it is software or hardware or if it is RAID1, 5, 6 or whatever. RAID itself is pretty much transparent to the filesystem. If it is hardware RAID, it offers a big container that mostly looks like a single disk. If it is software RAID, a piece of software does essentially the same, but to most parts of the operating system, it is still one large disk.

RAID usually doesn't move stuff around. The most popular RAID formats are RAID1 and RAID5. RAID1 is nothing more than a big mirror, everything that's been written to Disk A is written to Disk B, to the exact same location (as long as the controller is concerned). RAID5 uses striping, in short, if you write data to a RAID5 array, the data gets evenly spread over your disks. To every chunk of data, a parity information is added. If a disk fails and the parity data gets lost, your data is still intact, if a disk fails and a part of the data chunk is lost, it can be restored using the parity information.

In normal operation, RAID doesn't move around stuff, even if a disk fails, data isn't moved around. RAID is also not aware of any files, it just sees "blocks of data". The only reason to actually move data around, would be in the event of an expansion of the RAID container itself and that almost never happens.

But even if RAID would be constantly moving stuff around, that wouldn't matter a layer up. Because RAID is not aware of any filesystem specifics, it can't just destroy information. Even if it moves stuff around, to the OS/FS layer, it would still be the same data.

2) The actual hard disk itself. A modern hard disk is primarily a big block of data storage to the controller. Most of the lower-level stuff is completely transparent. A hard disk might actually move stuff around, if a sector becomes bad, but that's also transparent to the controller, the hard disk keeps track of this itself.

3) The filesystem. That's the layer that implements the actual files and access to those files. There are many filesystems around, but the most common ones on modern Linux based systems are EXT3 and EXT4. EXT3 and EXT4 also don't happen to move stuff around very much, unless you specifically ask for it.

If you delete a file from an EXT3 filesystem, then you're just deleting the inode for that particular file. The data itself does not get removed. The inode stores, amongst a few other things, the locations of the blocks on the disk/RAID where the file is stored. It is a "virtual" location, as it doesn't directly match with the location of a physical disk, it matches with a location given by the RAID itself.

If you run a specific "undelete" utility, that utility will scan your filesystem at a low level and search for blocks that look like they contained a file, but do not have an active inode. That way, deleted files can often be restored.

There obviously is a catch: The blocks that were previously occupied by your deleted file, are now once again available for allocation. So the operating system can use those blocks for new files or growing, existing files.

So, the longer your system is running and the more data you either changed, moved around or added, the bigger the chance will be that at least one of the blocks of your deleted files will be overwritten with new data. At that point, your data becomes at least partly unrecoverable.

There are a few modern filesystems like ZFS that can improve file and "data recoverability" by using "copy-on-write" techniques. That way, as long as there is space left, data essentially is never overwritten.

Regarding the recovery of lost DCPs, there is, as Carsten has put it, another layer of complexity. It's a bit more difficult as restoring an accidentally deleted Word or Excel file. The DCPs are probably not just copied verbatim into a single directory, as they appeared on the original disk. To do a successful undelete, you also need to know some internal specifics off your playback server.

Posted by Randy Stankey (Member # 64) on 05-10-2013, 09:46 AM:

Okay. I think I understand.

A RAID system CAN move data around but, unless the system is reconfigured or it needs to recover from a problem, it doesn't happen in normal operation.

I had the silly idea that RAID did something like a continuous defrag or, at least, did so periodically outside control of the main computer's operating system.

Thanks for setting that straight. [Smile]

Posted by Marcel Birgelen (Member # 6801) on 05-10-2013, 08:01 PM:

Defragmentation would happen at the filesystem level, not at the RAID level. And even if your RAID would do some kind of defragmentation (which would be rather stupid [Wink]

), it wouldn't really matter for the OS layer, as the bits in the RAID container as represented to the OS would not change.

It is actually quite uncommon to defragment modern filesystems, because most of them are designed to limit fragmentation. It is not always useless, but in a DCI cinema server, it would be. Since you're largely working with large files, the fragmentation that builds up over time would still be so minimal, there would be no real perceivable performance increase from a filesystem defrag.