Film-Tech Forum ARCHIVE: Need another reason to watch your movies before showing them to customers?

This is topic Need another reason to watch your movies before showing them to customers? in forum Ground Level at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=5;t=003051

Posted by Frank Cox (Member # 6258) on 08-25-2017, 06:57 PM:

Yesterday afternoon I was doing my regular pre-screening of tonight's movie and there was a very short (less than a second) stutter in the picture and sound at one point in the movie.

I paused the movie and replayed that portion a number of times but couldn't make it happen again. The server control panel still says "Raid online", too.

After sending the logs to GDC for evaluation it's been determined that one of the hard drives needs replacement.

And it occurs to me that a very short-duration issue that apparently cleared itself up right away would never cause one of my customers to come out of the auditorium and tell me about it. Had I not been watching that movie myself I would never have known there was any problem. And I would really like to know about these things before it turns into a show-stopper.

So here's yet another reason to watch your movies before showing them to the public. You'll find out about issues that the nobody in the audience would bother to tell you about.

Posted by Justin Hamaker (Member # 2165) on 08-25-2017, 07:26 PM:

I completely agree that it's good to sit and watch a movie on each screen from time to time. It allows you to check to see if screen files need tweaking, see if you have a speaker going bad, or any of a variety of other issues. But it doesn't necessarily do anything for a RAID issue. Those issues can come up at any time, so just because you've pre-screened the movie, doesn't mean you are safe from those issue when customers are present.

Posted by Frank Cox (Member # 6258) on 08-25-2017, 07:33 PM:

That's true, but this may or may not have been an "unreported issue" for the past week. I know that it wasn't a problem a week ago when I last watched the movie from beginning to end.

My point is that I'll find out about any unreported issues on a weekly basis. That may not be hard drive issues; perhaps an intermittent issue with the sound could manifest itself like that too. Doing a full inspection of the movie every week gives me a chance to find out about any issues that may not have been reported by the public over the previous week.

Posted by Mike Blakesley (Member # 26) on 08-25-2017, 11:53 PM:

Those RAID problems do indeed crop up anytime without warning. It's happened to me 3x now in the 7 years since we went to digital. Since I can hear the sound from the office, my ears are pretty tuned-in to anything going wrong with the sound. The RAID problem always shows up with one little two-second (or less) glitch. It may or may not repeat in the same spot. If you're lucky, it'll happen in a trailer.

When that happens, might as well go ahead and replace the hard drives because even if the problem doesn't recur right away, it WILL come back and will just get worse, until (at least in our case) within 2 or 3 days it will make your movie unwatchable. No amount of rebooting, reloading content or anything else will fix it, in my experience. And most maddening, the system will still show the drives as healthy!

Best advice is to keep a spare set of drives on hand.

Posted by Stephan Shelley (Member # 8587) on 08-31-2017, 08:55 PM:

Good luck doing that with the rerelease of Close Encounters of the 3rd Kind. 7pm show keys unlocked 1 hour before and we were on screen with another feature till 6:30pm.

Posted by Steve Guttag (Member # 268) on 09-01-2017, 06:16 AM:

Yes, because a 40-year old movie has to be kept under lock and key! I wonder how it might end this time! [Roll Eyes]

Posted by Stephan Shelley (Member # 8587) on 09-01-2017, 08:01 PM:

Must be for that preshow stuff they added. Lots of secrets told.

Posted by Marcel Birgelen (Member # 6801) on 09-04-2017, 07:59 AM:

Hollywood just has no clue about day to day operations. You can't just dump some content somewhere and expect it to play flawlessly the first run.

Regarding those RAID glitches. It is pretty unfortunate they still need to happen. Seemingly only the most expensive storage systems know how to handle those kind of situations. In case a RAID array with parity, instead of waiting for the lagging disk to return the data, the controller or software RAID should immediately reconstruct it from parity data and flag the affected sector for further inspection during a background operation. Once too many of those time-outs on a single disk occur, it should flag that disk for replacement, before permanent failure occurs. That way, the playback can continue uninterrupted and the operator gets informed about possible issues with a disk.

Like I indicated, those systems do exist, but are usually reserved for enterprise grade storage appliances. Apparently, cinema equipment isn't deemed to be "enterprise grade".

Posted by Mike Blakesley (Member # 26) on 09-04-2017, 03:31 PM:

quote: Marcel Birgelen
In case a RAID array with parity, instead of waiting for the lagging disk to return the data, the controller or software RAID should immediately reconstruct it from parity data and flag the affected sector for further inspection during a background operation. Once too many of those time-outs on a single disk occur, it should flag that disk for replacement, before permanent failure occurs. That way, the playback can continue uninterrupted and the operator gets informed about possible issues with a disk.

I thought that was the whole POINT of a RAID...?

Apparently having a RAID only protects from catastrophic disk failures.

Posted by Frank Cox (Member # 6258) on 09-04-2017, 04:38 PM:

As far as I know, if a hard drive is dying in a GDC server the only way to know that is when the picture and sound start to stutter, and the only way to find out which drive is at fault is to send the logs to GDC for inspection.

Posted by Marcel Birgelen (Member # 6801) on 09-04-2017, 05:29 PM:

The problem here is that off-the-shelf RAID systems are usually more tuned to data consistency than to continuous throughput.

For normal business and database applications, you don't want to give up on a disk, just because of an occasional timeout. A timeout could be caused by the disk remapping a bad sector to a spare one, for example.

If this would've been a normal business application, the impact would be rather minimal, maybe you'd experience a few seconds of non-responsiveness. But for real-time applications, like the streaming of a movie for playout, those hiccups can be killing.

To fix this, the manufacturers like Dolby/GDC/etc. would need to alter the RAID implementation or replace those off-the-shelf solutions with one of those expensive storage manufacturers. Obviously they're not really keen on doing that, it's also not really their core business anyway.

You can already see where it's heading though. Integrated Media Servers are becoming the norm now for new installs, they usually don't have a lot of on-board storage. So, handling storage the "correct" way will become the responsibility of your NAS supplier.

Posted by Mike Blakesley (Member # 26) on 09-04-2017, 07:42 PM:

quote: Frank Cox
As far as I know, if a hard drive is dying in a GDC server the only way to know that is when the picture and sound start to stutter, and the only way to find out which drive is at fault is to send the logs to GDC for inspection.

You're right except that last part...even after it starts to stutter the logs will show everything as being "A-OK." The on-screen report will show the drives as "healthy" while your picture and sound are hiccuping like a drunk on a Saturday night.

The last time it happened here, they thought the trouble was in the RAID box (not the drives.) So we swapped out the box, the trouble continued. I had a spare set of drives on hand and swapped those in, problem solved.

Posted by Steve Guttag (Member # 268) on 09-05-2017, 07:03 AM:

Generally, when I see hiccups on screen I look to the SMART data on the RAID. You can do this on the GDC if you have Administrative privileges (you get there from the Control Panel). Check each drive and look at item 5, reallocated sectors and for the number of errors showing on that drive. It isn't a guarantee but if you see double-digit reallocated sectors, odds are, there have been visible issues on screen due to that drive. If you do have some reallocated sectors, keep an eye on it and see how fast they are rising. More than one a month, I'd yank the drive.

Posted by Scott Jentsch (Member # 1681) on 09-07-2017, 10:16 AM:

What kind of drives are going into these RAID systems? Are they high-grade NAS drives, mid-grade NAS drives, or just consumer desktop drives?

Posted by Steve Guttag (Member # 268) on 09-07-2017, 10:41 AM:

All of the server companies have their "approved" drives. They are all going to be "Enterprise" grade drives (desktop drives can be a disaster in a RAID because they will try to re-read things figuring they are the only means of getting the data). By and large, HGST (Hitachi) seem to be the favored drives in most of the applications but definitely not all. Often the manufacturers will also list those by Western Digital and Seagate.

On the 2.5" drives (used by the IMS1000/2000, GDC's PSD system and Barco's ICMP) they are even more fussy about the drives since often 2.5" drives are not used in arrays but by laptop type PCs.

Posted by Marcel Birgelen (Member # 6801) on 09-07-2017, 10:52 AM:

Usually, you should use the disks specified by the manufacturer. When retrofitting out-of-waranty units, I tend to look at the Backblaze statistics and see if they have statistics regarding the target sizes and form factors.

"Enterprise grade" disks are not always a recipe for success. You should stay away from desktop disks though, not because the disks themselves are necessarily more unreliable, but because the power-saving features in such disk interfere with RAID operations.