Film-Tech Forum ARCHIVE: Lost a show / Learned a lesson

This is topic Lost a show / Learned a lesson in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=001027

Posted by Mike Blakesley (Member # 26) on 02-12-2012, 08:38 PM:

Well -- tonight we are going to lose our 9:00 show.

We're playing 2 different movies this week. On Friday night, the second show (on which I had done the standard QC check of running the first and last couple of minutes) started out fine, but about 5 minutes into the feature, the playback suddenly just stopped and went back to the start of the program. (Not the start of the feature but the very beginning of the trailers.) I caught this and was able to pause and skip ahead to approximately the right spot in the feature and things went on as normal.

I figured this had just been a freak thing. I attributed it to a power glitch or some other weirdness. So, I let it go.

Then last night, the same thing happened again in the exact same spot in the movie. So, I figured, the feature file must be corrupt for some reason. So I planned to come in this morning and re-load it.

I came in at about 10:00 this morning, started the re-load process and went about my day. When I got here tonight to start the evening's shows, the status report on the server showed the re-load was only at 31%!

I thought maybe the server could use a reboot, so I stopped the download and rebooted the server, and started the download again. It skipped up to 31% right away, then stayed there for about 10 or 12 minutes before it finally went to 32%.

Normally it only takes about 20 - 25 minutes to load a whole movie.

I'm sure the problem must be in the Deluxe feature hard drive, because the other movie loaded just fine in the normal manner.

I didn't catch this slow-loading drive on the original load on Wednesday because I loaded movie #1, then started movie #2 loading and went home.

So.....anyway, lessons learned: Verify how fast a drive is loading before leaving for the night...if it's a crawler, suspect a problem. And, maybe that full pre-screen is a good idea after all.

At least this movie isn't exactly a blockbuster so we probably will only have a few people for tonight...passes for everyone!

Posted by John Wilson (Member # 269) on 02-12-2012, 10:23 PM:

George Lucas will now come and hunt you down.

Posted by Frank Cox (Member # 6258) on 02-12-2012, 10:36 PM:

Sounds like a bad hard drive to me. I had one of those a couple of months ago. Started loading The Thing, got to 31% and said "Exception". End of the line. I had to get another drive sent to me. The new drive didn't arrive until Saturday morning so I didn't have a show that Friday night.

Ever since I got my digital setup I have sat down and watched every minute of every movie I've played before playing it for the public.

With film you can see what you have -- is the film actually present, are all of the reels here, is it assembled in the right order, is it chopped up, is it scratched. With digital you get none of that. As far as I can see, the only way to know what you've really got is to watch it from start to finish. And I'd rather "waste an afternoon" (or late night) and watch the movie that get skunked when I've got a room full of customers.

Posted by Mike Blakesley (Member # 26) on 02-12-2012, 10:42 PM:

It wasn't too bad, as it turned out. We had a total of 18 people show up for the late movie. I gave free passes to six of them and they were very understanding and said they'd be back tomorrow. The other 12 elected to watch a repeat showing of the early movie instead, and I still gave them passes to come back for another show of the late one.

This drive IS ingesting, it's just awful slow. At the rate it's going, it'll be morning before it's finished.

Posted by Frank Cox (Member # 6258) on 02-12-2012, 11:14 PM:

I just had a thought: Have you tried hooking that drive up via the USB port instead of just plugging it into the CRU bay? Perhaps there's a problem with the drive interface that you could bypass by going through the USB instead.

Thinking about it some more, the "boot" that contains the USB interface attaches to the CRU interface so that may not actually change anything. But it's probably worth a try.

I had one drive a while back that wouldn't slide into the CRU bay; I think the drive case was slightly bent. I ingested it via the USB hookup instead and while it took a very long time compared to the usual CRU ingest, it did work.

Posted by Brad Miller (Member # 2) on 02-13-2012, 12:38 AM:

Be sure to drop that drive down the stairs a couple of times to get it out of circulation Mike. Help save someone else the same grief.

Posted by Tony Bandiera Jr (Member # 2365) on 02-13-2012, 01:10 AM:

^^^^ LOL what Brad said. [thumbsup]

Posted by Phil Ranucci (Member # 3820) on 02-13-2012, 01:18 AM:

Probably too late now, but on a GDC "exception" can mean not enough space. Can you highlight in the status page and see if you get a message?

Posted by John Thomas (Member # 6509) on 02-13-2012, 01:19 AM:

It could very well be a bad drive.

However, how much free space did you have available? I ask because depending on what type of server you have, getting too close to full causes pretty much exactly the symptoms you've described. On a Sony for example, 75% full is the "danger zone" where movies start to take 12 hours to ingest, and then refuse to play. I believe it's a function of what level RAID is implemented.

Posted by Richard Orsak (Member # 4472) on 02-13-2012, 02:46 AM:

Mike,

How were you able to book 2 features on a single screen. I've tried many, many times, but never had any luck (unless one of them was basically on second run and I wouldn't want that)...

Thanks!

Richard

Posted by Monte L Fullmer (Member # 2797) on 02-13-2012, 03:52 AM:

Just wondering if you deleted the failed ingest content before you tried to re-ingest, and how full were the drives in the server.

I've always maintain to keep the RAID below 60% to prevent ingest problems - get rid of content not being used.

Another question that one can think of: after all of the ingests that these servers goes through, what about a good defrag/wipe to clean things up at times?

Posted by Brad Miller (Member # 2) on 02-13-2012, 04:08 AM:

Is that a "feature" of GDC and Sony? I have NEVER seen an ingest slow down on a Doremi or Dolby when it reaches close to capacity...ever!

In fact I have done tests on bloating the raid intentionally and ran a few week's worth of shows on Doremi and Dolby servers with only 1-5GB left of free space on the drive. What problems happened you ask? None. Absolutely nothing went wrong.

Posted by Frederick Lanoy (Member # 5416) on 02-13-2012, 04:28 AM:

I fully agree. Once, i saw on my Dolby TMS that there was only 1 GB left on a Dolby SMS screen. It worked perfectly.

Posted by John Thomas (Member # 6509) on 02-13-2012, 04:29 AM:

Brad, I think what's going on is that the total space the server reports is the actual physical space of the drives, but it doesn't take into account that a major chunk of that space is required for parity. If I'm correct, that's a pretty obvious design oversight, which makes me hope that I'm way off.

I had the same experience as you when we had a Doremi DCP-2000: 99% full without a hitch.

Posted by Scott Norwood (Member # 30) on 02-13-2012, 05:33 AM:

Shouldn't an incomplete or corrupt file have failed the checksum test? Isn't it the responsibility of the player to tell the operator that the files are defective or incomplete?

I really don't know, but I am curious as to why it should even be possible to load and attempt to play a bad file.

To quote Brad: this is 2012.

Posted by Bernie Anderson Jr (Member # 410) on 02-13-2012, 06:34 AM:

I've gotten my Sonys to take an ingest to 99% without a problem and actually playback. Most of the problems I had with the Sonys was Software versions out of date and not matched from projector to projector. Sony came in 2 weeks ago and updated everything and we're working perfect now. So its probably a bad drive. I had issues when the ingest was disrupted for some reason and you had to reboot. The content would get corrupted and you would have to delete the incomplete file and re-ingest.

Posted by Geoff Newitt (Member # 6663) on 02-13-2012, 10:14 AM:

Unlike Brad, I have seen Doremi servers with very full RAIDs do some wacky things - failing to execute macros, skipping content. I'm also told they have a limit to the number of 'clips' they can keep track of - 1024 is the number that comes to mind; sounds like a lot, but in the UK we have extensive Ad reels, with each advert an individual CPL... One of our cinemas managed to have 1035 CPLs on one server through poor management of their content, and this definitely caused problems.

Posted by Frank Cox (Member # 6258) on 02-13-2012, 11:03 AM:

One other thing that I've had happen here (twice so far): The screws that hold the CRU drive together were loose. There are either 4 or 6 small screws on the bottom of the drive that attach the top of the housing to the bottom. I noticed the loose screws before I tried ingesting the content, so I tightened 'em up before inserting them into the drive bay.

Perhaps loose screws could lead to wobble in the drive platters and cause the content to either not read or to read very slowly.

Posted by Edward Havens (Member # 4715) on 02-13-2012, 11:24 AM:

The only times I ever have trouble with slow loads from a hard drive is when we are asking the LMS Server to do multiple tasks at the same time. I have trained my staff, not always successfully, to not do any transfers to individual screen servers when ingesting, and to never do more than two transfers at the same time. But then, I have 14 screens to contend with.

Posted by Manny Knowles (Member # 1171) on 02-13-2012, 12:11 PM:

quote: Mike Blakesley
Normally it only takes about 20 - 25 minutes to load a whole movie

My DSS-200 normally takes about an hour to load and another hour to verify.

Posted by Mike Blakesley (Member # 26) on 02-13-2012, 01:25 PM:

Space isn't an issue. I always delete last week's movie before putting this week's in, and I keep the trailers trimmed back too.

The reload finished sometime during the night, but when I tried it out this morning the problem was still the same.

Doing some further looking, I noticed that there are three filnames associated with this movie, due to the captioning available. There is a CCAP file, an OCAP file and the third one has neither of those notations. All three in the 120gb size range. (The movie in question is "Red Tails.")

I first thought that third file, the one with no caption notation, would be the one used to play the movie from without captions, but in the info sheet, it said to use the "CCAP" file to play the film without captions, so that's what I've been doing.

This morning I ran the "Verify" test on all three of those files and the one with no notation in the name came back showing a failure. So right now I'm reloading that file again. It should be done by tonight and I'll report back what happens. (I had run the verify on the "CCAP" file previously, since that's the one I was playing from, and it showed no problems. So maybe it uses both of those files somehow??)

quote: Richard Orsak
How were you able to book 2 features on a single screen. I've tried many, many times, but never had any luck (unless one of them was basically on second run and I wouldn't want that)...

That's basically it...one older movie plus a newer, semi-popular one.

Posted by Monte L Fullmer (Member # 2797) on 02-13-2012, 09:34 PM:

Ya, have to watch out for the last section of the filename in the naming convention of the "OV" (original version) and the "VF" which are the patches that goes with the "OV" version, after the "OV" version is ingested first.

Posted by Chad Bateman (Member # 6006) on 02-14-2012, 03:46 PM:

Although many of the previous comments could have lead to the problem, I would suggest contacting your integrator. Maybe he could shed some light. Maybe he could entertain us with some posts to give a better idea of what is going on.

Posted by Mike Blakesley (Member # 26) on 02-14-2012, 10:48 PM:

Wellllll.... It's got to be the hard drive (or more accurately, a bad FILE on the hard drive) but I'm still not quite sure why it's happening.

I did go in and check the space on the server, just to make sure, and we're only at about 35% so that can't be the issue. All the drives in the RAID are showing OK, too.

I looked over the info sheet again for the drive and it does say to play the file that's not marked for captions (contrary to my post above, sorry), and that's what I had in my playlist -- but when the show is running, the "now playing" note on the monitor still has the CCAP filename showing. So apparently that unmarked file references the CCAP file. Both of the files (CCAP and unmarked) are over 100GB and they're all marked "OV".

I ran all the files through the "quick verify" and it showed no problems on any of them, so then I ran them through the more thorough verify option. This time, it showed the unmarked file as a fail. So, I changed my playlist to just using the CCAP file directly thinking that might solve it.

The CCAP file showed no problems, yet the glitch still remains. At about 3:15 it just goes back to the beginning of the program.

Tonight I tried to stop the playback just before the "bad" part and skip it ahead over the defect and get it back on the screen a few seconds faster. That backfired on me -- I think I cut it too close and the system froze entirely, so I had to reboot to get going again.

So, we have one more night of this nonsense and then it'll be time for something else. I have gone to the low-tech solution of putting an explanatory sign up saying telling people that there will be about a ten-second gap in the movie due to a malfunction and thanking them very much for their understanding. I'll be glad to send this hard drive down the road. (And yes Brad, I will definitely throw it down the stairs a few times. And maybe "freight damage" it with my truck for good measure.)

In 19 months since we went to digital, this is the first "bad" drive we've had. Had to happen sometime I guess.

Posted by Brad Miller (Member # 2) on 02-14-2012, 11:23 PM:

Yup, every hard drive will fail, but hard drives that are "handled" in the way CRU drives are will fail MUCH sooner!!!

It's just bad luck. Nothing more than that. That being said if the hard drive is still write-able, they WILL re-use it on another movie!

I'll bet if it isn't already happening it will soon...buying new replacement drives just like the film depots used to have to do with new reels.

At least Deluxe FINALLY dumped all of those non-CRU craptacular drives!!!

Posted by Monte L Fullmer (Member # 2797) on 02-15-2012, 12:18 AM:

..and had this happen to me in a TMS server - couldn't do any LS ingesting at all when one of the RAID drives conked out on me and locked out the ingest procedures.

Had to do local server ingests until the drive was replaced and then it took most of the day for it to be assumulated in with the other 8 drives.

Posted by Marco Giustini (Member # 4544) on 02-15-2012, 01:00 PM:

I'm with Scott here, if you try to ingest a DCP from a bad drive, or a corrupted file, the server will reject it. Each file will be loaded and then analised and the HASH key compared with the good one listed in the CPL. If your feature was corrupted on your server, it is more likely a server issue. Or it could be a "bad" DCP, meaning the mastering is somehow wrong - but the files and the HD are fine.
Unless your server does things differently: I'm familiar with Doremi, Dolby and XDC. But the HASH key in the CPL is there for a reason!

Posted by Mike Blakesley (Member # 26) on 02-15-2012, 04:01 PM:

I am reluctant to blame the server. The movie was deleted and then re-ingested, yet the glitch happened in the exact same spot. Also, before I deleted it, I had done my weekly "purge" of older trailers so there was even more space available on the second ingest. Besides, like I said, we have another movie playing and it works perfectly.

Bottom line -- if this weekend's new movie does a similar thing, then I will start making server-related phone calls! [Big Grin]

Posted by Brad Miller (Member # 2) on 02-15-2012, 09:12 PM:

I'm with Mike. I don't think it was the server. Sounds to me like a faulty hard drive that read just good enough to pull the file in and validate it, but still produced a corrupted file.

That being said, I've never had that happen before to me.

Posted by Scott Norwood (Member # 30) on 02-15-2012, 09:40 PM:

If I read the DCI specification correctly, specifically section 9.4.3.5, it seems that the server is not actually _required_ to detect a corrupt DCP. [Roll Eyes]

Still, there is enough information in the DCP to do this, and to not do it (and allow a corrupt DCP to play without explicitly warning the operator that there may be issues with it) seems like either a bug or a design flaw with the server.

(This assumes that the DCP in Mike's case was actually corrupt, and not just improperly created.)

Posted by Steve Guttag (Member # 268) on 02-16-2012, 06:05 AM:

I believe Mike said that the normal "quick verify" didn't show any problems but using the more thorough verify DID show problems with the main OV file.

Oh and Manny...I've never had a verify take as long as the load on a DSS200 (or 100)...in fact, they are normally not even close in length. The 1-hour also seems long, unless you are running a movie at the the time. It will load faster via CRU than USB though...USB will indeed take upwards of an hour or more, depending on the size of the movie.

Posted by Manny Knowles (Member # 1171) on 02-16-2012, 06:14 AM:

Steve,

I use the CRU unless it's an indie that wasn't delivered that way.

I've never ingested while also running a DCP. There just hasn't been a need to (yet).

Perhaps the verify takes less time. I have yet to time that stage. I guesstimated based on how slow the progress bar was moving. It seemed to match the rate of ingestion.

Posted by Steve Guttag (Member # 268) on 02-16-2012, 06:18 AM:

Your next assignment...time both and report back. Hop to it mister!

Posted by Mike Blakesley (Member # 26) on 02-16-2012, 07:41 PM:

quote: Steve Guttag
I believe Mike said that the normal "quick verify" didn't show any problems but using the more thorough verify DID show problems with the main OV file.

Right, that's what happened.

About the verify time --- the quick verify on our server takes just a couple of seconds. The full verify, probably about 20 to 30 minutes for a typical movie.

Posted by Steve Guttag (Member # 268) on 02-16-2012, 08:06 PM:

Dolby does the full verify on each movie...since only the truly board will stare at the verify bar-graph as it crawls...it would seem to me that doing a full verify would go a long way to not having this sort of problem. That is, identifying the problem before the show so one might have a chance at replacing the content before it is needed.

-Steve

Posted by Andrew Maddison (Member # 4499) on 02-17-2012, 04:17 AM:

quote: Geoff Newitt
I'm also told they have a limit to the number of 'clips' they can keep track of - 1024 is the number that comes to mind; sounds like a lot, but in the UK we have extensive Ad reels, with each advert an individual CPL... One of our cinemas managed to have 1035 CPLs on one server through poor management of their content, and this definitely caused problems.

Unless I've got the wrong end of the stick, would this only affect the people with Digital Cinema Media adverts rather than those from Pearl & Dean as P&D send out theirs as one large package and then a KDM which chooses the specific ads that are relevant to the screen?

Posted by Manny Knowles (Member # 1171) on 02-17-2012, 07:12 AM:

Preliminary/intermediate experiment, since I don't have a feature to copy right now...

I copied a 14GB short from the RAID to the backup drive (CRU). It took just under 5min.

Would a 140GB feature take 10x as long as 14GB short? 30min? It seemed to take longer but maybe it's just that old adage at work: "a watched pot never boils."

Would it take the same amount of going from the (CRU) backup drive to the RAID? Or is that a different process?

(Repeating for convenience - I'm working with a Dolby DSS200)

I'll be able to experiment more next week, or whenever I get another feature.

Posted by John Thomas (Member # 6509) on 02-17-2012, 02:23 PM:

It's a bit faster going from the CRU drive back to the RAID. The bottleneck becomes the read speed of the single drive, which is faster than its write speed.

If you were using USB for the same experiment it would be slower and the same speed both ways as the bottleneck would be USB's sad 480mbps transfer rate.

Posted by Marco Giustini (Member # 4544) on 02-23-2012, 12:24 PM:

Brad,

The files are copied to the RAID and THEN verified. The HASH key is then compared with the one listed in the packing list on the drive. Again, not familiar with GDC, but this is what Dolby/Doremi/XDC do. If the file is unreadable after is has ingested, it can only be a server failure - unlikely, since it's a RAID, I know.

Andrew,
The AdPacks you receive contain all the adverts individually - they are just not visible on the server (Sony actually show the individual CPLs). The file you receive is not a KDM, it's another CPL that defines the playlist for your theatre.
To answer your question I'd say yes. But you just need to remove the older AdPacks and you should be on the safe side.