Film-Tech Forum ARCHIVE: What would cause a jitter?

This is topic What would cause a jitter? in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=002200

Posted by Frank Cox (Member # 6258) on 01-31-2015, 09:10 PM:

Every time I play American Sniper, there is a period of about two minutes early in the movie during the scene where the father is talking about predators and sheepdogs where I would swear the film is dragging.

The picture gets jerky and the sound slows down, rumbles a bit and starts snapping, and it's exactly what you would see and hear if the film is under too much tension for whatever reason.

This happens for about five or ten seconds at a time, off and on over the course of about two minutes, and then that's it -- everything works perfectly for the remainder of the show.

I suspect that the server isn't feeding data to the projector and sound system quickly enough to keep the movie going, but why? I've played this movie three times so far and I've seen it happen three times, always in the same spot.

This is a GDC server and a Christie projector. The SMS screen on the server shows nothing out of the ordinary and if I didn't see it happen myself I would never know anything had occurred.

The control screen on the server says Raid Status: Online so I guess I haven't lost a disk.

Posted by Paul H. Rayton (Member # 1817) on 01-31-2015, 10:07 PM:

I just saw this movie this afternoon, and don't recall being struck by any such anomaly as being apparent in that scene as presented today. Nor can I figure out how the data stream could be "slowed" to create such a weirdity as you describe.

If you continue to observe it, I suppose you could request a replacement DCP and see if that "cures" the issue?

Posted by Brad Miller (Member # 2) on 01-31-2015, 10:24 PM:

For kicks, why don't you try to live play that scene?

Posted by Frank Cox (Member # 6258) on 01-31-2015, 11:40 PM:

Thinking about this matter further, I had the same issue a few weeks ago when I played Annie. I also played The Gambler that same week.

When I watched the movies by myself, Annie did the same slow-down off-and-on for about two minutes at one point in the movie, but The Gambler played fine.

I then watched that portion of Annie when I was playing it afterward for the public, and there was no problem. So it only happened once, the first time I played it.

That was a couple of weeks ago, so I've played a two or three different movies since then with no issue, up to now when it's consistently back with American Sniper.

I wonder if it's some sort of a disk error where the movie happens to be stored on a portion of the disk that requires several retries before providing the data, and that creates the slowdown. It affects both the picture and the sound, so I'm guessing that something is not feeding the data quickly enough and to me that suggests that some sort of a retry effort is going on and while we eventually get the data it arrives late. If that's the case, it's odd that there is no error reported on the SMS screen, unless there is an error log somewhere else that I should be looking at?

Posted by Brad Miller (Member # 2) on 02-01-2015, 12:23 AM:

Send the logs to GDC. Only they can truly tell if there is a hard drive issue. The GUI will say the drives are fine when they aren't.

Posted by Paul H. Rayton (Member # 1817) on 02-01-2015, 12:50 AM:

I just saw this movie this afternoon, and don't recall being struck by any such anomaly as being apparent in that scene as presented today. Nor can I figure out how the data stream could be "slowed" to create such a weirdity as you describe.

If you continue to observe it, I suppose you could request a replacement DCP and see if that "cures" the issue?

Posted by Monte L Fullmer (Member # 2797) on 02-01-2015, 03:10 AM:

Not uncommon for a HD to get grumpy in a GDC server, esp if the Raid Status comes up.

Click on Control Panel and see what the server information looks like above the control tabs.

See what the Raid Status looks like with all drives showing "OK"

If you got something different, prob got a drive flaking out.

But, you could delete and re-ingest the content to see what the results are...or as Brad mentions in doing "Live Play."

Posted by Steve Guttag (Member # 268) on 02-01-2015, 03:38 AM:

At the very least, go to the Admin screen (presuming you have access) and check each drive's "SMART" data and look for both Errors and reallocated sectors. They are definitely NOT definitive but they CAN be an indicator of drive issues. If you have a drive that has over 100 reallocated sectors or you note one that has a rising number on a daily basis...change it out.

But Brad's advice of pulling the logs and sending to GDC is an excellent one. If you can give them specific date/times of the incident, it will go a long way to finding the problem because they will look for that date/time stamp.

Posted by Justin Hamaker (Member # 2165) on 02-01-2015, 04:28 AM:

This sounds similar to what I have experienced when a RAID drive starts to fail. It could also be some corrupted data. As Steve said, check your Admin screen for errors. And if you have tech access, check the status of your drives.

If you don't see anything on the admin screen indicating a drive is failing, you could always delete the content from the server and re-ingest.

Another thought is it could be a heat issue. In this case, it could indicate on of the fans is failing. If it feels abnormally hot, you could position a house fan to blow over the server to see if that makes a difference.

Posted by Marcel Birgelen (Member # 6801) on 02-01-2015, 04:43 AM:

I'd go with the RAID hard-disk problem, I've seen similar issues before due to a single broken disk. A single disk lagging behind on several occasions can drag down the performance of your entire RAID considerably.

The problem with many of those server RAID solutions is they're standard "plug in" hardware or software solutions. Those implementations go for data availability and don't count in the I/O needed to sustain a fluent playback. In some cases it would be preferable to drop the disk out of the array if it cannot cope with the required transfer rate.

The biggest problem now is to identify the potential problematic disc. Steve already explained it. Usually, it should be the one with the highest error rates, but you should check SMART values at several intervals. If one disk is rapidly increasing in error counts, it's the most likely culprit.

Another less likely thing you should need to check is your local network. I've seen servers getting into trouble, because there was a virus infected host on the network, constantly bugging the servers, like trying to brute-force itself into some of the services that were open. This caused such a CPU load that it interfered with normal playback operations.

Posted by Frank Cox (Member # 6258) on 02-02-2015, 03:08 PM:

Well, I've succeeded in moving the glitch from near the start of the movie to nearer to the end.

I checked the SMART data on each drive and there is nothing reported there that appears particularly notable as far as I can tell; none of the numbers are excessively high and nothing says "error".

I then deleted the movie and re-loaded it from the distribution drive, then sat down to watch it this afternoon.

No glitch during the sheepdog vs predators scene.

However, further on into the movie during the funeral scene where they are doing the gun salute, playing taps and handing over the folded flag, I got the same glitch that I had before during the sheepdog vs predators scene.

I played the movie a few minutes past the glitch, then paused it and moved back about five minutes before the funeral scene and played it again.

Same glitch at the same general location, but not exactly what it was before. The second time the glitch seemed to be slightly worse and last slightly longer, and while it happened during the same two minute period it didn't occur at the exact same second. In other words, a slightly different portion of the movie was messed up on the replay vs the first play through.

So....

I don't know exactly how files are allocated on the disks, but I get the impression that I'm looking at a glitch in a fixed location on the disk or disks.

I think would have had something like this last Thursday:

|---Paddington---|---American Sniper---|

I deleted Paddington late on Thursday night. Then I deleted American Sniper today and re-loaded it, so now I probably have something like this:

|---American Sniper---|---Empty Space---|

The overlap between the first part of American Sniper in the first diagram, and the last part of American Sniper in the second diagram is likely where the glitch lives, since it has moved toward the end of the movie and it was near the beginning before.

Posted by Marcel Birgelen (Member # 6801) on 02-02-2015, 04:16 PM:

quote: Frank Cox
I checked the SMART data on each drive and there is nothing reported there that appears particularly notable as far as I can tell; none of the numbers are excessively high and nothing says "error".

Write down those numbers per disk and check them after another show with glitches and look if they moved in any way. It's most probably just a single disk causing the issues, but it's often rather hard to identify it.

If you can afford it (it's not horrendously expensive) and do have the time, I would actually just swap all the disks. If it doesn't work, you'll end up with a bunch of spare disks.

Posted by Frank Cox (Member # 6258) on 02-02-2015, 05:00 PM:

That will be my fix. I've just asked the techs who look after my equipment to send me a complete set of new disks.

If one of them is going, then the others probably aren't too far behind anyway.

Posted by Steve Guttag (Member # 268) on 02-02-2015, 09:10 PM:

Actually no...every once in a while, you'll get a clunker of a disc and the others in the same batch will run seemingly forever. I've had much worse luck with Seagates than Hitachi discs. If your drive model begins with an "ST"...then it is a Seagate. Hitachi drives begin with "HUA"

Posted by Monte L Fullmer (Member # 2797) on 02-03-2015, 12:34 AM:

Just for comments sake: I use PSD-3000 storage units and in one feature, I had skipping in a certain section. I also did the delete and re-ingest, went into ADMIN and did the checks where everything was good, but my skipping was still there.

I was planning on just getting new HD's after calling up GDC support, but they went the easy way to send me a new storage unit.

They figured that the controller in the storage unit wasn't accepting the new RAID 3.03 update and was conflicting with the SX-3000 IMB card.

Got the new unit, hooked it all up so the IMB can recognize it, re-ingested content and I was on my way with no problems since.

-Monte

Posted by David Buckley (Member # 2600) on 02-03-2015, 03:24 PM:

quote: Steve Guttag
...you'll get a clunker of a disc and the others in the same batch will run seemingly forever.

And sometimes an entire batch of disks all start failing at about the same time...

Not too bad in a cinema context, lose a couple of shows, yeah, that's really annoying, but when several disks in a big array with business information go bad almost simultaneously and take the array out, it can be very ugly...

Posted by Marcel Birgelen (Member # 6801) on 02-03-2015, 07:09 PM:

That's why it's a good idea to mix batches of disks. I even advocate mixing suppliers or series for anything "business critical". Unfortunately, many hardware storage solutions are still picky in accepting disks from multiple vendors, if sector sizes don't match up.

Also, RAID-5 might still be mostly fine for a single screen, but with ever-increasing disk sizes, it's even becoming a liability in those kinds of applications.

For DCI, future playout systems should have better I/O monitoring and reporting per disk, just like many modern SAN systems already have. Such monitoring could identify and tackle defective disk issues rather easily and much more reliable than stuff like SMART. But since we seem to steadily move to some sort of centralized storage / NAS solution, often done with cheapo hardware, I expect throughput and other I/O related issues to become ever more common in the future.

Also, once every projector is hooked to some central storage, you will now not only lose one auditorium once stuff goes haywire, but all the other rooms as well.