Film-Tech Forum ARCHIVE: Mysterious short pause during playback

This is topic Mysterious short pause during playback in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=001088

Posted by Frank Cox (Member # 6258) on 04-19-2012, 06:38 PM:

Here is something I haven't seen before. I have a GDC server and a Christie projector.

I'm playing Titanic 3d tomorrow night so I was doing my QC screening of it this afternoon. Just about exactly one hour in (during the dinner party scene) the picture and sound froze. As soon as I got over my surprise I zipped up to the projection room. By the time I got there (maybe one minute later), it was playing again just like nothing happened. No error messages on the server display, either.

I paused the playback and backed up to just before the start of the dinner party scene and started playing it from there. No problem, no pause, from there right through to the end of the movie.

When the credits were running I paused it again and backed up to just before the start of the dinner party again and played through that scene again. No pause, no problem.

So... what happened there and should I be worried? Seems ok now and there were no error messages anywhere that I saw.

Posted by Justin Hamaker (Member # 2165) on 04-19-2012, 06:51 PM:

I would suspect some sort of processing issue on the GDC which cause the hang - perhaps a file finished ingesting or something similar.

If the just the picture paused and the sound continued uninterrupted, then there was probably some kind of data bottleneck in the projector.

Posted by Frank Cox (Member # 6258) on 04-19-2012, 07:08 PM:

Both the picture and sound froze, as though the movie went on pause. I still had a picture, but it stopped moving. No sound at all while it was stopped.

Absolutely nothing else was going on, other than playing the movie. My server and projector are stand-alone, they aren't hooked up to anything other than themselves.

I checked the server status screen and it reports OK for everything that's listed there.

Posted by Brad Miller (Member # 2) on 04-19-2012, 07:52 PM:

Seen it happen a lot on GDCs. I think that is called a "feature". I've never been able to get a straight answer, but if anyone here knows of a fix, please do share.

Posted by Manny Knowles (Member # 1171) on 04-20-2012, 06:55 AM:

It's your intermission [Big Grin]

Posted by Rick Raskin (Member # 1561) on 04-20-2012, 07:45 AM:

I am not familiar with the architecture of these servers but from my experience with broadband data networks, I would suspect that there is a buffering issue in the GDC. I did see a similar issue where resizing the buffer solved the problem. In another instance the scheduling algorithm needed tweaking. In either case I think the problem is with the undocumented "feature" of the GDC.

Of course, the real test here is does the problem reoccur?

Posted by Scott Norwood (Member # 30) on 04-20-2012, 09:41 AM:

Bad disk in the array or a RAID controller issue, perhaps?

Posted by Frank Cox (Member # 6258) on 04-20-2012, 12:32 PM:

I would think that if it was a disk or disk controller board issue that would be reported as an error on the status screen. In fact, a disk error was my first thought when it happened. But there are no errors reported that I can see.

I had no issues with the two (different) movies that I played last night, and we'll see what happens with Titanic tonight. Needless to say, I'll be babysitting the projector during the dinner party scene.

Posted by Brad Miller (Member # 2) on 04-20-2012, 02:09 PM:

quote: Scott Norwood
Bad disk in the array or a RAID controller issue, perhaps?

FYI: 3 drive raid5 and the raid is software, not hardware.

Posted by Scott Norwood (Member # 30) on 04-20-2012, 02:15 PM:

Wow. Software RAID 5 is not safe. A power failure during a write operation will probably kill the filesystem. The only way to do RAID 5 safely is with a controller with a battery backup on board (they usually use little button batteries like motherboards use for CMOS).

Also, RAID 5 implemented in software has some significant performance penalties in write operations, although it should be a nonissue for reads.

Software RAID is fine for striping and mirroring (RAID 0 and RAID 1, respectively), but not really a good idea for RAID 5, RAID 6, etc.

Posted by Jason Raftery (Member # 6291) on 04-20-2012, 02:28 PM:

Perhaps the array encountered an unrecoverable data block during playback and that was what caused the brief interruption? I've seen this happen once on a Dolby DSS100/DSP100 where playback briefly stalled before continuing as if nothing had happened. A check of the number of bad data blocks recorded by the server on each drive showed that one of the drives recorded a new bad block during the show.

Do GDC's servers record a similar metric?

Posted by Frank Cox (Member # 6258) on 04-21-2012, 12:42 AM:

Tonight's show went off without any problem or pause. So I guess that was a one-time glitch. Hopefully it doesn't decide to return.

Posted by Terrence Meiczinger (Member # 5059) on 04-21-2012, 12:58 AM:

quote: Scott Norwood
Wow. Software RAID 5 is not safe. A power failure during a write operation will probably kill the filesystem.

No, it really shouldn't. Most software raid implementations catalog data that they claim to have written into parity. After a power failure the drives should resync (not rebuild) next time they boot to ensure everything is correct. The issue would be if a drive failed prior to the resync. However, even in a hardware raid this could be an issue depending on the RAID type and which drive failed.

As far as performance, it's hard to say as it depends on a few variables, CPU, drives, number of reads/writes, data chunks, etc. In a degraded state, software raid could take a bigger hit. I've even seen software RAID outperform cheap hardware controllers.

RAID is not a backup solution. It's a redundancy solution.

Posted by Monte L Fullmer (Member # 2797) on 04-21-2012, 04:01 PM:

Question: do you power cycle (cold boot) your servers once a week to rid of "stale" memory?

How full are the drives, for possible that when the drives are getting above a certain percentage, access from the drives could be a bit troublesome, but don't know about the GDC setup.

I usually maintain drive space to be less than 70% at all times.

Posted by Marco Giustini (Member # 4544) on 04-21-2012, 04:56 PM:

Hi Scott,
Interesting, I did not know. But how can the HW RAID controller complete the writing process if the drives are shutting down as well?

Posted by Carsten Kurz (Member # 5396) on 04-21-2012, 05:46 PM:

Scott was not just referring to a hardware-raid controller, but a hardware-raid-controller with battery buffering. When the system shuts down unexpectedly, it can buffer the pending data transfer in memory and perform it on the next power-up.

However, there are other strategies to recover from power outages even with software raid. Software RAID5 without battery buffering is in widerspread use on computer systems and RAID appliances nowadays. If they'd be prone to critical file system corruption the way Scott describes it, that whole business would be dead already. I don't know of any DCI server with hardware+battery buffered RAID5.

A UPS is another quite useful (and cheap) investment not only to aid in file system consistency, but also for fast show-time recovery from power-outages. A UPS is standard with all Sony installations, with the Sony using RAID6+dedicated spare drive. But all other projectors can be retrofitted with a UPS as well.

- Carsten

Posted by Scott Norwood (Member # 30) on 04-21-2012, 07:36 PM:

I realize that a DCI server can "get away" with software RAID 5 because it is mostly reading from the array, not writing to it, and mdadm and other software RAID mechanisms aren't as fragile as I may have made it sound.

Still, I would not trust my own data to any software RAID variant other than level 1 or 1+0 (or maybe RAIDZ). Something like RAID 5 or RAID 6 really requires the aforementioned battery-backed hardware controller to be considered safe. And good backups are, of course, always essential.

Posted by Terrence Meiczinger (Member # 5059) on 04-21-2012, 08:36 PM:

I think there are misconceptions on the purpose of the battery backup on RAID controllers and that leads people to believe it is safer than software raid.

The reason for battery backup is to allow the RAID controller to operate in a caching mode which increases performance. In this mode, data writes are cached on the controller. When the controller cache becomes full (or sometimes a period of time), it writes the data out to the drives. The issue in this scenario is that if the power failure occurs and data is still cached, it will be lost. Hence, battery backup. Although, many controllers now use flash backed cache and eliminate the battery.

Regardless of whether you use hardware or software, a power failure during a drive write can cause a mismatch in parity. Both will detect this and resync. The difference is that a hardware controller can typically resync faster and sooner than software RAID upon reboot.

I don't see hardware RAID being any safer than a good software implementation.

Posted by Eric Robinson (Member # 2966) on 04-22-2012, 10:33 PM:

UPS systems seem to be held in high regard; however, even if a UPS system sustains power to a server during a power outage, it will eventually drain its reserves and the server will abruptly loose power just as it would without a UPS.
So, it would seem that the value of the UPS would be that it could notify the server to shutdown gracefully; although, all of the UPS installs I have seen are not integrated to communicate with the server...so what good are they?

Posted by Brad Miller (Member # 2) on 04-23-2012, 02:15 AM:

Assuming that a GOOD UPS is installed and not some piece of cheap junk, it will keep the system running for quite some time, typically more than adequate enough for power to be restored to the building.

Posted by Steve Guttag (Member # 268) on 04-23-2012, 06:24 AM:

The vast majority of power interruptions are in the split-second to a second or two. For longer periods then yes one would need to shut down the system prior to the battery dying...but even if it died via battery...it would be a clean stop. Normally if there is a power interruption via the power company, there is a surge/spike associated with it and the sensitive equipment will be shielded from that so it functions as an uber surge/spike arrestor too since the power is fed via the UPS directly, not the power grid (depending on the model of UPS and style).

Better automation designs also will, upon sensing a power loss of greater than say a second, will stop the show in progress too since the sound/lighting will likely go away (plus the picture because very few put a UPS on the xenon lamp supply). Having a UPS for the automation allows it to do these functions.

Most of the TMS/LMS servers I've seen do, in fact, have a means for the UPS to shut down the TMS/LMS gracefully before the battery completely dies.

-Steve