Film-Tech Forum ARCHIVE: Is it necessary to wait for degraded drive or detect failure ahead Dolby Show Store.

This is topic Is it necessary to wait for degraded drive or detect failure ahead Dolby Show Store. in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=000617

Posted by Eric Robinson (Member # 2966) on 08-02-2010, 02:46 PM:

I just realized that the only way we have discovered degraded hard drives on our Dolby Show Stores is during a feature when the drive fails. Unfortunately this is also visible to the patrons while the system is unable to access the drive the screen either freezes or goes black several times before the drive is automatically kicked off line. It would be nice; however, to eliminate the step where we piss off an entire theatre of patrons and detect the dfective drive ahead of time. Is this possible? Or, must we just suffer?

Posted by David Stambaugh (Member # 1102) on 08-02-2010, 02:57 PM:

Does the server RAID controller use SMART drive error detection? If the RAID controller picks up any SMART errors from a drive before it completely fails, and alerts you, you could preemptively replace the drive and rebuild the array after-hours (although SMART isn't foolproof). Generally though drives are most likely to suddenly fail while they're being heavily accessed. If that happens and the RAID array goes into degraded mode in the middle of a show, there's not much you can do.

I say that while having no firsthand experience with the specific hardware involved. [Smile]

Posted by Phil Ranucci (Member # 3820) on 08-02-2010, 03:09 PM:

I believe that if you watch the bootup screen you'll see a report on the drives.

Posted by Pete Naples (Member # 761) on 08-02-2010, 04:21 PM:

Are we talking DSS100 or DSS200?

Is the software up to date>

As Phil said, you should see a report from the RAID BIOS at bootup.

I've seen plenty of degraded arrays, but never seen any playback interruption. Usually it alerts you via Show Manager, and a red light on the front panel.

Next you should perform a RAID rebuild, the procedure is in the book. If it can't see the drive to rebuild, or the rebuild fails, then replace the drive with the spare one you keep on a shelf, and rebuild again. All should now be well.

If you have DSS200 with V4.2 software, I believe that the unit has the capability to auto-initiate the RAID rebuild, which is nice.

Posted by Kris Verhanneman (Member # 5562) on 08-02-2010, 04:45 PM:

On the new dolby soft (version V4.2.something) the rebuild has become just a click of a button. You don't even have to reboot the server.

Posted by Brad Miller (Member # 2) on 08-02-2010, 06:57 PM:

The auto-initiate only happens upon a power cycle, so if you are in the group of leaving the servers on 24/7, it will never auto-initiate. There is an option in the servers tab to rebuild the array though.

Posted by Scott Norwood (Member # 30) on 08-02-2010, 08:39 PM:

What David said. If the disks and RAID controller support SMART, this should be fairly trivial to implement. Agreed that it isn't a perfect indicator, but it should help identify impending hardware failures about half of the time, anyway.

What RAID level do these things use, anyway? Some are better able to withstand failures than others.

Posted by Steve Guttag (Member # 268) on 08-03-2010, 06:06 AM:

RAID 5. Three active, one hot swappable. I've never had a drive failure glitch a show (that anyone has reported to me). They normally just either see the RED LED or when they are loading a show, they notice that it is taking a very long time.

The auto rebuild thing is nice on version 4.2 Unfortunately, it already had to be used on a new DSS200 (installed in May).

Steve

Posted by Mark Gulbrandsen (Member # 72) on 08-03-2010, 10:21 AM:

If you are seeing drive failures often on DSS-100s then check with Dolby on the firmware versions on the drives themselves. They had a batch of Seagate drives with buggy firmware and Dolby has an upgrade disk to take care of the problem.

The first drive failure I can remember on a DSS-100 caused the player to do strange things. This was with earlier DSS-100 firmware so a drive failure could cause problems. Not on later firmware versions though...

Posted by Eric Robinson (Member # 2966) on 08-04-2010, 11:04 AM:

The failures have occurred on DSP100's which have hardware RAID and run on the Linux OS. I have not seen and SMART drive monitoring for this unit and I likely would not enable just in case it would cause some type of conflict.

So far all of the drives I have removed were indeed mechanically trashed. That is you can hear that the drive bearings are shot. One of them actually fell apart and you could hear parts jiggle around inside when you shook it.

We do power the system down nightly but have not had the benefit (in all cases) of having a hard drive taken off line during the boot up.

We are running 4.08 firmware and likely will not upgrade since there is a cost involved.

Posted by Scott Norwood (Member # 30) on 08-04-2010, 01:41 PM:

If you have access to the Linux shell, you can install smartmontools to read the SMART information from the drives. In principle, you cound configure it to run periodically and send email to someone when there are potential issues.

Personally, I am not a believer in regularly powering down anything that has hard disks. My experience has been that disks tend not to fail when kept spinning, and do tend to fail when powered up after not spinning. That said, I am not familiar with the Dolby unit; if Dolby (or your dealer or service tech) has a recommendation either way, I would recommend following it.