Film-Tech Forum ARCHIVE: Dolby Show Store RAID degradation

This is topic Dolby Show Store RAID degradation in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=000408

Posted by Terrence Meiczinger (Member # 5059) on 07-11-2009, 06:39 PM:

We have 3 Dolby Digital Cinema systems and have seen 3 red light drive degradation errors in less than 6 months. Each time Dolby has sent out new drives, which fixes the problem. However, I'm having difficulty believing we had 3 drives spontaneously fail.

I'm more inclined to think the issue is with Dolby's procedure of just having us hard shut off the Show Store/Show Player each night. I remember from my Unix days that the OS and RAID were never happy being treated like that. Thoughts?

Posted by Brad Miller (Member # 2) on 07-11-2009, 06:48 PM:

Don't power down the player ever. You can (and probably should) reboot it once a month, but powering down hard drives daily is just asking them to fail.

Posted by Monte L Fullmer (Member # 2797) on 07-11-2009, 06:55 PM:

We have GDC servers and was told never to power them down being thatthey also contain three HD's in a RAID setup within this setup.

We just turn the monitor off and let the thing buzz away happily every night.

Posted by Mark Gulbrandsen (Member # 72) on 07-11-2009, 08:52 PM:

First off for sure do not turn your units off! If you loose a second drive the entire OS has to be reloaded as well as your content and shows.

Always try a RAID rebuild FIRST!!! This works 93% of the time. The procedure I believe is in the latest gestation of the manual... if you can't find it just call Dolby tech support for the correct proceedure. Allow about 3+ hours for this to complete. If a rebuild doesn't fix the problem then replace the bad drives and leave the servers powered on through a decent line regulating type UPS.

Mark

Posted by James B Gardiner (Member # 5151) on 07-11-2009, 09:05 PM:

Whats the standard for you guys.

UPS or no UPS?

And if UPS, what is on the UPS and what is not?

Posted by Terrence Meiczinger (Member # 5059) on 07-11-2009, 09:16 PM:

Is there a reboot sequence? I didn't notice one in the manual, power cycle seems to be their solution. I never got the point of rebuilding RAID arrays (outside of drive size changes). If it can be rebuilt, shouldn't it just be automatic by nature.

I have a replacement drive, so I'll just swap it out. The last time, I didn't need to rebuild manually, just put in the new drive and it did its thing. In theory these are hot-swappable... maybe I'll be brave enough to try that one sometime.

Posted by Steve Guttag (Member # 268) on 07-11-2009, 09:52 PM:

Rebuilding is required even if you swap the RAID drive......you have to go into the 3WARE interface as the unit boots up (After it sees/doesn't see all of the drives...it will prompt you on the screen and tell you the button sequence.

We recently had a drive with less than 3-months on it go down...3WARE couldn't even find it...so forget about a rebuild...Dolby sent a drive...DOA...second drive did the trick.

Honestly...you should expect a drive failure about once/per year per unit.

We UPS our servers with true sine-wave output (always on battery power so there is a zero transition time)...only the critical stuff is on the UPS. Though the monitor is on it, we have our clients merely turn it off when not in use (At the end of the day).

I'd recommend having a spare Raid drive on-site AND having a hard drive to store all of your snipes and such (or to bounce content off when doing RAID repairs...much easier than to start from scratch and load all of the clips back on.

Steve

Posted by Terrence Meiczinger (Member # 5059) on 07-11-2009, 10:26 PM:

We just replaced the drive and rebooted. It looks like it is rebuilding... flashing green on the good drives, flashing yellow/red on the replacement. I'll see if it ends up rebuilding automagically, if not I'll go into 3-ware bios and mark a rebuild. I think I'll take the good advice and get a UPS in each rack and stop powering these things off. If we keep getting drive failuires, I'll switch to solid state drives... or this thing called film. :-)

Posted by Brad Miller (Member # 2) on 07-11-2009, 10:52 PM:

Solid state drives would break the bank. Have you looked at the price of a SSD and the small sizes they are offered in? Not an option at this point in the game.

What kind of drives is Dolby using? Does anyone have a manufacturer and model number?

Posted by Terrence Meiczinger (Member # 5059) on 07-11-2009, 11:07 PM:

off the shelf $99 drives and since I happen to have one in my hand fresh out of a dolby unit... Seagate Barracuda ES.2 ST3500320NS. Considering the same sized SSD is $1500... yeah, I'll stick with the mech drives for awhile.

Posted by Tony Bandiera Jr (Member # 2365) on 07-11-2009, 11:25 PM:

quote:
If we keep getting drive failuires, I'll switch to solid state drives... or this thing called film. :-)

(Bold added by me. )

Going to film would be your best option. [Smile]

Posted by Brad Miller (Member # 2) on 07-12-2009, 01:46 AM:

Wow, yeah those drives are nothing special at all. In fact odds are Dolby is getting them in OEM bulk, which only carries a 3 year warranty. The retail packaged drives (which are expected to last longer) carry a 5 year warranty. Regardless, exact replacements for those are under $75 delivered all over the internet.

Personally I would go with a higher quality drive with retail packaging if I was going to replace them.

Posted by Stu Jamieson (Member # 4626) on 07-12-2009, 05:24 AM:

So what's the rationale behind not rebooting RAID drives? Why does this make them more prone to failure?

Posted by Steve Guttag (Member # 268) on 07-12-2009, 08:00 AM:

Dolby claims that the drives they source are not off-the-shelf in that they are rated for longer life cycles than those, of the same model number, sourced through normal outlets....or so they said in their class.

Steve

Posted by Mark Gulbrandsen (Member # 72) on 07-12-2009, 08:52 AM:

quote: Steve Guttag
Dolby claims that the drives they source are not off-the-shelf in that they are rated for longer life cycles than those, of the same model number, sourced through normal outlets....or so they said in their class.

Yea, they give that hard drive spheil in every training class!! With the frequency of the red drive lights occuring I'd bet that one would have the exact same luck with off the shelf Seagate drives of the same/similar type when it's a failed drive. Don't forget that a red light doesn't always mean the drive indicated has failed. It's intersting that I have ONLY seen frequent RAID failure on Dolby Show Store units... not on other manufacturers Servers. The other thing here is that the RAID card in the Dolby is now a dinosaur among RAID cards and will only recognize about 400gb maximum. It's getting more and more difficult to find drives that size. Dolby also charges a ludricous price for the drive... Something like $400.00 list. One can remove the plastic rails from the old drive and put them on the new drive in less then a minute.

SSD's are the way of the future... still a little too early to utilize them in this sort of application both because of price and because of write limitations of flash memory. Several Servers (Doremi is one) use a SSD or flash memory for the OS.

Alas, Dolby is comming out with a new one piece server anyway in the very near future!!

Mark

Posted by Steve Guttag (Member # 268) on 07-12-2009, 10:29 AM:

Uh....not the near future...the future is HERE! The drives for it, though also 500GB are less expensive too.

Dolby DSS200

Steve

Posted by David Stambaugh (Member # 1102) on 07-12-2009, 01:25 PM:

quote: Terrence Meiczinger
Seagate Barracuda ES.2 ST3500320NS

That's an "enterprise-class" drive made for 24/7 operation in servers and such. It has a 5 year warranty and the MTBF is 1.2 million hours. I've installed bunches of them at work, in both desktops and servers. Have had numerous failures. Look up that item on newegg.com and read the user comments. That specific drive model has a history of very high failure rate. Only Seagate knows why. I switched to the Western Digital equivalent model, haven't had one fail yet that I can recall.

OEM ST3500320NS at newegg.com has a 5 year warranty through Seagate. Read those user comments though. [Eek!]

Posted by Brad Miller (Member # 2) on 07-12-2009, 01:50 PM:

I didn't realize those OEM drives had a 5 year warranty. They certainly don't anymore though, as Seagate changed their warranty policy for all OEM drives down to 3 years.

Regardless, Dolby should be using something more reliable, such as the Seagate Barracuda 7200.11 series. I've had plenty of WD failures and Maxtor drives are absolutely awful. At the end of the day, Dolby probably purchased 50 thousand of them to get some magnificent price and now have a warehouse full of them. Only Dolby knows though.

Posted by Jeremy Weigel (Member # 4208) on 07-13-2009, 01:02 AM:

If memory serves, Seagate bought Maxtor a couple of years ago, which might explain Seagate's current higher than normal failure rate given that Maxtor's drive as Brad has mentioned were/are terrible.

Posted by David Stambaugh (Member # 1102) on 07-13-2009, 10:01 AM:

I dunno, not trying to start anything, but I found this Seagate Technology OEM Limited Warranty Statement

quote:
For Product sold to a Seagate OEM customer (“OEM”), the standard warranty period is three (3) years for Notebook and Desktop drives and five (5) years for Enterprise and Nearline drives.

The drive Dolby is using is definitely Enterprise/Nearline.

Of course Dolby, as the system integrator, can warrant the drives for whatever period it wants.

Posted by Dennis M Dow Jr (Member # 1970) on 07-13-2009, 03:34 PM:

I have been using WD drives for as long as I can remember never had a problem. Did have issues with Maxtor in any computer that I had to build with it. Seagate used in my very first build way back when and it served me well for the time I had it. Just feel more comfortable with the WD Caviar drives

Posted by Mark Gulbrandsen (Member # 72) on 07-13-2009, 07:00 PM:

IMHO they blew it big time by not going to RAID 6. Not only is it far more redundant but it doesn't cost all that much more over the RAID 5 cost.

Mark

Posted by David Stambaugh (Member # 1102) on 07-21-2009, 06:32 PM:

Heh, I had another of these drives fail today, RAID-10 array in a server, server being backed up at the time the drive croaked. [uhoh]

Replaced drive with shelf spare. Costs $19.99 to get an advanced warranty replacement for the bad drive directly from Seagate, has already shipped.

Posted by Eric Robinson (Member # 2966) on 08-28-2009, 10:45 AM:

My failed drive from a Dolby Show Store is:
Seagate ST3400620NS 400GB
When I removed the drive I heard something inside of it go "Clunk" After a little "shake test" I concluded that something inside the drive is very loose, you can hear the disks flopping around inside. Its hard to resist my natural impulse to take it apart, but I have to send it back for warranty replacement.

The show store is 2 months old.

Posted by Andy Bajew (Member # 5330) on 08-28-2009, 12:33 PM:

quote: Stu Jamieson
So what's the rationale behind not rebooting RAID drives? Why does this make them more prone to failure?

When you constantly shut down any type of computer system, you are putting extreme thermal stress on all of the components from heating and cooling.

Posted by Mark Gulbrandsen (Member # 72) on 08-28-2009, 01:54 PM:

You can still re-boot most computer and servers running a RAID by doing a hot re-boot instead of a cold reboot in which the power is removed to re-boot. I always felt the Dolby RAID card is way way way too busy all the time... even at idle its alot more active than most RAID arrays I've worked with and I believe this contributes to the HD failures. Other manufacturers servers don't have nearly as active of RAID and rarely seem to loose drives. The 2 Dell 2850 servers I run at home each have 6 148 gig SCSI type drives in them that are older than Digital Cinema is! And they have never failed... they are always running though and on UPS in case of power bump.

Mark

Posted by Demetris Thoupis (Member # 833) on 08-29-2009, 10:17 AM:

Aren't drives supposed to be horizontaly mounted!? When I was doing PC services I attended a seminar and someone was mentioning that the drives are supposed to be horizontal mounted and not vertical as they are prone to more failures.

Posted by David Stambaugh (Member # 1102) on 08-29-2009, 10:32 AM:

"Modern" drives are designed to be mounted in any horizontal or vertical plane, including upside down.