Film-Tech Forum ARCHIVE: Degraded array ?

This is topic Degraded array ? in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=001922

Posted by John Eickhof (Member # 323) on 05-30-2014, 02:13 PM:

Running 13 GDC SX-2100A servers, just had the second array crash in two years of use! (all three drives) Is this going to start being a problem? I would think they would hold up longer than a couple years?

Posted by Stephen Furley (Member # 1277) on 05-30-2014, 02:23 PM:

I've seen disks fail within hours of being installed, and I have one still in use that's over fifteen years old. I think two failures in two years is too small a sample to draw any conclusions from.

While three drives all failing together is possible it's very unusual; are you sure you don't have a RAID controller problem?

Posted by Mark Gulbrandsen (Member # 72) on 05-30-2014, 02:35 PM:

John,

It depends on the brand of drive. Seagates pretty much suck. Hitachi's are much better but also seem to have slightly gone down hill since bought out by Western Digital. I'd consider three to four years on time the average expectant life span of any SATA drive. Out of well over 100 cinema servers I have installed in the last few years I've lost less then 8 drives total. It is very uncommon to loose more than one drive, has only happened to me once on a DSS-200. In your case I doubt that all three drives are actually bad. I'd rebuild the raid and see what happens with it. After rebuilding it look at the SMART info in the raid panel for bad sector counts. If they are high then replace those drives. Since GDC warrants their servers for three years you should still be in warranty and get new drives for free.

Also... If you loose one drive and someone accidentally pushes one of the green illuminated buttons next to the drives then you loose the entire raid!!

On the OS drives....since you are an important location and can't afford any down time you should seriously install an extra OS drive into each server, there is plenty of space next to the existing OS drive. You will have to get this drive from GDC because it has to have the SN (Certificate) burned into it in order to work. Not expensive to do and it could save you up to 4 days of lost screen time if you lost a drive on say a Friday evening. All of my furthest away sites and important locations have extra OS drives in them. I have yet to loose an OS drive but some of the GDC's are approaching four years old. Getting back on screen is as easy as moving the SATA connectors over and rebooting the unit.

As far as drives failing... there is no set frequency but I also consider them an expandable like a lamp. A complete extra set of raid drives ready to plug in place and go can also save your day!

I monitor some of my customers drives quarterly to see what the bad sector count is at. Mainly the sites that run lots and lots of hours.

Mark

Posted by Brad Miller (Member # 2) on 05-30-2014, 03:03 PM:

Mark, you can clone the GDC OS drives. Get it on whatever firmware you deem the most stable (what do you recommend anyway since you're Mr.GDC) and then clone it.

Since you love solid state drives and the OS doesn't need to be a large drive, that's actually an even better option.

I do agree with you I think something went stupid in the software raid. I would simply delete it and re-create the raid and re-load content. Odds are it will be fine.

Posted by Mark Gulbrandsen (Member # 72) on 05-30-2014, 07:02 PM:

Yes, they can be cloned... Just didn't want to get into that on here. The OS drives they sell have been dam reliable for me. Not sure on the SSD aspect though. I'll have to try that.

Mark

Posted by Scott Norwood (Member # 30) on 05-30-2014, 07:16 PM:

Statistically, SSDs are _less_ reliable than mechanical drives. Strange, but true. Intel and Samsung seem to be the most reliable brands of SSDs at the moment.

Posted by Mark Gulbrandsen (Member # 72) on 05-30-2014, 08:50 PM:

I have quite a few SSD running in various computers over the last four years and no failures...

Mark

Posted by John Roddy (Member # 7348) on 05-31-2014, 03:58 PM:

SSDs still need some work before they're ready to be used in a commercial environment like this. For media storage, a RAID of mechanical drives is vastly superior. It's just a matter of which brand. In my experience, Western Digital has been very reliable. Overall, Hitachi/HGST is considered the best. If you're a masochist who loves facing regular issues though, definitely go for Seagate. Every GDC server we got here had a RAID of Seagate drives in it. Over the past year or so, I ended up having to replace every single one of them. Even GDC seems to agree that Seagate is crap. Whenever I reported issues with a Seagate RAID, they sent me an entire new set of Hitachi or HGST drives instead of one replacement Seagate. Blech.

Posted by Brad Miller (Member # 2) on 05-31-2014, 05:14 PM:

quote: John Roddy
For media storage, a RAID of mechanical drives is vastly superior.

In case there was confusion of my suggestion to consider a SSD (since Mark loves them) was because the GDCs entire operating system relies on one NON-raid consumer mechanical hard drive, which is why I suggested they be cloned and for the consideration of putting them on an SSD.

I must be the only person without any real issues on Seagate drives. My drive failures have been minimal. I don't think I'm doing anything special.

Posted by John Roddy (Member # 7348) on 05-31-2014, 05:20 PM:

Ohhhhh, right. Yeah, an SSD for the OS would be fine. Not really sure if it's worth the investment though. I would argue that a mechanical drive is better for its longevity there. The biggest advantage of an SSD is the enhanced speed, but that's really not a big deal for this particular use. As long as it's not Seagate, it should be fine.

Also, in all fairness, the Seagate drives I had here never gave me trouble until recently. It took about three years of constant use before they started crapping out.

Posted by Steve Guttag (Member # 268) on 05-31-2014, 08:26 PM:

Brad, you ARE the only one that doesn't have Seagate problems. I want nothing to do with them...they are the same crapola that Osrams are to xenon lamps. I can't disparage them enough.

It isn't that there aren't good examples of them...(Osrams too) it is that there are enough bad examples to make them worth going out of business so as to not infect anyone else with their crap.

Posted by Scott Norwood (Member # 30) on 05-31-2014, 09:55 PM:

Which Seagate models are failing for everyone? I have had good luck with their Constellations (SATA and SAS) and Cheetahs (SAS only) in an IT context. What are the D-cinema systems using?

I agree that Hitachis are good (at least the ones pre-WD-buyout).

I have no opinion on Western Digital, since they mostly make consumer-grade drives and don't really do anything enterprise-y.

Posted by Steve Guttag (Member # 268) on 05-31-2014, 10:14 PM:

I've had Seagates fail at every level from consumer to Enterprise..eff em all.

I've had nothing but success from Western Digital.

Why do you say they don't do Enterprise grade drives? http://www.wdc.com/en/products/internal/enterprise/

Posted by John Roddy (Member # 7348) on 06-01-2014, 01:27 AM:

It's to my understanding that Seagate had pretty solid products up until around 2010. Everything after that was apparently designed and built by Satan himself.

Posted by Carsten Kurz (Member # 5396) on 06-01-2014, 07:23 AM:

http://blog.backblaze.com/2014/01/21/what-hard-drive-should-i-buy/

Interesting graph of failure rates in there. They are, however, using consumer grade drives, but very large quantities.

Seagate sucks, it seems...

- Carsten

Posted by Scott Norwood (Member # 30) on 06-01-2014, 08:05 AM:

I was thinking in terms of 15k SAS or fiber-channel drives as being "enterprise-grade" products. As far as I know, WD does not make those (though their Hitachi division does).

Posted by Mark Gulbrandsen (Member # 72) on 06-01-2014, 10:18 AM:

I'm with Steve on the Seagates. All levels of drives are absolute crap... I just won't even bother with them any more, they also build up error rates much faster than any other type of,drive out there. Hitachi has also gone slightly down hill since being bought out by WD. I have seen more Hitachi failures on the new production drives than I used to when Hitachi owned the company. Although not anything like the Seagate failures I have had.

Still waiting for an OS drive failure to happen in a GDC, the oldest one is about 5 years old now. Also, still waiting for a SSD to fail some place... Been waiting for like 7 years for an SSD to fail...

Scott... SAS drives are in a class by themselves... They rarely fail no matter who makes them, all my TMS's run SAS 15k drives in raid 1 for the OS. Zero failures.

Mark

Posted by John Eickhof (Member # 323) on 06-02-2014, 01:35 PM:

Thanks guys! Of course they were seagates!! HaHaHa

Posted by Carsten Kurz (Member # 5396) on 06-02-2014, 02:45 PM:

Were you able to verify that actually all three drives were bad?

- Carsten

Posted by John Eickhof (Member # 323) on 06-03-2014, 01:21 PM:

Yes, when GDC logged on they found degradation on all three and said the best thing to do is replace the whole array. It's actually quicker than replacing a single drive and configuring it.

Posted by Mark Gulbrandsen (Member # 72) on 06-03-2014, 02:49 PM:

And they usually ship them out with all the basic content on the drives already loaded.

Posted by Justin Hamaker (Member # 2165) on 11-10-2014, 03:53 PM:

Since we are 3+ years in on all of our GDC servers - and several are 4+, we're going to buy some spare drives to keep on the shelf.

With regard to the OS drive, is the cloning process any more difficult than cloning a HDD under normal circumstances? Would cloning the OS drive for one projector preclude it from being used in a different projector? Is it just a matter of having GDC apply the appropriate certificate when the drive is placed into use?

With the array drives, how critical is it to use the same drive? Most of our servers use the Hitachi HUA7220, but a few have various Seagate drives. While I can find plenty of the HUA7220 drives right now on eBay, it appears this is a discontinued model. Can a single failed drive be replaced with an off-the shelf drive of the same specs (2TB 7200 RPM is what most of our drives are), or does the whole array need to be replaced with the same drive? Can a single 1TB Seagate drive be replaced with a 2TB Hitachi drive?

Posted by Dave Macaulay (Member # 813) on 11-11-2014, 06:41 AM:

My experience is that although it may be best to use identical drives in a RAID set, mixing doesn't seem to be a problem.
You should use "enterprise" drives, not consumer drives. How they handle errors is differently and consumer drives can cause playback issues if (when) they are dealing with error correction, from what I can learn.
Installing a replacement 2TB drive into an array of 1TB drives works fine, but you only use 1TB of the new drive. With 4K releases getting more common, rebuilding the RAID with all 2TB drives in is a good idea if you tend to need several features in your server concurrently.
The drives originally installed in servers installed a couple of years ago are going obsolete, and identical replacements becoming unavailable. Check with your server manufacturer for a current list of recommended drives. Buying the drives from the manufacturer will be considerably more expensive than purchasing from a local supplier. However, if your server is under warranty, doing that might have consequences.

Posted by Mark Gulbrandsen (Member # 72) on 11-11-2014, 04:45 PM:

You should try to get HGST drives which is the new name for Hitachi. All three should be the same in a GDC because it is software RAID. You can also use 2 TB drives if you wish.

When you clone the OS drive just be sure to clone all the partitions on it and it has to be a bootable system drive when it's all said and done! Some of this depends on what HD cloning software you are using.

Mark

Posted by Marcel Birgelen (Member # 6801) on 11-11-2014, 05:18 PM:

What you should know is that some firmwares are known to "fudge" the numbers regarding read failure rates you're getting from Smart. Seagate, for example, reports actual numbers, whereas Western Digital reports some number they made up themselves. This might make Seagate look bad in this regard.

Furthermore, it's really difficult to point out one brand as the clear winner. Essentially, there are only two manufacturers left: Seagate and Western Digital. HGST is a WD subsidiary and it probably won't take long before the Hitachi will disappear entirely.

Most manufacturers have had their good series and their bad series. Some had major firmware snafus, etc.

Posted by Steve Guttag (Member # 268) on 11-12-2014, 06:17 AM:

Yeah...like Seagate...way too many of their drives have had issues. Without a doubt, if you are retaining the 1TB drive size, I'd stick with the HUA722010CLA330. It is, by far, the best drive used in DCinema on any brand server in the 1TB size. Just like xenon lamps, I'm not saying there haven't been ANY failures or premature aging...but compared to Seagate...they have been golden.

They are also in current manufacture under the HGST badge of Western Digital. You can still find some left under the Hitachi badge but they are clearly left overs from the last production.