Film-Tech Forum ARCHIVE: Degraded Array on Dolby Library Server

This is topic Degraded Array on Dolby Library Server in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=000993

Posted by Nate Anderson (Member # 2711) on 12-31-2011, 11:31 AM:

I got a warning that there was a degraded array on my library server. I've done multiple rebuilds on both Series 1 and Series 2 Dolby processors so I proceeded to try and rebuild the Library Server as well but was unsuccessful.

Am I right to assume that I need to install a new hard drive and then attempt another rebuild or is there another option?

Thanks.

Posted by Frank Cox (Member # 6258) on 12-31-2011, 01:10 PM:

In any RAID, a "degraded array" means that one or more of the drives is inaccessible for some reason, the most common reason being that the drive has failed. It's also possible that the drive isn't physically seated properly so the connectors are snug -- basically anything that will cause the drive to stop communicating reliably with the RAID controller.

Posted by Mark Gulbrandsen (Member # 72) on 12-31-2011, 02:07 PM:

Normally but not always a bad drive also has a red busy light. If not then take a look at the logs to try to determine which one is bad. I have one DSS-100 that has a bad RAID card in it. It works fine but it will not rebuild the array properly if it is playing back... or so to say it has lost it's throttle. The array also will mysteriously become corrupted on it's own. If you're not able to look into the logs your self then send then to Dolby tech support.

Mark

Posted by Steve Guttag (Member # 268) on 12-31-2011, 05:48 PM:

Note, NO DSS100 can rebuild its RAID while the show is playing...the time slice given to the show in progress prevents a successful rebuild...a DSS200 can do it, albeit slowly. In fact, starting with version 4 software, a DSS200 will self-rebuild when it finds itself with a degraded RAID.

If you are not on version 4.2.x or beyond on your DSL100, then rebuilding is more difficult. I'd recommend updating the DSL100 and all players in the system to 4.3.x.x software...quite a few improvements in there. 4.3.0.36 is the current release version but 4.3.5.x should be coming out in January (it is in beta testing).

One thing to look at is the number of block errors on each RAID drive (go to System>Server Tools) and it will list the block errors on each drive...a high number is the first indicator that the drive is headed for failure.

Another thing to make sure is that everything is kept clean inside of the player and that things are not overheating.

-Steve

Posted by Mark Gulbrandsen (Member # 72) on 12-31-2011, 06:46 PM:

Steve... Reread my post again. The problem is it is not throttling down if you put the thing into play which is what it should do... It keeps rebuilding instead of stopping rebuilding but it also does play at the same time for a while and it keeps this up until the whole RAID collapses into itself then requiring an OS and content reload. This issue has nothing to do with this version of the OS and the ultimate problem was determined by Dolby tech support. The thing in reality needs either a sledge hammer blow to its mid section or to be replaced by that "other brand" of server which has always been able to ingest, play and rebuild at tge same time...

Posted by Steve Guttag (Member # 268) on 12-31-2011, 06:56 PM:

Oh, the other brand has not always been all that great and remains a slower ingesting 3-drive system. Heck, playing off the CRU is a "new" feature for the other brand.

-Steve

Posted by Mark Gulbrandsen (Member # 72) on 12-31-2011, 07:37 PM:

Actually the "other brand" has been able to "live play" since version 7.7a. That was about the time the DSS200 appeared on the market and quite a while longer than I have been installing the "other brand". And if you take into account the verification the Dolby does AFTER the ingest it takes nearly twice as long to do the ingest. Of course you can skip the verify if you choose to do so and take that risk. The "other brand" on the other hand does verification in the background while it is ingesting. The "other brand" also boots up in 58 seconds verses 6+ minutes. It's a good thing Dolby hired some one capable of straightening out the mess!!

Posted by Brad Miller (Member # 2) on 12-31-2011, 07:38 PM:

Boys, Nate has a problem with his LIBRARY, not his screen server.

Nate what firmware are you running on the library server? Click SYSTEM/THEATRE DEVICES then scroll down to "version".

Also Mark, I've no idea what bs you are spewing, but I've never had nor heard of the problem you claim Dolby has. What I can interject into this thread is that the server your company sells is slow as hell to ingest and we have seen stuttering issues when ingesting during playback. Dolby has NO such issues. Dolby also doesn't require you to run back through the control panel to get to the main controls during ingesting to prevent canceling an ingest. [Roll Eyes]

Facts my friend, not opinion. You really need to stop smoking that I-30 stuff.

Posted by Steve Guttag (Member # 268) on 01-01-2012, 01:38 AM:

Faster way to check software version on the Dolby (DSL or DSS). Click on the Dolby logo in the lower-left corner of the screen...it will put up the "splash screen" again. click on the splash screen to close it.

-Steve

Posted by Kris Verhanneman (Member # 5562) on 01-01-2012, 03:00 AM:

Hello Nate,

yes you are correct. Just shut down the server and pop in a new drive.
If you are on one of the more recent versions the rebuild will start all by it self (if not you may have to do it by hand).
Before replacing the hard drive do check if the drive is fully inserted.

Posted by Scott Norwood (Member # 30) on 01-01-2012, 10:17 AM:

How does one tell which drive has failed?

Random thought #1: with most RAID cards (in general, not D-cinema specific), even if a degraded array rebuilds successfully without having to replace a drive, a degraded array is generally a sign of an impending disk failure.

Random thought #2: when replacing a disk in a RAID, be sure to use a type of drive which is designed for RAID use. Not all are, and they tend to drop out of the array (at least with most hardware RAID implementations; it is not a problem with most software RAID implementations) even when there is nothing wrong with the disk itself. In general, "enterprise" or "RAID edition" disks should be fine. Don't use consumer-grade or, worse, "green" disks in a RAID. Actually, don't use consumer-grade or "green" disks at all, for anything, if you value your data and don't want to have to restore from backups. Ideally, all disks in a RAID should be of the same type, capacity, and firmware; otherwise, performance will suffer.

Posted by Steve Guttag (Member # 268) on 01-01-2012, 11:30 AM:

In the systems I've worked with (not just Dolby), the offending drive will have a flashing RED LEDs to signify the problem child.

In the case of Dolby, it will also alert the operator with a warning message. In the case of a full TMS system with the DSL100, all theatres will report a problem with any of the systems and identify which player/drive has the issue.

-Steve

Posted by Michael Voiland (Member # 5386) on 01-02-2012, 12:32 AM:

Hopefully they will start to use what is called a global hot spare in future systems. They need to catch up with using server grade hardware. If you can access the raid controller software / configuration find the bad drive. Replace the bad drive with a good one. Rebuild array if it does not start automatically. Call and get a replacement. Make sure noting is accessing the array well you are working on it if you can as if the array has to regenerate data well rebuilding the array performance and the rebuild will suffer. If you can force the rebuild into high priority do it. Since its degraded the array is vulnerable to another hard disk crash.

Posted by Bruce Hansen (Member # 281) on 01-02-2012, 04:51 PM:

The answer to all these problems: 35MM FILM (sorry, I couldn't help it). [Big Grin]

Posted by Michael Voiland (Member # 5386) on 01-02-2012, 05:28 PM:

Second the above.

Posted by Mark Gulbrandsen (Member # 72) on 01-02-2012, 06:14 PM:

Michael... Actually the Cinedigm TCC racks do operate a hot spare and they also run a raid 50 instead of a raid 5. I doubt you'll ever see this on a local server because the content is easily replacable or you can instantly go to live play if an issue exists with the main raid.

Mark

Posted by Carsten Kurz (Member # 5396) on 01-15-2012, 04:52 PM:

Sonys have RAID6 + HotSpare (4+2+1). Two drives can fail, and one is swapped automatically.

- Carsten