Film-Tech Forum ARCHIVE: Dolby DSS 200 Array Degraded and Interrupted Playback

This is topic Dolby DSS 200 Array Degraded and Interrupted Playback in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=001724

Posted by Mohd Ridzuan (Member # 7054) on 12-10-2013, 02:27 PM:

Hello guys, first of all thanks to FT because i learn so much thing in this forum. But still there were many other things i need to learn and all of your help is very appreciates.

I had a problem with my Dolby dss 200 + NEC 2500s. There were 2 warning error messages appear.

1. The array was degraded. I try to rebuild the hdd and successful. Delete the the content and reingest back. However after one show the hdd became degraged again. Ther hdd is new one. Just replace it this year about 6 months. Is it possiblem the hdd can be degraded within a short periods.

2. There was warning Interrupted playback. In the middle of the show the playback got stuck and freeze. Then we pull out the degraded hdd, we can run the show smoothly. But if we insert the hdd back the show get stucks.

Anyone can help me what cause this problem and what necessaries step to solved this problem.

Thanks

Posted by Marco Giustini (Member # 4544) on 12-10-2013, 02:44 PM:

Check how many reallocated sectors that HDD has (SYSTEM-THEATRE DEVICES). More than a few is bad, I bet you'll find a lot.

Anyway, an HDD can become faulty after a few months, if it's new Dolby will replace it for you. The ARRAY degrades (it's an IT term, it's not physically degrading!), that means that one HDD is possibly bad.

You shouldn't need to delete and re-ingest any content.

Bottom line, just replace the disk.

Posted by Mohd Ridzuan (Member # 7054) on 12-10-2013, 06:59 PM:

Thanks Marco. What cause the HDD could be degraded? Is it cause by corrupted files? I had this problem before, but after doing rebuilding the RAID the problem solved. In this case its not. [Frown]

Posted by Manny Knowles (Member # 1171) on 12-10-2013, 07:36 PM:

A DSS200 is capable of functioning even if one hard drive goes bad. I know this because I have run shows with a degraded array, due to one faulty drive.

If your array is degraded to the point where you are having failures, then you might have more than one dead drive (or, possibly, some other problem that I haven't encountered).

My first degradation (and my only one, so far) occurred during the warranty period. So, yes, hard drives can die young.

When you replace the bad drive(s), do yourself a favor and order an extra one to keep on-hand as a spare. Steve Guttag made sure we were provided with a spare drive as part of our package, and I was glad to have it ready when it was needed. (Thanks, Steve!) The dead drive was replaced by Dolby, so that is now the spare drive.

Posted by Mohd Ridzuan (Member # 7054) on 12-10-2013, 09:22 PM:

Just been inform by operator on site, from 4 HDD now 2 of them had degraded. Now they playing with only 2 HDD in the slot as they had pull out the 2 degraded HDD. Both degraded HDD had to pull out because if its was inside it will interrupt the show became stuck n freeze. My question is, is it possible to play with just 2 HDD? and how long it will stay?

p/s: waiting for the replacement HDD

Posted by Marco Giustini (Member # 4544) on 12-11-2013, 03:54 AM:

No, a DSS system won't play with only two drives, your operator must be mistaken.

Probably the system was badly handled when being shipped. An array degrades when a disk fails. It could degrade because the disk has a temporary glitch - always worth try to rebuild the array if the disk does not show any - or few - bad sectors before replacing the disk - but in general that means you have a bad disk. Check the reallocated sectors count as I suggested.

RAID stands for Redundant Array of Independent Disks. Basically if you have 4 HDD of 1TB each the system will write the information on three disks only; the fourth will be use as a parity disk. That is, say disk 1-2-3 are the data disks and disk 4 is the parity one, if disk 2 fails the system can extrapolate the missing information using the data contained in disk 4 (it does not work like that but just to give you an idea). Your storage capacity of the RAID will be only 3TB.

If ONE drive fails, the system is still perfectly capable of accessing all the information - one disk will also be redundant. If TWO drives fail, the information on your RAID will be lost permanently.

When that happen you'll have to replace the two drives and reinstall the OS on the system. You'll lose your content and some of your settings.

When a disk fails the RAID controller should be capable of assessing that and should disconnect the faulty disk before it causes trouble but that doesn't happen all the time and you may end up with an interrupted show before the controller takes the disk out of service.

Check the number of reallocated sectors on all your servers, replace anything with more than 10 bad sectors (this is what Dolby recommends, from my point of view a brand new drive should show ZERO bad sectors). If servers are brand new Dolby will pay for them.

When you replace a drive the system will realise that one drive does not contain any information and will start the "rebuild" process which is recreating the data on the new drive based on the data on the other three drives. The process will take several hours. Can be done during playback but it's not recommended, better to replace a drive early in the morning or last thing in the night.

Posted by Steve Guttag (Member # 268) on 12-11-2013, 08:51 AM:

Close on the RAID description but not quite...All drives carry parity for the other drives as well as "their" data. Thus, drive one will have parity on drives 2, 3 and 4. There isn't a parity drive. The data is striped across all four drives (there are 3-drive RAID 5 but the same principles apply).

Perhaps you are thinking of a RAID array where a "hot spare" is in the frame ready to come on line if one drive craps out. Dolby does not have a hot-spare...all 4 drives work all the time. I believe what you were describing would more closely resemble a RAID 4...which isn't used in DCinema, as far as I know.

Other than that you are spot on. I'm a little more lax on how many reallocated sectors I'll allow...generally if I see less than 100, I'll let it fly. What I'm more interested in is how fast that number climbs. If it was 5 yesterday and 25 today...get it out of there FAST!

The other thing to check is in the logs and see how many errors, if any are showing as past events on the disks. I'm more strict on those because those are hidden unless you specifically look for them. You might have a drive that is having errors and then another drive degrades the Array and it can't rebuild because you have one with errors on it preventing the rebuild...I've had that happen.

Note, these principles apply to ALL of the servers, not just Dolby's It is a good idea to check the drives once or twice a year to see if anything is brewing. It is much easier to change drives out one at a time than to go through the whole re-initialization and loading content back on.

Posted by Marco Giustini (Member # 4544) on 12-11-2013, 12:32 PM:

Steve,

You're correct, but I also said

quote: Marco Giustini
it does not work like that but just to give you an idea

I just wanted to simplify!

I agree that you may end up with a disk with the same 50 bad sectors for years but if a drive is brand new I wouldn't accept anything more than zero. On a more aged one I may be a bit more relaxed as long as - as you say - the number does not quickly increase.
Dolby do not warn the customer of bad sectors and I've seen servers before ending with two bad HDDs in an endless rebuilding process. At that time it's usually too late and a full reinstall is required.

Posted by Steve Guttag (Member # 268) on 12-11-2013, 01:37 PM:

On a new drive...I would agree that zero should be the case...though I don't mind seeing one.

But at what point do you stop declaring a drive new?

I have noted that those theatres that shut the servers down at night, the reallocated sector counts tend to be higher.

Posted by Gordon McLeod (Member # 33) on 12-11-2013, 01:53 PM:

We also use only enterprise grade drives
some of the consumer ones have issues especially the green ones as they go to sleep

Posted by Marco Giustini (Member # 4544) on 12-11-2013, 01:57 PM:

It's food for thought. I would say that I'd like to see zero bad sectors on a out-of-the-box machine and for at least a month or two. Then I would have the manufacturer to replace a drive which shows a number of bad sectors high enough to allow warranty claim. If Dolby say 10 (I believe it's on their knowledge base) I would replace a drive showing 11 bad sectors straight away while under warranty. After the warranty I'd do as you say, keep an eye on it.

But I'd rather replace a 3y old drive showing 40 bad sectors before the customer experiences troubles with it, even though that drive could potentially last for another 3 years.

Posted by Mohd Ridzuan (Member # 7054) on 12-11-2013, 06:21 PM:

Thanks to Steve n Marco.

Steve you were right our server was shutdown every night after the session of the night. But we had another 3 halls that had no problem until now almost 2 years. Marco, this server was a new set, just been installed about 2 weeks back replace the old server which got mother board faulty. After 1 weeks using it this problem occurs.

My updates for this issue. Last night my operator said they now running without any hdd inside as we had a TMS system suggested by our installer. Is it possible? is there any incoming problem running the server without hdd? Our installer said try to reformat the hdd. If cannot then sent to Dolby for replacement.

Can anyone teach me how to reformat the hdd? If reformat failed, what is the step if i get a new replacement hdd. Just slot in the server and it will format n rebuild the RAID itself?

Thanks guys, i'm just a newbie...

Posted by Monte L Fullmer (Member # 2797) on 12-12-2013, 04:18 AM:

Pull/disconnect the two power plugs from time to time to clear out stale memory.

Posted by Marco Giustini (Member # 4544) on 12-12-2013, 06:37 AM:

Mohd,

You probably misunderstood, a server cannot run without HDDs. If that was what your projectionist and installer actually suggested... get rid of them, quickly! [Smile]

I wouldn't bother for a second, if your disk is showing bad sectors - did you check that? - I would ask for a replacement straight away. Your server is 2 weeks old!

Turning the servers off at night won't hurt them in the short term. It's proved that on the long term your HDDs won't like it and will fail sooner than when the servers are on 24/7. But at this moment this is not the cause of your problem.

Anyway, if a disk is showing bad sectors you cannot cure them. You can rebuild the array if it says "degraded". It should happen automatically, or just go under SYSTEM/Server tools and click on REBUILD. To be on the safe side, do it overnight, after the last show.

If the array is NOT degraded, it does not need to be rebuilt.
If any of the HDDs is showing bad sectors, replace it.

Posted by Marcel Birgelen (Member # 6801) on 12-12-2013, 02:59 PM:

quote: Monte L Fullmer
Pull/disconnect the two power plugs from time to time to clear out stale memory.

No modern computer system needs to be unplugged to "clear stale memory" and it also won't fix any degraded or even failed RAID.

I also totally agree with Steve on this one, the rate in which bad sectors are increasing is by far the most important number. Having a few bad sectors on a drive isn't an issue, as long as they're not increasing at a fast rate. If you replace an otherwise well performing drive with a new one, you never know what you get in return.

Posted by Marco Giustini (Member # 4544) on 12-12-2013, 04:00 PM:

Marcel

On a Dolby server the cat 862 is powered from the main PSU, rebooting the server by software won't fully reset it. A full shutdown every now and then is recommended.

I may agree with your statement, but if your hard drive is 2 weeks old I can't see how a few bad sectors could be acceptable.

If the drive shows a few bad sectors over its life, it's acceptable - as long as they're stable

If a drive reaches 30.000 hours and shows 50 bad sectors, I would replace it. Its unlikely a brand new replacement lasts less time than a 30.000hrs unit.

Posted by Marcel Birgelen (Member # 6801) on 12-12-2013, 04:56 PM:

quote: Marco Giustini
On a Dolby server the cat 862 is powered from the main PSU, rebooting the server by software won't fully reset it. A full shutdown every now and then is recommended.

That might be the solution if your Media Block is causing the troubles, but here it's clearly a RAID related issue.

quote: Marco Giustini
I may agree with your statement, but if your hard drive is 2 weeks old I can't see how a few bad sectors could be acceptable.

Why not? The whole idea of modern hard drive design is that sectors that become bad are being relocated. Not one drive that rolls of the production line is entirely flawless. So what does it matter if a drive has two or three relocated sectors right from the start if they do not increase over time? This drive is operating perfectly fine, any replacement has no guarantee to be any better.

Posted by Marco Giustini (Member # 4544) on 12-12-2013, 06:50 PM:

I understand that the drive has a hidden area for bad sectors which are expected over its lifetime. All bad sectors are reallocated over this area.
When the area is full then the drive start marking new bad sectors as 'not available'. Unless on Server-graded drives this is not the case that means that one bad sector is not actually one.

Anyway, I would not allow a single bad sector on a brand new drive. That's what I'd do. Those machines - and those drives - are too darn expensive for me to be relaxed on bad sectors! [Smile]

When the drive gets old, I think it's wiser to invest a little money in replacing it when it shows a number of bad sectors. If we lose a show because of it it's good money and you'll have to replace the disk anyway at some point so you're really saving pennies.

On my computer it's another story, I can keep an eye on the drive and live with some glitches, but I would not advise a customer not to replace the drive unless the number of bad sectors quicly increase. Most cinemas don't have a projectionist, this kind of quality control is simply not achievable.

Sure, removing the power from this server won't cure the bad disk, but when a problem shows up a power cycle is always recommended and most of the time fixes the problem - that's why we recommend a full shutdown every now and then, I believe I'm not the only one recommending it.

Posted by Gavin Lewarne (Member # 5397) on 12-12-2013, 07:58 PM:

dss200 here, 13 months old, came with the factory fit hitachi drives.

UPS power reboot once a week (to totally cut power for a few seconds)

0 bad sectors on all 4 drives.

Server is left on 24/7 on a good UPS and have never had any sort of raid related warnings or errors show up in the logs let alone needing a rebuild.

I was expecting at least some bad sectors by now but seems we have a very reliable trouble free unit. Its only ever had 1 real glitch and that was solved by replacing the silly blue link cable.

I was half expecting to have at least some raid related troubles by now as we generally only ever have a maximum of 4 films stored, and I believe the XFS file system used on the content partition will always try to write from the beginning of the partition so I imagine we are using the same few hundred GB of space on all the drives over and over again

Posted by Manny Knowles (Member # 1171) on 12-12-2013, 10:05 PM:

Give it time.

Posted by Brad Miller (Member # 2) on 12-12-2013, 10:52 PM:

quote: Marco Giustini
Anyway, I would not allow a single bad sector on a brand new drive. That's what I'd do. Those machines - and those drives - are too darn expensive for me to be relaxed on bad sectors

It is unreasonable to expect/demand a replacement from any server manufacturer on a new drive with a few bad sectors. As long as they aren't suddenly increasing noticeably, there is no problem.

Posted by Marco Giustini (Member # 4544) on 12-13-2013, 05:54 AM:

Ok, maybe I'm a little too inflexible here. But what would you do if your drive out of the box had 30 bad sectors? I think Dolby would replace it, and I'd have it replaced.

Posted by Jock Blakley (Member # 6552) on 12-13-2013, 08:43 AM:

Has Hitachi's quality moved on from the days when their Deskstar drives were known as Deathstars?

Posted by Carsten Kurz (Member # 5396) on 12-13-2013, 08:48 AM:

Hitachi does not produce hard drives anymore, since 2011.

And the 'Deathstars' were made by IBM when the product line was not Hitachi's.

- Carsten

Posted by Brad Miller (Member # 2) on 12-13-2013, 12:43 PM:

I wouldn't worry about 30 sectors if it wasn't increasing rapidly.

Posted by Marco Giustini (Member # 4544) on 12-13-2013, 03:20 PM:

Wouldn't be easier to ask Dolby to replace it? How do you find time to monitor the customers' drives to check whether the drive has to be replaced or not?
Time is money, the disk is under warranty, Dolby would replace it. Why wasting time?

Posted by Chris Slycord (Member # 4239) on 12-13-2013, 04:22 PM:

Also, SMART errors tell you that they relocated the data to one of the known good sectors, which may or may not indicate that the original sector is actually bad. There can be software issues that cause an error in reading the data that causes it to be relocated when the actual sector itself wasn't bad.

Posted by Marco Giustini (Member # 4544) on 12-13-2013, 04:29 PM:

And are you willing to wait till the customer calls reporting that they had to cancel a show because the picture was stuttering to verify that?
I see your point, but I'm not sure I agree with that.

Posted by Steve Guttag (Member # 268) on 12-13-2013, 04:38 PM:

IF I had customers complaining about stuttering images with a low relocated error count, I'd change my position. The fact is, I haven't. There has been no demonstrated reason for changing out a drive with a low error count (unless it is rising rapidly). As you say, time is money...who pays for the time it takes to order a replacement, send back the "defective" drive that works fine and changes the drive? I can guarantee you that it takes less time, effort and money to check the status of the drives every once in a while. We do it for all of our customers. Checking reallocated sectors takes very little time on most all servers. Again, seeing "errors" in the logs, to me, is a bigger indicator of a drive problem.

But hey...handle your service your way...nobody is saying you are wrong. We try to provide our customers with zero issues at all times (not always successful but mostly so)...like everything else, we concentrate on what we have found to be more problematic or leading indicators of failures and catch them when they happen...a drive with a dozen reallocated sectors hasn't shown itself to be a problem. A drive with 50 reallocated sectors today that had 25 yesterday...IS a problem.

Posted by Chris Slycord (Member # 4239) on 12-13-2013, 04:40 PM:

I don't see where I implied that.

And there are ways to find determine if a sector is actually bad. Heck, there's even software out there to help fix the sectors that are actually bad.

Posted by Marco Giustini (Member # 4544) on 12-13-2013, 05:07 PM:

Steve,

I did not mean to criticise your or anybody's service procedures, if it looked like that I apologise of course.

Your point is solid but I still believe in mine. I don't know how I could regularly access every system and track the number of bad sectors on every single drive. It means to track how the bad sectors number behaves on thousands of HDDs.

That being said, if I know the projectionist is a good one I could simply ask them to check the status of a 'suspected' drive over time and send a replacement if/when it's needed.
But the catch on the previous sentence is "is a good one" [Smile]

Chris,
It's very unlikely a bad sector can be repaired. And as I said time is money. I would definitely try to repair a disk in my home computer but when it comes to professional service where the customer is paying for my time and expecting a fully up and running system when I leave, that does not apply. IMHO of course.

Posted by Scott Norwood (Member # 30) on 12-13-2013, 05:26 PM:

Quite a bit of my day job involves storage issues, and I am with Brad and Steve on this: all disks have bad sectors. This is not a problem per se, but a rapid increase in the reallocated sector count (SMART parameter) does correlate well with impending disk failure (see the Google study on disk failure).

Also, I would pay attention to the hardware ECC recovered parameter (if there is a way to view this).

That said, SMART is not actually all that good at detecting disks that are about to fail. Really, we shouldn't have to worry all that much about this stuff as long as the RAID controller is doing its job. Sometimes good disks will be reported as "failed" by the RAID controller, yet the array can safely be rebuilt without replacing them. If this happens more than once, though, it is another sign of impending failure.

Posted by Marcel Birgelen (Member # 6801) on 12-13-2013, 05:53 PM:

quote: Scott Norwood
Sometimes good disks will be reported as "failed" by the RAID controller, yet the array can safely be rebuilt without replacing them. If this happens more than once, though, it is another sign of impending failure.

That can also happen if there is buggy firmware present on your disk. Another problem are disks that aren't designed for RAID purposes. Those disks often take too long to respond to a request, especially if an automated relocation is happening, that causes the RAID controller to drop the disk from the array. While they still may operate quite stable in a software-RAID environment, they usually fail rather rapidly in a hardware RAID situation.

And then there are those absolute terror drives like the WD Green series, those that put themselves in standby mode and get kicked out of the RAID once that happens...

Personally, I'm not really fond of RAID5 for those ever increasing arrays. Rebuilds are taking longer and longer and the risk of another disk failure is especially large during a rebuild. Something like RAID6 or a software based solution like ZFS with multi-disk redundancy would probably be better suited for future storage implementations.

Posted by Chris Slycord (Member # 4239) on 12-13-2013, 06:06 PM:

On whether you can/can't keep track of the number of bad sectors on thousands of disks, am I wrong in assuming that one could run something like the linux smartd daemon to track the count day to day on each server then send an email to you if a particular disk starts changing relatively quickly?

Posted by Marco Giustini (Member # 4544) on 12-13-2013, 06:16 PM:

Scott,

Don't get me wrong, what you say is sound.
I don't know you but my idea of average cinema is a forsaken booth managed by people who barely know how to press play. Yes, you can find all the information you need in the logs but it takes some time to download a set of logs and cannot be done while the server is playing.

Sure, you can access the server by terminal and cat/pico the files themselves but I find it a little impractical?

Marcel,
Again, you can do that on your PC or on servers you build. I would never run anything on a D-Cinema server which does not come from the manufacturer.

Posted by Carsten Kurz (Member # 5396) on 12-13-2013, 08:21 PM:

quote: Marco Giustini
It means to track how the bad sectors number behaves on thousands of HDDs.

That would be necessary now, yes, but of course should be done by the servers - detect a sudden increase in bad sectors and report automatically. Doremis recent reporting is a clear improvement. At some time, it may also consider reporting potentially dangerous RAID conditions. You could, however, also request these from the logs or via SNMP automatically. The reason it is not done now is that a single drive failure is already taken care of by the RAID redundancy, which then SHOULD trigger an alert.
Besides that, Google demonstrated that SMART reports are not a too reliable indicator of future drive failures.

- Carsten

Posted by Steve Guttag (Member # 268) on 12-14-2013, 03:38 PM:

There are numerous ways of remotely checking the drives and some have been listed here (e.g. a NOC with a decent SNMP system).

We have access to just about every theatre we install for the purpose of support. Lets say it is a Dolby server...how long does it take to go to the service tab and check the reallocated sector count? In GDC one can pull up the whole SMART report without ever pulling the logs. The answer by the way...seconds...you can spend about 15-seconds a screen checking that sort of stuff so you can remotely be in and out of a theatre...even a large plex in just a few minutes having checked 48 drives in a 12-plex. Again, will you catch everything? No...but it is a RAID 5 so even if a drive does go down...SNMP will already tell you that...if not the customer asking about the error/LED flashing. All without harming a show in progress.

If you see a drive with higher than normal error counts...take a quick note so the next time you go through you can compare to see how fast it is rising.

I know that as these systems are now getting older we are starting to see the effects of age/wear. One one of my regular remote checks...I found a 3-drive system were one disc had a LOT of reallocated sectors and the other two drives had a bit of a history of other ERRORs (according to the SMART report). My solution...all three drives got changed...an not one show was compromised.

As to drive types...all of the servers we work with use the Enterprise grade drives from the factory and we replace them with that sort of drive with near identical read/write speeds. Personally, I hate Seagate as seemingly all of my failures come from them (not just in the theatres...but in personal use too). I've had great luck with Hitachi and Western Digital. Again, for the WDs...it is their Enterprise grade drives, not their green energy efficient drives that are used. My Mac at home runs on one of their "Black Caviar" drives...never a lick of trouble in years and boots up real fast too. We all have our personal preferences based on past experience...my past is not your past so our opinions may vary.

I think, as time goes, we'll all develop our service techniques the same way we did for film systems...some will do better than others. I think, like with other systems, it is a mater of doing things as smartly and efficiently as possible to make the most use of one's time while minimizing problems. To me, wasting time on sending back a perfectly good drive and having the customer pay for that returned drive (the return freight is almost ALWAYS on the customer) for a drive that would have a 99.99% change of having a normal life-span just seems like a waste for all unless you could demonstrate that such a drive causes any show degradation or premature system failure. If the server manufacturer says that if a drive has 10 or more bad sectors within the warranty period they'll change it...okay...they have set that threshold. I haven't seen any manufacturer claiming just 1 bad sector is the threshold.

Posted by Brad Miller (Member # 2) on 12-14-2013, 05:39 PM:

For giggles, we have a large complex with 4 screens off on it's own wing we lovingly call "death row", as those auditoriums only seat a handful of people and exist purely for move-down booking purposes.

They have DSS200 servers in them and for kicks I tested my theory before they opened by putting a mix and match of drives in them, all of them used. One enterprise drive, one consumer drive, one green drive, etc...all different specs and manufacturers. Some were even repaired discs back from the manufacturer. We couldn't make them fail so we left them there and kept a real close eye on them.

They have been open for over a year now without a single failure. Gotta love a real hardware raid!

Posted by Steve Guttag (Member # 268) on 12-14-2013, 07:03 PM:

That is what makes you giggle? You really need to get out more [Razz]

Posted by Marcel Birgelen (Member # 6801) on 12-14-2013, 07:52 PM:

quote: Marco Giustini
Again, you can do that on your PC or on servers you build. I would never run anything on a D-Cinema server which does not come from the manufacturer.

It was more a general observation, I would not advice anybody to mess with an existing configuration, if that would be even possible. Since most current servers only carry 3 or 4 disks, it would even be quite useless.

But there is obviously a trend where the "traditional" function of the server is progressively being transferred into the IMB and the server is becoming a storage-only affair.

Just like it happened in many data centers, you will slowly see a demand for centralized storage, that's not only being used for distribution, but for direct playback as well. Such a storage solution is not something that I would ever consider running off a RAID5 system. If that's a desirable thing is something that can be debated of course.

Posted by Marco Giustini (Member # 4544) on 12-15-2013, 08:28 AM:

In fact today I ended up with a server showing 118 bad sectors on it. Using GREP under Linux I was able to see the previous RAID status from the logs in a blink. The same disk was showing 0 bad sectors just a few days ago, then 4-10-17-20 over the following days.

Posted by System Notices (Member # 2357) on 02-11-2016, 10:04 AM:

It has been 788 days since the last post.

Posted by Leo Enticknap (Member # 534) on 02-11-2016, 10:04 AM:

Thought I'd write this up in case anyone else experiences this.

Arrived in the booth about an hour and a half ago, and noticed that the DSS200 was doing the same thing as it did when I had a hard drive fail a few months ago. The thing would freeze totally for about 30 seconds (as in, Show Manager totally unresponsive), then all eight hard drive green LEDs would blink for about half a second in succession, the RAID card would say "Bleep Bleep!," followed by the server unfreezing and behaving normally for about a minute, rinse and repeat.

So I immediately had a look at the theatre devices tab to see which of the drives was buggered this time (we keep a spare in the booth, so at this point I wasn't too worried), but it said that the RAID was OK, and reported no reallocated sectors on any of the drives! I tried rebooting the server (properly - entered the reboot command into the shell, then yanked both power cords after the PSU fans surged, waited for 30 seconds and then powered up again), but it still did the same thing after the reboot.

At this point I was starting to panic: we've got a big show tonight with a Hollywood A-lister in attendance, the studio's tech coming to inspect us at noon and the prospect of the DCP server throwing a hissy fit not exactly making my day. After a deep breath and a coffee, I decided to try removing each drive, cleaning the contacts and reseating them. My plan was to do the same thing with the RAID card if this didn't work. My thinking was that given that Show Manager didn't report any bad drives, the fault, if there actually was one, was probably in the RAID card.

Anyway, cleaning the contacts and reseating the drives appears to have worked. The RAID card gave a single bleep on the reboot - a hopeful sign - and it's been about an hour since then without any trouble. Furthermore it's been ingesting throughout that hour. No freezes or double bleeps so far. My guess is that repeated mechanical (ramp up and down) and/or heat cycling caused one of them to work loose, hence the RAID card's temper tantrum.

Posted by Tony Bandiera Jr (Member # 2365) on 02-11-2016, 11:23 AM:

quote:
Anyway, cleaning the contacts and reseating the drives appears to have worked. The RAID card gave a single bleep on the reboot - a hopeful sign - and it's been about an hour since then without any trouble. Furthermore it's been ingesting throughout that hour. No freezes or double bleeps so far. My guess is that repeated mechanical (ramp up and down) and/or heat cycling caused one of them to work loose, hence the RAID card's temper tantrum.

Wait, so are you saying you shut down the server when not in use? The part I put in bold seems to suggest so.

There was a very long (and contentious) thread on whether to leave servers on 24/7 or shut them down....consensus was that since server hardware is designed to be on 24/7 it should stay on for maximum hard drive life. (And so far history has shown that to be correct.)

If you are not doing so already, leave the server ON 24/7 and if you haven't already, invest in a good quality UPS for the server and projector electronics. (Especially important if you are doing screenings for A-List folks.)

Posted by Leo Enticknap (Member # 534) on 02-11-2016, 11:29 AM:

No, we're leaving the server on 24/7, but the hard drives won't be spinning all the time, will they? Surely the RAID card will spin them down after a given period of time in which nothing is asking to read or write to the array? That was what I had in mind by mechanical and/or heat cycling.

Posted by Tony Bandiera Jr (Member # 2365) on 02-11-2016, 11:48 AM:

I see..do you have the server on a UPS?

AFAIK they are pretty much spinning constantly (perhaps at a reduced speed when data is not being read), I do know that the server constantly "polls" the drives as long as it is powered up.

All things being equal, even of the drives go idle there will be little or no thermal cycling. Mechanically maybe.

Brad and Steve would know, but if there is some sort of possibility of a contact issue developing, I wonder if an annual "pulling of the drives" might prevent it? So far in all the installs I have done and/or are aware of, have never had that issue arise. (Only a few random drives go bad and one (unrelated) case where a client cut power to the server and projector for a few months and bricked the enigma card in the CP2210).

Posted by Steve Guttag (Member # 268) on 02-11-2016, 12:50 PM:

I've never had that issue (yet). We do normally leave servers on 24/7 unless the venue is a weekend only other infrequent use where there would be much more non-use time than use-time.

The drives never spin down...they are at 7200RPM so long as power is supplied. The hardest thing for a HDD to do is spin up. In an Enterprise application and especially as a RAID...you wouldn't want the drives spinning up/down...etc. In a cinema server application you'd doubly wouldn't want them spinning up/down!

Posted by Leo Enticknap (Member # 534) on 02-11-2016, 01:23 PM:

Must have been constant vibration rather than cycling that worked at least one of them loose, then. It's been around four hours of operation since I cleaned the contacts and reseated them, with no hint of any glitch so far.

Agreed with Tony that doing this at least annually is probably a good idea, probably in the same maintenance session that the fans are cleaned out and the CMOS battery replaced.

The server is on a surge protector but not a UPS. I don't see any point, because if the power to the booth went out there is no way the show could continue (a UPS isn't going to keep a 6kW xenon bulb lit for very long!), and AFAIK there is no way to interface the DSS200 with a UPS, such that a failure of mains power would tell the server to perform an orderly shutdown in the time provided by the battery. We'd need a UPS that would keep the DSS200 alive for 20-30 hours to be sure of that (there are some nights on which we have no show and the theater is dark).

Furthermore, there is no way (that I know of) to tell the DSS200 to perform an orderly shutdown. If I ever need to, I give it the reboot command, and then when I hear the PSU fans surge, I'm guessing that no HDD activity is taking place and that it's safe to pull the power at that point.

I guess that if a power failure happened while one of us was in the booth, we could shut it down using this method, but if it happened, say, in the middle of the night, it is going to go through a disorderly shutdown anyways, and if we need to reinstall the system software and/or reingest the DCPs afterwards, then we'll just have to do that.

So, given that a UPS on the DSS200 would not prevent a power failure from resulting in show disruption or loss, and we would probably have no way of shutting the server down in an orderly way even if a projectionist was in the booth when the power went out, I can't see any point in using one.

Posted by Steve Guttag (Member # 268) on 02-11-2016, 02:02 PM:

The reason for the UPS is so that brown outs/momentary blips don't "hit" the server and also cause you to have to ensure another 7 minute boot up. The same goes for the projector. The goal is not to sustain a show (because the sound and the lamp are going to go out and the emergency lights in the theatre are going to go on). In the theatres we set up, we have an automation and have issue a pause the moment the power drops so that none of the movie is lost either.

If the power out happens in the middle of the night...so what...the Dolby server is perfectly happy to just drop out and when power comes back, it will come back on on its own.

Posted by Leo Enticknap (Member # 534) on 02-11-2016, 07:09 PM:

Thanks Steve - I must admit that this angle didn't occur to me. Will look in to putting that server on a UPS.

It's been working fine without any glitches or complaints all day, and so I'm convinced that a weak connection in one of the SATA interfaces was causing the problem I discovered this morning.

Posted by Steve Kraus (Member # 476) on 02-11-2016, 10:18 PM:

Also, if it's a full-time UPS, where power is constantly being converted to DC and the output is always from the inverter, it should provide isolation from power line glitches. The electronics side of my NEC NC1200C and the Dolby DSS200 are on such a UPS.