Film-Tech Cinema Systems
Film-Tech Forum ARCHIVE


  
my profile | my password | search | faq & rules | forum home
  next oldest topic   next newest topic
» Film-Tech Forum ARCHIVE   » Operations   » Digital Cinema Forum   » GDC SX3000 - movie freezing randomly (Page 2)

 
This topic comprises 2 pages: 1  2 
 
Author Topic: GDC SX3000 - movie freezing randomly
Marcel Birgelen
Film God

Posts: 3357
From: Maastricht, Limburg, Netherlands
Registered: Feb 2012


 - posted 11-07-2018 03:21 AM      Profile for Marcel Birgelen   Email Marcel Birgelen   Send New Private Message       Edit/Delete Post 
quote: Steve Guttag
Note, I distinction between Enterprise drives and desktop/laptop drives is that an Enterprise drive should NOT re-read a bad area, it should mark the sector bad and move on since the other drives in the RAID have the data.
Unfortunately, it's not entirely consistent across the line. At least not with the multitude of different types of disks I've seen go through across the years.

The biggest difference between "Enterprise" and "Desktop" drives is that Enterprise firmware ensures you the drive will not go to sleep after a certain period of idle time. Those sleep modes are obviously killing for RAID. That's why you most definitely want to avoid Desktop disks for any RAID solution.

But nowadays, there are many different types of drives, even across SATA drives, targeted at different applications. But most of those distinctions are pretty much bollocks, they make no noticeable difference. For example, I haven't really seen any improvement using "highest end" disks targeted at stuff like camera surveillance, compared to "vanilla" enterprise disks. Their failure modes seem to be identical and they also tend to slow an entire array down if they start to fail intermittently.

I've also tested disks that claimed to be "RAID optimized", but I've also seen the typical failure modes on those disks, where an intermittent failure leads to abysmal RAID performance.

(The biggest troubles I've had with this were actually on RAID arrays using SSDs, since SSDs are not so easy to debug and not always seem to log their internal struggles...)

The re-read instructions in SATA that are blocking are mostly initiated by the controller. A simple, recoverable error usually doesn't induce so much latency it really blocks the RAID array operation. The disk will silently relocate the bad block to another spare block. Most disks also have sufficient cache to do so during some idle loops.

Most enterprise systems don't run RAID-5 anymore. It's at the very least a double-parity system like RAID-6 or often a combination, like RAID-60 or a vendor specific RAID-like solution. Those kind of configurations give a lot more redundancy, but also require a bigger overhead and large disk count.

The biggest difference although, is inside the controllers. Those controllers are specifically built for those tasks and are highly expensive.

Modern storage manufacturers like EMC, NetApp, etc. allow you to configure an array for a specific task. Those controllers constantly monitor the throughput of the disks. Also, most of them are backed with a significant amount of non-volatile memory used as write buffer, which can be used as a buffer, if the performance of a disk suddenly drops. Within the time of this buffer, the array can decide whether or not to eject a disk from the array, without creating an impact on the applications atop of the array. Also, if you're running a double-parity or even triple-parity system, the potential impact on the availability and consistency of the data on the array is way less if you eject a single disk, than compared to a RAID-5 array, where after the second disk dies, you're in for a likely disaster.

Those enterprise systems also tend to employ stronger hardware than average RAID controllers, so that a missing disk, where missing data needs to be reconstructed from checksums doesn't lead to any noticeable impact. Many of those systems also don't wait for retries from the disk, instead they automatically start to reconstruct the data for the requested block and return this, if one of the disks starts to lag behind.

 |  IP: Logged

Mark Gulbrandsen
Resident Trollmaster

Posts: 16657
From: Music City
Registered: Jun 99


 - posted 11-07-2018 07:09 AM      Profile for Mark Gulbrandsen   Email Mark Gulbrandsen   Send New Private Message       Edit/Delete Post 
Ahhh hummmm.... You can indeed lose two drives from a PDP-3000 raid as long as it is allowed to rebuild each time you remove a drive. I have done it where there is more than one drive with high SMART errors... the drives they use are really crappy! Ditto for almost all other RAID cards out there unless there is a hot standby available, in which case it is then put on line. Many software raid systems will also kick drives out if there are high errors. GDC's is well known for doing that. Put that drive back in the array and if it gets kicked back out again then simply replace it. And when a drive was kicked out the "Degraded RAID" message pops up when you boot the server.

Mark

 |  IP: Logged

Steve Guttag
We forgot the crackers Gromit!!!

Posts: 12814
From: Annapolis, MD
Registered: Dec 1999


 - posted 11-07-2018 08:03 PM      Profile for Steve Guttag   Email Steve Guttag   Send New Private Message       Edit/Delete Post 
Again, if it is a RAID-5, it has a maximum 1 drive failure tolerance. Look it up if you don't believe me. RAID-6 can lose two drives.

 |  IP: Logged

Mark Gulbrandsen
Resident Trollmaster

Posts: 16657
From: Music City
Registered: Jun 99


 - posted 11-07-2018 08:18 PM      Profile for Mark Gulbrandsen   Email Mark Gulbrandsen   Send New Private Message       Edit/Delete Post 
You need to re-read the post above.

 |  IP: Logged

Steve Guttag
We forgot the crackers Gromit!!!

Posts: 12814
From: Annapolis, MD
Registered: Dec 1999


 - posted 11-08-2018 07:58 AM      Profile for Steve Guttag   Email Steve Guttag   Send New Private Message       Edit/Delete Post 
Here is what Mark says:

quote: Mark Gulbrandsen
A 4-drive RAID 5 you can lose up to two drives and still retain all data.
Here is what I say:

quote: Steve Guttag
Again, if it is a RAID-5, it has a maximum 1 drive failure tolerance. Look it up if you don't believe me. RAID-6 can lose two drives.
So Mark implies you can lose 2 drives when he gives an example where two drives have ATA errors. ATA errors != two failed drives. Failed drives are drives that will no longer or stay joined to the RAID, for whatever reason.

You can have ATA errors on all of the drives and it will still run.

 |  IP: Logged

Marcel Birgelen
Film God

Posts: 3357
From: Maastricht, Limburg, Netherlands
Registered: Feb 2012


 - posted 11-08-2018 08:11 AM      Profile for Marcel Birgelen   Email Marcel Birgelen   Send New Private Message       Edit/Delete Post 
quote: Mark Gulbrandsen
Ahhh hummmm.... You can indeed lose two drives from a PDP-3000 raid as long as it is allowed to rebuild each time you remove a drive.
Well, to me it's also not entirely clear what you're saying here.

You can go through a degraded -> rebuild -> healthy -> degraded ... cycle as many times as you want, as long as you only lose one disk at a time.

If you lose a second disk in a RAID 5, what you can try with some controllers is to force the array on-line. Normally, the RAID controller will immediately stop all I/O to and from the array as soon as the second disk in your RAID-5 array gets ejected.

Not all controllers do have the option, but forcing the array on-line can give you a last resort of evacuating some remaining data from the array, as long as the last disk being ejected isn't completely dead. The problem though, is that as soon as you hit one of those sectors that got the disk ejected the last time, you will most likely see your array drop off-line again. I have recovered some essential data from some failed arrays this way.

The only other way to recover (parts) of your data otherwise from a failed array is sending your disks to a company specialized in reading the data directly from the platters with very specialized equipment. Those services can cost multiple tens of thousands of dollars, depending on the amount of data to be recovered.

 |  IP: Logged

Mark Gulbrandsen
Resident Trollmaster

Posts: 16657
From: Music City
Registered: Jun 99


 - posted 11-08-2018 08:17 AM      Profile for Mark Gulbrandsen   Email Mark Gulbrandsen   Send New Private Message       Edit/Delete Post 
No, not that post... my other post. You can indeed lose up to two drives if it's a four drive raid and if the raid is rebuilt between removing the drives. And one drive on a three drive system. I've done it dozens of times on the full size servers and probably a half dozen times on the PDP-3000. And on the PDP-3000 it is all automatic.

Mark

 |  IP: Logged

Monte L Fullmer
Film God

Posts: 8367
From: Nampa, Idaho, USA
Registered: Nov 2004


 - posted 11-12-2018 02:15 AM      Profile for Monte L Fullmer   Email Monte L Fullmer   Send New Private Message       Edit/Delete Post 
We had a PSD go bad with the same issues.

Drives mentioned OK in the Control Panel

Replaced all drives and rebuilt the RAID

All good now.

Ya, I think those 5400 spin drives are not the best around. Should be 7200 spin drives.

Data stream, plus, where we have ATMOS, seems it can't keep the cache filled quick enough with 5400 spin drives.

-Monte

 |  IP: Logged

Steve Guttag
We forgot the crackers Gromit!!!

Posts: 12814
From: Annapolis, MD
Registered: Dec 1999


 - posted 11-12-2018 06:58 AM      Profile for Steve Guttag   Email Steve Guttag   Send New Private Message       Edit/Delete Post 
Again, if it is a RAID-5, you can't lose more than 1 drive (3 or 4 drive, no matter what Mark says). If he can lose 2 drives (physically remove them from the chassis) then it isn't a RAID-5.

 |  IP: Logged

Mark Gulbrandsen
Resident Trollmaster

Posts: 16657
From: Music City
Registered: Jun 99


 - posted 11-12-2018 08:55 AM      Profile for Mark Gulbrandsen   Email Mark Gulbrandsen   Send New Private Message       Edit/Delete Post 
quote: Monte L Fullmer
Drives mentioned OK in the Control Panel

Replaced all drives and rebuilt the RAID

You can't just go by the OK signal in Control panel if you are running PSD-3000 boxes. You have to get the logs and send them to GDC and have them check the drives SMART Errors. It's also advisable to write down the date and time the error happened so they can also look back then to see of anything else is wrong.

Steve Guttag... This is what I said...
quote: Mark Gulbrandsen
Ahhh hummmm.... You can indeed lose two drives from a PDP-3000 raid as long as it is allowed to rebuild each time you remove a drive.
Removing two drives assumes you have a four drive RAID, which most of my customers do. You can also remove one drive from a 3 drive RAID and still play back from the PSD-3000. Will a DSS-200 play back sporting only two data drives? In short, no. The key here is letting the RAID rebuild before removing the next drive... if you have a three drive box and power all stuff down and pull one drive it rebuilds in the background while you are on screen. You can remove one drive from a four drive RAID and still have a RAID 5. Remove two drives from the 4 Drive box and I can only assume it brings it down to a RAID 1. I never inquired as what RAID # it is running at on two drives, but I will. Doesn't really matter, because what ever it is the system still plays back just fine. And when you install the replacement drive it also automatically rebuilds back to what it was. I don't think Dolby can play back at all on just two data drives. GDC's full size servers can. Mike B has had to do this and run on two drives.. he is running a 3-Drive box, and I did it with a customer in Wendover, Utah two nights ago who will receive his replacement drive tomorrow.

Mark

 |  IP: Logged

Steve Guttag
We forgot the crackers Gromit!!!

Posts: 12814
From: Annapolis, MD
Registered: Dec 1999


 - posted 11-12-2018 09:37 AM      Profile for Steve Guttag   Email Steve Guttag   Send New Private Message       Edit/Delete Post 
They only way what you say can be true is that it is not a RAID-5.

As for the Dolby DSS series, it is a RAID-5 so, by definition, it can only survive a single drive failure.

Now, comparing to the SX-3000 drive system on the PSD...I'll take the Dolby RAID ANY day and so would most anyone else. What is the storage failure rate on a DSS server versus the PSD? something like a 1000 failures on the PSD to one on the DSS servers?

You can run 4 mismatched drives on the DSS server without issue (ask Brad).

I don't think I've ever had a DSS server crash its RAID (if i have...it has been a long time ago and I guarantee it was Seagate related). I often just change out the odd bad drive that ages out, the server just keeps marching along.

 |  IP: Logged

Mark Gulbrandsen
Resident Trollmaster

Posts: 16657
From: Music City
Registered: Jun 99


 - posted 11-12-2018 09:49 AM      Profile for Mark Gulbrandsen   Email Mark Gulbrandsen   Send New Private Message       Edit/Delete Post 
Actually, the PSD RAID has been totally reliable for me, and it too is a hardware RAID that is capable of almost completely automatic operation. Now that is great for the customer but bad for lack of technician service calls. Now, the cheapo HGST drives they chose to send out in them are another story. They are garbage, as is the eSATA interface itself. The PSD RAID would be awesome interfaced a better way, say through PCIe or similar. But since GDC has switched to cache data storage and no longer has a need for any RAID storage in their current IMB offerings, RAID is a dead issue there. Still alive in the outboard storage and TMS world though.

Mark

 |  IP: Logged

Steve Guttag
We forgot the crackers Gromit!!!

Posts: 12814
From: Annapolis, MD
Registered: Dec 1999


 - posted 11-12-2018 11:12 AM      Profile for Steve Guttag   Email Steve Guttag   Send New Private Message       Edit/Delete Post 
quote: Mark Gulbrandsen
Actually, the PSD RAID has been totally reliable for me
Of course it is, Mark, for you...

Never mind that for you to discuss about how many drives you have had fail or systems to have been changed out...would indicate failures.

On the HGST drives...they are probably being ask to operate in a manner inconsistent with their manufacture. It isn't the drive's fault.

eSATA is a dodgy connector, at best. Possibly the only thing worse than micro-USB or HDMI. Most cables for eSATA suck too. They could probably switch to USB3 and get more reliability, at this point. Many people that have the SX-3000 have also switched over to the "Enterprise" storage, which would solve your crappy HGST surveillance drives.

 |  IP: Logged

Mark Gulbrandsen
Resident Trollmaster

Posts: 16657
From: Music City
Registered: Jun 99


 - posted 11-12-2018 12:00 PM      Profile for Mark Gulbrandsen   Email Mark Gulbrandsen   Send New Private Message       Edit/Delete Post 
Well, out of 54 units installed I've had just one actual failure... Network stuff died on the thing. Other than that it's been cert batteries that I couldn't get to in time, or at theaters that prefer to do it themselves. Not unlike the Dolby Amnesia that is beginning to set in on many CAT 745's. But not a single PDP-3000 box has died. You have the option to replace the original drives with Enterprise drives if you want to.

Mark

 |  IP: Logged



All times are Central (GMT -6:00)
This topic comprises 2 pages: 1  2 
 
   Close Topic    Move Topic    Delete Topic    next oldest topic   next newest topic
 - Printer-friendly view of this topic
Hop To:



Powered by Infopop Corporation
UBB.classicTM 6.3.1.2

The Film-Tech Forums are designed for various members related to the cinema industry to express their opinions, viewpoints and testimonials on various products, services and events based upon speculation, personal knowledge and factual information through use, therefore all views represented here allow no liability upon the publishers of this web site and the owners of said views assume no liability for any ill will resulting from these postings. The posts made here are for educational as well as entertainment purposes and as such anyone viewing this portion of the website must accept these views as statements of the author of that opinion and agrees to release the authors from any and all liability.

© 1999-2020 Film-Tech Cinema Systems, LLC. All rights reserved.