Film-Tech Forum ARCHIVE: GDC SX3000 - movie freezing randomly

This is topic GDC SX3000 - movie freezing randomly in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=003469

Posted by Mike Blakesley (Member # 26) on 10-15-2018, 12:38 PM:

We're seeing a new thing -- it's happened twice now. The movie just stops cold, with the picture frozen as if I had pressed the pause button, but it won't start playing again.

The first time it happened, the onscreen controls were completely locked -- the only solution was to power down the whole system and reboot from scratch.

Last night, the play/stop/pause buttons were locked, but the "shutdown" button at the bottom WOULD work, so I re-started the software and resumed the movie from where it was. That was a much quicker fix, only took a little over a minute.

Very irritating, and I wonder if anyone else has seen this. GDC says it's a pretty rare occurrence.

Posted by Sam D. Chavez (Member # 1841) on 10-15-2018, 01:21 PM:

Just a guess, but check the drives. You've probably done this already.

We had a case where the drives all checked out fine, low bad sectors etc, nonetheless, one drive would stall and freeze the image. Had one show that ran over 10 minutes long. Finally caught it in the act.

Posted by Mike Blakesley (Member # 26) on 10-15-2018, 01:59 PM:

The drives all report as OK but that is my first suspicion, too. They're going to pull the logs today and see if anything stands out.

Posted by Greg Routenburg (Member # 1742) on 10-16-2018, 08:03 AM:

I also find that the first order of business when troubleshooting playback issues with the SX-3000 is the reseat the eSATA cable both at the IMS and the storage device. It seems to be a weak point for these servers and often helps resolve most pesky playback issues.

Posted by Mark Gulbrandsen (Member # 72) on 10-16-2018, 12:29 PM:

The last time this happened they actually told us the PDP-3000 was defective and had us swap it out. But the last time the drives were what was wrong. But it makes me wonder if that PDP-3000 is actually bad. I foreworded you the email from support Mike.

Mark

Posted by Nguyen Jack (Member # 7530) on 10-17-2018, 04:37 AM:

Maybe the CompactFlash Card is problem, i got problem same lasttime.

Posted by David Meechan (Member # 8423) on 11-04-2018, 12:56 PM:

I helped open a new theatre back in April, using Barco and GDC. One of our projectors was skipping segments in the playlist, rewinding segments of the feature, and other gremlins.

Turned out to be a bad IMB (SX-3000) and once it was replaced, all was well. Now I have a random serial # in amongst all the sequential serial #'s - argh! [Wink]

Posted by Mark Gulbrandsen (Member # 72) on 11-04-2018, 01:08 PM:

Almost all the time it's a hard drive error issue in the PDP-3000, or a cable thats bad. eSATA is somewhat temperamental as it is... I personally don't care much for the drives they use in that thing. They are only 5400 RPM laptop drives... there are some 7200 RPM drives that will work fine and seem more reliable. I've only had to replace SX-3000's because thay lost their certs due to battery failure.

Mark

Posted by Steve Guttag (Member # 268) on 11-04-2018, 05:26 PM:

I've had every piece of the SX-3000 system fail...the cable, the PDP, the drives and the IMB itself. Fortunately, that unit is no longer an option, going forward.

Posted by Mark Gulbrandsen (Member # 72) on 11-05-2018, 06:24 PM:

Not here! In fact three of the first 12 units in this country are still humming away just fine in Wendover, Utah. Hard drives are also still original in two of those three units there. Of the 50 or so units I sold, about half have been exchanged due to the battery issue. My only issue with them are with the stupid Cert batteries!

Mark

Posted by Mike Blakesley (Member # 26) on 11-05-2018, 10:27 PM:

Beyond this recent movie-freezing issue (which was fixed by reseating everything and a firmware update...at least, it hasn't happened in a month now), ours has been fine. The only exception is those silly hard drives. I still don't understand why, if one drive is failing, the others don't just pick up the slack and light-up an indicator somewhere to replace the failing drive. Isn't that what a RAID is supposed to do??

Posted by Marcel Birgelen (Member # 6801) on 11-06-2018, 01:35 AM:

I think I've explained the behavior of Joe Average's RAID here before.

The problem we're facing is that the RAID systems being used in cinema playout servers are "simple" office RAID systems. They target on data reliability rather than guaranteed performance.

If you want to open that Word document from your network drive, you mostly don't care that you need to wait a second or two extra for it to open. It's more important it's still there.

So, a normal RAID system, if one of the disks encounters a reading or writing problem, it will retry to read or write that same block from the disk a few times, every time with a slight delay in between. In many cases, a failing disk will still comply and the RAID array will not push the disk out of the RAID. Only if it triggers a pretty high threshold, the RAID array will finally decide to dump the offending disk. But for any real-time application, some negative impact will already have occurred by then.

The reason why this is designed this way is easy to explain: Reliability. You don't want your office RAID array to drop disks too easily, or you may end up with a broken array pretty quickly and gone is all your data on top of it.

Now, for real-time applications, that rely on a constant performance from a storage array, those average RAID systems are, if we're being honest, not good enough. There are solutions for this: Many enterprise RAID storage arrays can be configured to ensure a constant throughput. But those systems are pretty expensive and therefore not being used in current cinema playout hardware...

Posted by Steve Guttag (Member # 268) on 11-06-2018, 09:01 AM:

Note, I distinction between Enterprise drives and desktop/laptop drives is that an Enterprise drive should NOT re-read a bad area, it should mark the sector bad and move on since the other drives in the RAID have the data. When this happens, though, you'll get a hiccup in the image because the RAID thought it had the data from the drive that has an issue. In a desktop/laptop drive, that is not the scheme because it cannot presume there are any other drives to back it up...it will re-read to get the information, which in real-time can cause glitches on screen.

Generally, when there are glitches on screen...something is up with one or more drives. There are indicators for which drives are having issues (reallocated sectors that are either high are increasing daily, ATA errors, read times that are notably different than the other drives in the RAID since they have to work as a team). I've had them even then still have issues.

I've had drives that appear to be okay cause boot-up issues too. I'm a pretty staunch Hitachi (HGST) supporter. I've had some Western Digital (which now owns HGST) cause the odd boot up issues though their current "Gold" series seem to be fine. Seagate...well, they've let me down the most.

Posted by Mark Gulbrandsen (Member # 72) on 11-06-2018, 09:35 AM:

quote: Mike Blakesley
I still don't understand why, if one drive is failing, the others don't just pick up the slack and light-up an indicator somewhere to replace the failing drive. Isn't that what a RAID is supposed to do??

Raid does not necessarily guarantee no read back errors if there are hard drive issues. What it does guarantee is data integrity. Once the bad drive is located and removed and the raid rebuilt minus the bad drive, or with a new replacement, only then will read back errors stop. In a 3-drive RAID 5 you can lose one drive and the data will still remain error free. Loose two drives and you lose all the data because you have lost the redundant data that's been striped across the other drive(s). A 4-drive RAID 5 you can lose up to two drives and still retain all data. The PDP-3000 can accommodate 4 hard drives and it is hardware RAID of some sort.
Some really high end hardware raid controllers can automatically insert a hot standby drive in place of a failed drive. But those cards are thousands of dollars by themselves.

Agree with Steve about the Enterprise drives. Unfortunately the 2.5" enterprise type hard drives are rather pricey. One would really need to go to SAS type drives to achieve extraordinary reliability. But then they are about double the price or more over Enterprise SATA drives, but SAS may be the last HD's you ever buy for a given unit. I have TMS servers that the OS and APPs are running on SAS drives and many of those are now 7+ years old and still showing no smart errors. Note that you can install a SATA drove in place of an SAS but you can not install a SAS drive in a SATA device because of very slight difference in the connector.

Mark

Posted by Steve Guttag (Member # 268) on 11-06-2018, 12:06 PM:

Sorry no...in a RAID-5, 3-drive, 4-drive, whatever drive, only one drive may be lost without loss of data. A RAID-6 allows two drives to be lost without loss of data (typically seen in LMS systems...that and RAID-50, which seem to be more popular with the larger RAIDs).

RAID-5 is known for its read-speed and hence it is popular with SMS players where timing is critical.

Some RAIDs are better than others. The 3WARE system that Dolby used on the DSS series was good about kicking out a drive causing issues.

Note, when a drive fails in a RAID-5...it doesn't rebuild on the remaining drives, it runs on those drives and uses the error correction to create the data that was on the missing drive. That RAID will run slower (loading content on, will really show it). Once the drive is replaced...THEN it rebuilds as the remaining drives provide the information to fill up the new drive (plus its portion of the redundancy storage).

Posted by Marcel Birgelen (Member # 6801) on 11-07-2018, 03:21 AM:

quote: Steve Guttag
Note, I distinction between Enterprise drives and desktop/laptop drives is that an Enterprise drive should NOT re-read a bad area, it should mark the sector bad and move on since the other drives in the RAID have the data.

Unfortunately, it's not entirely consistent across the line. At least not with the multitude of different types of disks I've seen go through across the years.

The biggest difference between "Enterprise" and "Desktop" drives is that Enterprise firmware ensures you the drive will not go to sleep after a certain period of idle time. Those sleep modes are obviously killing for RAID. That's why you most definitely want to avoid Desktop disks for any RAID solution.

But nowadays, there are many different types of drives, even across SATA drives, targeted at different applications. But most of those distinctions are pretty much bollocks, they make no noticeable difference. For example, I haven't really seen any improvement using "highest end" disks targeted at stuff like camera surveillance, compared to "vanilla" enterprise disks. Their failure modes seem to be identical and they also tend to slow an entire array down if they start to fail intermittently.

I've also tested disks that claimed to be "RAID optimized", but I've also seen the typical failure modes on those disks, where an intermittent failure leads to abysmal RAID performance.

(The biggest troubles I've had with this were actually on RAID arrays using SSDs, since SSDs are not so easy to debug and not always seem to log their internal struggles...)

The re-read instructions in SATA that are blocking are mostly initiated by the controller. A simple, recoverable error usually doesn't induce so much latency it really blocks the RAID array operation. The disk will silently relocate the bad block to another spare block. Most disks also have sufficient cache to do so during some idle loops.

Most enterprise systems don't run RAID-5 anymore. It's at the very least a double-parity system like RAID-6 or often a combination, like RAID-60 or a vendor specific RAID-like solution. Those kind of configurations give a lot more redundancy, but also require a bigger overhead and large disk count.

The biggest difference although, is inside the controllers. Those controllers are specifically built for those tasks and are highly expensive.

Modern storage manufacturers like EMC, NetApp, etc. allow you to configure an array for a specific task. Those controllers constantly monitor the throughput of the disks. Also, most of them are backed with a significant amount of non-volatile memory used as write buffer, which can be used as a buffer, if the performance of a disk suddenly drops. Within the time of this buffer, the array can decide whether or not to eject a disk from the array, without creating an impact on the applications atop of the array. Also, if you're running a double-parity or even triple-parity system, the potential impact on the availability and consistency of the data on the array is way less if you eject a single disk, than compared to a RAID-5 array, where after the second disk dies, you're in for a likely disaster.

Those enterprise systems also tend to employ stronger hardware than average RAID controllers, so that a missing disk, where missing data needs to be reconstructed from checksums doesn't lead to any noticeable impact. Many of those systems also don't wait for retries from the disk, instead they automatically start to reconstruct the data for the requested block and return this, if one of the disks starts to lag behind.

Posted by Mark Gulbrandsen (Member # 72) on 11-07-2018, 07:09 AM:

Ahhh hummmm.... You can indeed lose two drives from a PDP-3000 raid as long as it is allowed to rebuild each time you remove a drive. I have done it where there is more than one drive with high SMART errors... the drives they use are really crappy! Ditto for almost all other RAID cards out there unless there is a hot standby available, in which case it is then put on line. Many software raid systems will also kick drives out if there are high errors. GDC's is well known for doing that. Put that drive back in the array and if it gets kicked back out again then simply replace it. And when a drive was kicked out the "Degraded RAID" message pops up when you boot the server.

Mark

Posted by Steve Guttag (Member # 268) on 11-07-2018, 08:03 PM:

Again, if it is a RAID-5, it has a maximum 1 drive failure tolerance. Look it up if you don't believe me. RAID-6 can lose two drives.

Posted by Mark Gulbrandsen (Member # 72) on 11-07-2018, 08:18 PM:

You need to re-read the post above.

Posted by Steve Guttag (Member # 268) on 11-08-2018, 07:58 AM:

Here is what Mark says:

quote: Mark Gulbrandsen
A 4-drive RAID 5 you can lose up to two drives and still retain all data.

Here is what I say:

quote: Steve Guttag
Again, if it is a RAID-5, it has a maximum 1 drive failure tolerance. Look it up if you don't believe me. RAID-6 can lose two drives.

So Mark implies you can lose 2 drives when he gives an example where two drives have ATA errors. ATA errors != two failed drives. Failed drives are drives that will no longer or stay joined to the RAID, for whatever reason.

You can have ATA errors on all of the drives and it will still run.

Posted by Marcel Birgelen (Member # 6801) on 11-08-2018, 08:11 AM:

quote: Mark Gulbrandsen
Ahhh hummmm.... You can indeed lose two drives from a PDP-3000 raid as long as it is allowed to rebuild each time you remove a drive.

Well, to me it's also not entirely clear what you're saying here.

You can go through a degraded -> rebuild -> healthy -> degraded ... cycle as many times as you want, as long as you only lose one disk at a time.

If you lose a second disk in a RAID 5, what you can try with some controllers is to force the array on-line. Normally, the RAID controller will immediately stop all I/O to and from the array as soon as the second disk in your RAID-5 array gets ejected.

Not all controllers do have the option, but forcing the array on-line can give you a last resort of evacuating some remaining data from the array, as long as the last disk being ejected isn't completely dead. The problem though, is that as soon as you hit one of those sectors that got the disk ejected the last time, you will most likely see your array drop off-line again. I have recovered some essential data from some failed arrays this way.

The only other way to recover (parts) of your data otherwise from a failed array is sending your disks to a company specialized in reading the data directly from the platters with very specialized equipment. Those services can cost multiple tens of thousands of dollars, depending on the amount of data to be recovered.

Posted by Mark Gulbrandsen (Member # 72) on 11-08-2018, 08:17 AM:

No, not that post... my other post. You can indeed lose up to two drives if it's a four drive raid and if the raid is rebuilt between removing the drives. And one drive on a three drive system. I've done it dozens of times on the full size servers and probably a half dozen times on the PDP-3000. And on the PDP-3000 it is all automatic.

Mark

Posted by Monte L Fullmer (Member # 2797) on 11-12-2018, 02:15 AM:

We had a PSD go bad with the same issues.

Drives mentioned OK in the Control Panel

Replaced all drives and rebuilt the RAID

All good now.

Ya, I think those 5400 spin drives are not the best around. Should be 7200 spin drives.

Data stream, plus, where we have ATMOS, seems it can't keep the cache filled quick enough with 5400 spin drives.

-Monte

Posted by Steve Guttag (Member # 268) on 11-12-2018, 06:58 AM:

Again, if it is a RAID-5, you can't lose more than 1 drive (3 or 4 drive, no matter what Mark says). If he can lose 2 drives (physically remove them from the chassis) then it isn't a RAID-5.

Posted by Mark Gulbrandsen (Member # 72) on 11-12-2018, 08:55 AM:

quote: Monte L Fullmer
Drives mentioned OK in the Control Panel

Replaced all drives and rebuilt the RAID

You can't just go by the OK signal in Control panel if you are running PSD-3000 boxes. You have to get the logs and send them to GDC and have them check the drives SMART Errors. It's also advisable to write down the date and time the error happened so they can also look back then to see of anything else is wrong.

Steve Guttag... This is what I said...

quote: Mark Gulbrandsen
Ahhh hummmm.... You can indeed lose two drives from a PDP-3000 raid as long as it is allowed to rebuild each time you remove a drive.

Removing two drives assumes you have a four drive RAID, which most of my customers do. You can also remove one drive from a 3 drive RAID and still play back from the PSD-3000. Will a DSS-200 play back sporting only two data drives? In short, no. The key here is letting the RAID rebuild before removing the next drive... if you have a three drive box and power all stuff down and pull one drive it rebuilds in the background while you are on screen. You can remove one drive from a four drive RAID and still have a RAID 5. Remove two drives from the 4 Drive box and I can only assume it brings it down to a RAID 1. I never inquired as what RAID # it is running at on two drives, but I will. Doesn't really matter, because what ever it is the system still plays back just fine. And when you install the replacement drive it also automatically rebuilds back to what it was. I don't think Dolby can play back at all on just two data drives. GDC's full size servers can. Mike B has had to do this and run on two drives.. he is running a 3-Drive box, and I did it with a customer in Wendover, Utah two nights ago who will receive his replacement drive tomorrow.

Mark

Posted by Steve Guttag (Member # 268) on 11-12-2018, 09:37 AM:

They only way what you say can be true is that it is not a RAID-5.

As for the Dolby DSS series, it is a RAID-5 so, by definition, it can only survive a single drive failure.

Now, comparing to the SX-3000 drive system on the PSD...I'll take the Dolby RAID ANY day and so would most anyone else. What is the storage failure rate on a DSS server versus the PSD? something like a 1000 failures on the PSD to one on the DSS servers?

You can run 4 mismatched drives on the DSS server without issue (ask Brad).

I don't think I've ever had a DSS server crash its RAID (if i have...it has been a long time ago and I guarantee it was Seagate related). I often just change out the odd bad drive that ages out, the server just keeps marching along.

Posted by Mark Gulbrandsen (Member # 72) on 11-12-2018, 09:49 AM:

Actually, the PSD RAID has been totally reliable for me, and it too is a hardware RAID that is capable of almost completely automatic operation. Now that is great for the customer but bad for lack of technician service calls. Now, the cheapo HGST drives they chose to send out in them are another story. They are garbage, as is the eSATA interface itself. The PSD RAID would be awesome interfaced a better way, say through PCIe or similar. But since GDC has switched to cache data storage and no longer has a need for any RAID storage in their current IMB offerings, RAID is a dead issue there. Still alive in the outboard storage and TMS world though.

Mark

Posted by Steve Guttag (Member # 268) on 11-12-2018, 11:12 AM:

quote: Mark Gulbrandsen
Actually, the PSD RAID has been totally reliable for me

Of course it is, Mark, for you...

Never mind that for you to discuss about how many drives you have had fail or systems to have been changed out...would indicate failures.

On the HGST drives...they are probably being ask to operate in a manner inconsistent with their manufacture. It isn't the drive's fault.

eSATA is a dodgy connector, at best. Possibly the only thing worse than micro-USB or HDMI. Most cables for eSATA suck too. They could probably switch to USB3 and get more reliability, at this point. Many people that have the SX-3000 have also switched over to the "Enterprise" storage, which would solve your crappy HGST surveillance drives.

Posted by Mark Gulbrandsen (Member # 72) on 11-12-2018, 12:00 PM:

Well, out of 54 units installed I've had just one actual failure... Network stuff died on the thing. Other than that it's been cert batteries that I couldn't get to in time, or at theaters that prefer to do it themselves. Not unlike the Dolby Amnesia that is beginning to set in on many CAT 745's. But not a single PDP-3000 box has died. You have the option to replace the original drives with Enterprise drives if you want to.

Mark