Film-Tech Forum ARCHIVE: freezing

This is topic freezing in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=003775

Posted by Armand Daiguillon (Member # 10334) on 08-25-2019, 11:50 PM:

Hi all,

Today I had a movie that kept freezing and unfreezing on the screen to the point that it became unwatchable. Shutting it down and rebooting fixed it for a while ...but then the problem happened again. (We have NEC900c's)
I called our tech support people at their NOC and they logged in, downloaded logs which they then sent to GDC ...GDC got back to them and they said that some drives in the RAID were bad. The support people then said they'd contact me with a fix (still waiting...lol)
So I opened up the media block and found just regular SATA notebook drives.
I was a computer tech for a few years and have done my fair share of drive testing, so I took out the 3 drives and am testing them, 2 of the 3 are at full capacity with data, but the 3rd has no data on it at all. I tested the one with no data and found no bad sectors or other issues with the drive. (I am testing the remaining 2 right now)
So, my first question is - why does one have no data (but seems fine) What would cause that ?
I've been told to always replace all three drives at the same time with identical drives if needed.
Is there a reason why an SSD would not be better than these regular drives ? Spec-wise they are no different than what I can get at Best Buy. Is there any reason I cant get these at Best Buy (same capacity and transfer rate was whats in there now) and are SSD's an option for the future ?

Your thoughts are appreciated [Smile]

Posted by Marcel Birgelen (Member # 6801) on 08-26-2019, 01:04 AM:

Playback freezes are commonly caused by one or more bad drives in the RAID array. It's not that common that all drives are affected at the same time and usually, ejecting the defective disk and replacing it with a new one should solve the problem.

If you clearly can identify the offending disk and the other ones are still good and they're no older than 3 or so years, I don't see a reason to replace all three of them.

Please note that whatever software you're using may not understand RAID arrays and might simply not be able to see the data on the disk. If there is really zero data on the disk, then this disk was probably never part of the array or failed very soon after the initial RAID array was made.

The disks are indeed standard 2.5 or 3.5 inch SATA drives, depending on your server type. Although usually they're of the "Enterprise" or "RAID optimized" kind. While this is often nothing more than a bunch of different firmware on them, it's an important difference. Also, you can't just replace the disks with any odd disks from your local supplier, please check with GDC which disks are supported and stick to those disks. Depending on the kind of server/IMS you have, there might also be SSDs on the list of supported disks.

While other disks MAY work, keep in mind that if you run into trouble again, it might be very hard to get adequate support from the supplier, GDC, in this case.

Regarding SSDs and their improved transfer and access rate. Keep in mind that both of those properties are only of limited value for the average DCI playout system. The real bottleneck is often somewhere else. If you're ingesting content from a disk, you can't ingest faster than what that single disk can handle. If you're transferring files over the network, the network or the device on the other side may easily be the bottleneck.

Also, SSDs come with their own, more unpredictable failure modes. Many RAID implementations are also not really optimized for usage with SSDs.

Posted by Armand Daiguillon (Member # 10334) on 08-26-2019, 03:07 AM:

Thanks for the info [Smile]

I'm puzzled though as to why all 3 have tested as "good drives" with no bad sectors - yet one has no info and the other 2 are full. Is there something else that could be a factor here ?
I would have expected the one with no data to have bad sectors ...but it seems fine.....?
Also - shouldn't 2 out 3 drives operated just fine without the freeze ups ?
[I miss film ....] [Smile]

Posted by Carsten Kurz (Member # 5396) on 08-26-2019, 06:52 AM:

How did you test these drives? What methods did you use to find out the third drive is 'empty'?

Can you tell us the exact model numbers?

Freezing is indeed very often caused by bad drives, and checking for bad blocks is not sufficient to find a bad behaving drive. At some point, these drives report data or errors back with a delay too long for the server to maintain playback. This is a very typical behavior for aging notebook drives. The server logs should tell which drive is affected. If these are common drives, get a replacement and try it. I wouldn't choose SSDs for this application. While they usually work in standard computer environments, you should stick with the types the server/RAID manufacturer suggests.

- Carsten

Posted by Mark Gulbrandsen (Member # 72) on 08-26-2019, 09:29 AM:

Those notebook drives are not IT grade drives, just consumer grade. As such, they should ALL automatically be replaced every three years. You can get IT grade 2.5 inch drives that will last much longer for not a lot more money and that is what I have been doing for my customers. I reccommend that you do the same. They are easy to get. I have not tried SSD's in that system and if you do please let us know how it works out. I know that GDC does not reccomend it. It is also stranger than fiction that there is no data on the third drive and there most assuredly should be data there if the RAID is working properly.

Mark

Posted by Marco Giustini (Member # 4544) on 08-26-2019, 12:26 PM:

Armand
Bad sectors are not the only issue a drive could exhibit. I suppose GDC has found evidence in the logs pointing to a specific drive.
In my experience bad sectors may well appear after a while after the drive has struggled to provide data to the system.
Or, again, it could be some other sort of error. Unfortunately a clean SMART table is not guarantee of perfect operations.

Posted by Mark Gulbrandsen (Member # 72) on 08-26-2019, 01:07 PM:

Modern drives that utilize SMART log those bad sectors so they are never written to again. SSD's do much the same thing.

Mark

Posted by Marcel Birgelen (Member # 6801) on 08-26-2019, 01:45 PM:

Drives don't use the SMART log for that, they save it on the disk themselves. This has been done since the first "intelligent" disks came to market, even back in time when we still had IDE and SCSI disks.

Before that, your drive actually came with a list of bad sectors you had to feed into your hard drive controller.

Nowadays, practically every hard drive, even those directly from the factory, do have a few bad sectors, but they compensate those by remapping. So, if you scan a disk for bad sectors, you won't find any, but there are almost certainly some physically degraded sectors on the platters.

SMART is a later invention (although since about a year or 20 almost standard on any drive) and is intended to give you an indication of the longevity of a drive. It logs the amount of remappings your drive has done and usually also logs other problems like read errors, tracking errors and spin-up errors.

In general, the amount of bad sectors on a disk isn't really a reliable indication of the performance, but the delta over time is the most important indicator of trouble to come. A rapid rising number of bad sectors is a good indication of a drive close to failure.

Now, like Carsten indicated, even a drive that reports zero bad sectors after a surface scan, can be problematic in a RAID array. Because this software often doesn't measure the time that's needed to get data from a certain sector. But for a DCI movie to play correctly, the RAID array needs to achieve a certain constant throughput. Normally, a drive will retry multiple times to read data from the platter, but every re-read will incur a huge performance hit. This can cause the performance to drop to a level, where constant playback cannot be sustained.

Posted by Mark Gulbrandsen (Member # 72) on 08-26-2019, 08:49 PM:

Great explanatipon Marcel! The term I could not remember was remapping. I sure don't ever remember getting a list of bad sectors with drives to enter, going as far back as 1992 though.

Mark

Posted by Marcel Birgelen (Member # 6801) on 08-27-2019, 02:13 AM:

By 1992 most of the newer disks that were sold were "intelligent" IDE disks.

You can see the label of an old-school MFM disk here.

Those tables were often filled in by hand in the factory. Just like it was "acceptable" to have a few bad pixels in a new TFT for quite a while, a certain amount of bad sectors was acceptable for any new drive.

Posted by Mark Gulbrandsen (Member # 72) on 08-27-2019, 10:07 AM:

Yep! I've only ever seen those types of drives at surplus stores.

Mark

Posted by Armand Daiguillon (Member # 10334) on 08-27-2019, 10:27 PM:

Wow. I cant believe you mentioned MFM drives. lol - thats a blast from the past [Smile]

I remember thinking I was so cool back in the day with a 40 gb mfm hard drive. (noisy thing [Smile]

Posted by Frank Cox (Member # 6258) on 08-27-2019, 10:52 PM:

If you had put a RLL controller on that 40mb mfm drive you would have had a 60 mb hard drive!

I did that with a 5mb full-height (two bays) 5.25" hard drive that I used to keep Fidonet echomail on. Got 7mb out of it with a RLL controller card, and I remember doing something fancy with the sector sizes to maximize the used space for small files since that's what I had on there and "regular" sectors would have a lot of slack. There must have been some reason why I didn't do it with a database instead of having all of those individual files, but I can't remember what it was.

Posted by Armand Daiguillon (Member # 10334) on 08-27-2019, 11:01 PM:

Ok, so I got the new drives and installed them, I called support to configure the new RAID - it was a lengthy process ....
After they did their thing I asked the support people for the password to get into the config section (GDC) so that I could do what they did if needed in the future (or at least check the RAID to identify issues) - I was told they could not give me the password because they weren't allowed. I paid a lot for these projectors and feel I should be able to access whatever I like, I can understand that damage can be done if people do the wrong things with these passwords - but is that really a gdc policy, that only they can log into the config sections ? (or even look at logs, check raid status...all that stuff)

Posted by Marco Giustini (Member # 4544) on 08-28-2019, 02:08 AM:

It is unfortunately.
From my understanding of your OS drive fails you’ll have to get a new from them - or have it configured - as there is no user procedure to just reinstall the system.

Posted by Steve Guttag (Member # 268) on 08-28-2019, 02:39 AM:

It depends on the level of access. The root level is GDC's alone. And yes, GDC encrypts their logs so only they can read them.

They have an app if you change the OS Hard drive or motherboard (same app and there is a wizard to step you through the change). If a change involves a serial number, only they can do that, for obvious reasons.

As for changing RAID drives, there is an obtainable password to gain access to the administration area for servicing the RAID.

You have to remember, the tech at the other end of the phone doesn't know you and if you aren't an authorized dealer or trained GDC tech, the passwords aren't going to be issued and that isn't just a GDC thing. Anyone can claim to "own the equipment" to obtain passwords.

I've found that, generally, if there is a legitimate need for a password, there is a path to obtaining it through channels. For those passwords at the top most level, even if they are not issued, there is a means to obtaining the service required. While you may own the equipment, you, at best, are only licensing the software.

Posted by Mark Gulbrandsen (Member # 72) on 08-28-2019, 05:32 PM:

LOL! Steve, he doesn't even know there is a root level access. Oh well... now he does.

But at any rate they won't give passwords out to anyone that has not attended a training seminar and signed an NDA. If he attends one they wll give him the passwords he needs there. If he can not attend a training gig, then he should consult his dealer for the Admin and Config passwords and how and when to use them.

As far as the RAID configuration goes, assuming you have a PDP-3000 external RAID enclosure, it's pretty much automatic after you press a button, for which you need the admin password to access...

Talk to GDC and find out if they are planning any training seminars either at Showyeast. Or at Cinema Con next year. They have always been free.

Mark