Film-Tech Forum ARCHIVE: The Film Freeze For a Few Second

This is topic The Film Freeze For a Few Second in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=003016

Posted by Ananta Rusdyanto (Member # 9811) on 04-28-2017, 09:39 AM:

Hi all,
I'm a newbie here, i have a problem with my projector. I operate a Barco DP2K-20C projector, the problem is when i play a film, the film freeze for a few time. The duration of freeze is about 10 to 20 seconds. To dix it, i have to raid the storage and re-ingest the film. It does fix the freeze problem, but if i ingest a new film without raid, the film freeze again.
Is there anyone had the same problem with me and is there any solution for that?please let know...

Thank's

Posted by Jack Ondracek (Member # 1466) on 04-28-2017, 09:50 AM:

We had a similar problem this year. However, with the complexities of these machines in mind, the causes of our respective problems may be different.

At the start of our season, we had our machines updated, which is customary. Usually, this causes no problems. This year however, we had problems with two projectors, where we'd go into encrypted content with no sound, or no picture, or perhaps both. Power cycling the projector was required to fix the problem.

Then, "Beauty and the Beast" froze on us, twice in the same performance. The freeze lasted longer than a few seconds... more like 30, but the show started itself back up on both occasions.

Long story short, it turned out that two of our IMBs did not like the latest update. Our service company regressed back to what we had last year and there have been no problems. Dolby wants our IMBs so they can study the problem. However, we're just out of warranty and they aren't offering anything for the opportunity to study our cards... we'd have to pay to replace them. Our service company sees no immediate problem with leaving well enough alone, for now, so the cards are staying put.

We also discovered that, coincidentally, we were experiencing progressively-worse failures in 3 of the 9 hard drives in our booth. We replaced all 9, taking the opportunity to install larger drives.

So far, no further issues.

Barco DP2K-23 and 32B projectors, Doremi servers.

Posted by Steve Guttag (Member # 268) on 04-28-2017, 10:30 AM:

By and large, if you are getting image freeze on shows, you are experiencing an underflow issue. That almost always points back to the RAID. So that is where I'd start...are the drives good? Check the SMART logs and see if you have any errors or high reallocated sector counts (over 100). The other thing to note is if the reallocated sector count is climbing at anything higher than 1 every couple of months (if that). If you are getting several a day, that drive is failing.

If the drives are good (and the SMART data is not a vindication of the drive but it can point to a bad one), then check on communication between the RAID and the mediblock (Ethernet cables, PCIe cables, etc.).

The logs though are your best friend to see what happened. The server should note if the data isn't coming fast enough to keep the should continuous.

Posted by Marco Giustini (Member # 4544) on 04-28-2017, 01:01 PM:

(You are in the wrong section, this belongs to Digital Cinema!)

What server are you using? What do you mean with "raid the storage", wipe it?
As Steve says it's likely an HDD failing. Unfortunately RAID arrays are not always smart enough to prevent that from happening.
And - apologies for the bad joke - make sure you're running the latest Java version [Wink]

Posted by Carsten Kurz (Member # 5396) on 04-28-2017, 02:25 PM:

A fried recently had that on a Dolby DSS200, stuttering during a show, fine after a reboot, reoccurred the next day, and so on. I told him to look at the drive LEDs, and the second drive was red. I told him to replace it, and the problem was gone.

- Carsten

Posted by Ananta Rusdyanto (Member # 9811) on 04-28-2017, 10:02 PM:

Thank's all for the explanation, like i said in the first place, i was a newbie here, so i apologize for my mistake.
For the information, we use a barco alchemy server.
Like Marco and Carsten said, I think one of the HDD is failing, because i have seen it before that the third HDD is red. After i raid them the HDD led back to normal, but i think the third HDD still failed and didn't run properly.
I'll try to change the HDD with the new one and see if the problem fixed with HDD replacement.

Thank's all.
Best Regards, Ananta Rusdyanto

Posted by Justin Hamaker (Member # 2165) on 04-28-2017, 11:02 PM:

I'm with Steve. Every time I have had issues with shows freezing, it's been because one or more RAID drives were failing. Some times a quick fix is to remove the failing drive from the RAID, but this is only a short term fix.

Posted by Mark Gulbrandsen (Member # 72) on 04-29-2017, 10:05 AM:

First note that all server manufacturers consider hard drives an expendable item like xenon lamps are. Only the drive manufacturers warranty usually applies.

You can look at the individual drive errors in the S.M.A.R.T. info in the servers Admin panel to determine which drives are failing. If your server is at the 5 year old mark just replacing them is the right thing to do, because they ARE at this point in time going to fail sooner than later. 5 years is pretty good for a set of drives running in RAID where they are constantly checking the data. I change out all drives in a complex at the 5 year mark irregardless and never have any issues this way.

I have seen poorer life (2 to 3 years average) out of the 2.5" HGST drives that GDC uses in the PDP-3000, and I am looking at other drive options to use in those. The factory drives are only 5400 RPM and the 7200 rpm HGST drives might be a more reliable option for just a few bucks more because those are also used in IT data servers.

Mark

Posted by Marco Giustini (Member # 4544) on 04-30-2017, 09:43 AM:

I still do not know what "after I raid them" means.

Posted by Mark Gulbrandsen (Member # 72) on 04-30-2017, 11:10 AM:

After I raid them means build the raid....

Posted by Marco Giustini (Member # 4544) on 04-30-2017, 05:52 PM:

re-initialise the whole RAID array???

Posted by Monte L Fullmer (Member # 2797) on 05-01-2017, 04:06 AM:

When I do Barco maintenance where I have to shut down the GDC then the projector, I also reboot the PSD-3000 while projector is down.

We do weekly reboots to eliminate these freezes. Otherwise, I'll get a house where the presentation will start to skip and freeze.

Yes, we have those 54K spin drives in our PSD units, but read that these drives are specially made, even though they are not enterprise drives, for cinema use.

I always wonder on buffer underrun with the slower spin drives.

-Monte

Posted by Jack Ondracek (Member # 1466) on 05-01-2017, 04:19 AM:

quote: Marco Giustini
re-initialise the whole RAID array???

Simple process, so long as you switch only one drive at a time. not much more difficult to do all 3 at once, though you'll lose your content if you do it that way.

In my case, I duplicate my content across all 3 of my servers, so I was able to replace all 3 drives at once. Took 4 or 5 hours to rebuild the RAID, then the rest of the day to re-ingest all the content from one of the other servers. Did all 3 servers over a couple of days that way.

Don't know if it would be less time to replace only one drive...

Posted by Marco Giustini (Member # 4544) on 05-01-2017, 05:37 AM:

Jack

I guess you mean re-build a RAID by removing a drive and re-installing it back in?
Just to be clear, this should NEVER be necessary - as you should not reboot a server weekly to avoid freezes.
Those are computers, they are supposed to work. If your picture freezes on a regular basis, there is a SW/HW problem which will need to be addressed.

Re-building the raid/re-ingesting will probably cause the content to end up on a different part of the platters, which is not affected by the fault.

Re-building/re-initialising is not the solution and in the long term will unnecessarily wear those drives out.

Posted by Steve Guttag (Member # 268) on 05-01-2017, 07:14 AM:

I would hope that the drives the PSD are using are "enterprise" grade RAID drives. Using desktop/laptop drives would be a recipe for failure. The "mentality" of a desktop/laptop drive over a raid drive is completely different. A raid drive presumes it is part of a raid so if it gets something unreadable, it marks it bad and moves on. A desktop/laptop drive presumes it is the only drive in use and will re-read for some time trying to recover the data and thereby hang a raid that is trying to read new data continuously.

Posted by Dave Macaulay (Member # 813) on 05-01-2017, 09:12 AM:

There are 2.5" enterprise drives, I haven't checked an ICMP (or IMS) but I doubt that Barco would use anything else.
However, drives fail. You just can't guess exactly when.

Posted by Marco Giustini (Member # 4544) on 05-01-2017, 11:47 AM:

but GDC's - those never fail [Wink]

Posted by Mark Gulbrandsen (Member # 72) on 05-01-2017, 02:24 PM:

Very rarely. And I can speak as to the realibility with 334 servers in the field and only two actual media block failures in 5 years. And actual drive failures are also rare because the software raid is very easy on the drives. ( Just the opposite on the SX-3000 that uses hardware RAID where I have had a half dozen drive failures). Something on the order of three or four actual RAID drive failures in the same time period on the big servers. GDC is famous for kicking drives out of the RAID but you just add them back in if the SMART errors are within reason. And keep going....

Dave, I last looked at an IMS about two years ago (Thankfully) and it had the same exact drives that GDC is using in the PDP-3000.

Mark

Posted by Carsten Kurz (Member # 5396) on 05-02-2017, 05:00 PM:

ICMPs does not use 2.5" enterprise drives, but general AV/video surveillance drives.
At least our ICMP came with those. Dolby has spec'd the same drive type for the IMS1000 and IMS2000. It seems the ICMP is not too picky about specific drives anyway.

Hard to tell wether these 'speciality' drives could be called enterprise drives as well. At least they are usually not built for maximum reliability. The term 'enterprise drive' is probably too generic nowadays.

Steve is right that RAID optimized drives 'give up earlier' on read errors to let the RAID redundancy take over - however, many consumer or standard desktop/notebook drives can be configured to do the same by using manufacturer specific software tools, so, they can be tailored for RAID operation if needed.

I have an old notebook running windows on an aging 2.5" drive that has some bad sectors. Sometimes it takes ages (minutes) for it to recover from read errors. You don't want that to happen on a DCI server.

- Carsten

Posted by Steve Guttag (Member # 268) on 05-03-2017, 08:53 AM:

Since I know that the ICMP uses HGST drives and you say they are from the A/V and Video Surveillance line...that makes them part of the "CinemastarC5K1000". They only go up to 1TB in that line but the ICMP offers a 2TB drive solution.

HGST's Ultrastar C10K1800 line, which is "Enterprise Grade" and rated for operating in a RAID, they go up to 1.8TB. At 10,000RPM, they book! However the rated ambient temperature is lower (only 55C versus 70C of the C5K1000) than the other line.

Posted by Jack Ondracek (Member # 1466) on 05-03-2017, 10:16 AM:

quote: Marco Giustini
I guess you mean re-build a RAID by removing a drive and re-installing it back in?

Sorry if I wasn't clear.

We had the underflow problem that Mark referenced. Rather than just replace the failing drive, I took the opportunity to replace them all and increase the size for more capacity.

For me, this was made easier as I have all of my content duplicated on my other servers. Starting from scratch on one by replacing all of the drives with larger ones didn't pose a problem in this case.

Posted by Carsten Kurz (Member # 5396) on 05-03-2017, 10:43 AM:

quote: Steve Guttag
They only go up to 1TB in that line but the ICMP offers a 2TB drive solution.

That is correct, our ICMP came with 1TB Cinemastar drives. These are relatively inexpensive drives compared to 'real' enterprise or RAID optimized drives. As the 2TB original BARCO option was way to expensive for our needs, I have no idea what kind of drives the 2TB kit uses. There are many 2TB notebook drives available now, and the ICMP drive slots actually support extra high 2.5" drives as well. 'Cinemastar' sounds like a dedicated DCI drive type or something, but they are just optimized for video streaming, surveillance/CCTV VTRs, etc.

- Carsten

Posted by Marco Giustini (Member # 4544) on 05-03-2017, 12:26 PM:

quote: Jack Ondracek
Starting from scratch on one by replacing all of the drives with larger ones didn't pose a problem in this case.

I understand that but it has to be made clear that re-initialising a RAID should never be necessary and even though that may fix the issue, that is not the solution.

Posted by Mattias Mattsson (Member # 4308) on 05-03-2017, 01:31 PM:

quote: Carsten Kurz
I have no idea what kind of drives the 2TB kit uses.

On the ICMP:s with a 3x2TB configuration I have seen, Seagate ST2000NX0253 drives are used.

Posted by Marcel Birgelen (Member # 6801) on 05-03-2017, 03:25 PM:

Who provided those CinemaStar disks? Your integrator I presume?

While those "surveillance disks" usually get tested a bit more rigorously if they can sustain the troughput, those CinemaStar disks don't run RAID-optimized firmware.

So you might be careful with using them in a RAID configuration. Especially in a DCI environment, where a single bad disk can get your performance down sufficiently to stop a show. RAID optimized disks usually have a far lower retry count and get ejected earlier by the controller than your average non-RAID disk.

Posted by Carsten Kurz (Member # 5396) on 05-03-2017, 04:02 PM:

quote: Marcel Birgelen
Who provided those CinemaStar disks? Your integrator I presume?

While those "surveillance disks" usually get tested a bit more rigorously if they can sustain the troughput, those CinemaStar disks don't run RAID-optimized firmware.

They are BARCO supplied. HGST has a detailed description, and they do seem to have RAID optimizations, at least the early-give-in feature that is needed for guaranteed response time ('Error Recovery Time Limits').

- Carsten

Posted by Monte L Fullmer (Member # 2797) on 05-03-2017, 06:45 PM:

These PSD drives are the HGST CinemaStar Drives, the C5K1000 series were designed for specific video storage.

Doesn't mention anything on Enterprise on the drive itself.

I've already replaced a few in the 3 years we've been opened.

- Monte

Posted by Ananta Rusdyanto (Member # 9811) on 05-07-2017, 04:13 AM:

If i want to change the drive with the new one, does anyone have a drive recommendation beside HGST hdd?

Posted by Carsten Kurz (Member # 5396) on 05-07-2017, 06:20 AM:

Are you current on software and technotes for the Barco alchemy? There have been some issues with the RAID that may be resolved without changing the drive.

If you read up this thread, you will find the two drive types used in the ICMP. If your drive is actually dead or needs to be replaced for other reasons, get the same type of drive as is already used in the ICMP. The 1TB Cinemastar is very affordable.

- Carsten

Posted by Dave Macaulay (Member # 813) on 05-07-2017, 08:25 AM:

Basically you don't just go to the local computer store and get whatever laptop drive they have on sale. The drives listed above are useb by Barco so presumably they are what it wants.
You will probably need to order these and they will cost more than basic laptop drives.

Posted by Steve Guttag (Member # 268) on 05-07-2017, 10:30 AM:

One would presume that unless you have the very first ICMPs that all of the drives would be in warranty (by Barco, if not by HGST). Why pay for what support should provide?

Posted by Mattias Mattsson (Member # 4308) on 05-08-2017, 07:54 AM:

quote: Steve Guttag
One would presume that unless you have the very first ICMPs that all of the drives would be in warranty

That's really good advice. The 1TB drives (C5K1000) have a 3 year warranty from HGST and the 2TB drives (ST2000NX0253) have 5 years from Seagate.

You can check warranty status here (serial numbers needed):
https://www.hgst.com/portal/site/en/support/warranty/
http://support.seagate.com/customer/en-US/warranty_validation.jsp

Posted by Mark Gulbrandsen (Member # 72) on 05-08-2017, 11:00 AM:

C5K1000 is 60.70 at Newegg...

Posted by Carsten Kurz (Member # 5396) on 05-08-2017, 04:30 PM:

You should not assume that OEM supplied drives are eligible for 'standard' warranties. Of course, you can always ask ;-)

As in this case, the Cinemastar is not exactly expensive, and the RAID needs to receive a replacement drive soon (as now there is no redundancy), I would buy one replacement, and THEN try to get a warranty replacement. So you have a replacement handy when the next drive fails.

If this system is used for daily showings, I wouldn't wait for a drive manufacturer to answer a warranty claim. It can take weeks. If another drive fails, this screen is out of business.

Of course, there is a difference if the ICMP/projector is still under warranty by BARCO. In that case, you should receive a replacement drive within days from your integrator.

- Carsten

Posted by Mattias Mattsson (Member # 4308) on 05-09-2017, 08:20 AM:

quote:
You should not assume that OEM supplied drives are eligible for 'standard' warranties. Of course, you can always ask ;-)

Of course not, and as you say, this is often the case.

But the warranty periods I stated above comes from me actually entering serial numbers in the manufacturers warranty forms (to see if they are still in warranty) from HDD:s delivered with new ICMP units from Barco.

Posted by Mark Gulbrandsen (Member # 72) on 05-09-2017, 08:26 AM:

quote: Carsten Kurz
If another drive fails, this screen is out of business.

Not true with the GDC SX-3000 because you can live play from the CRU ingest bay. I can not speak for the IMS because I simply refuse to even service any location that has them. But perhaps you would know if the IMS can live play or not.

Mark

Posted by Monte L Fullmer (Member # 2797) on 05-17-2017, 07:03 PM:

Agree. I had to do three live play houses for one week due to issues with corrupt issues with the drives in their respective PSD units.

Since I have GDC TMS, I had to do the HD Sled/USB connection to the SX3000 to do Live Play

Got sent replacement drives from GDC, installed and got these PSD units back up online.

Does any other server offer this option for direct HD playback?

Posted by Stephan Shelley (Member # 8587) on 05-17-2017, 07:51 PM:

Dolby DSS line does from the CRU bay on the 100 and 200 and with a sleeve and Esata on the 220. Doremi showvault has a load and play function.

Posted by Carsten Kurz (Member # 5396) on 05-17-2017, 08:13 PM:

Doremi Pingest from a confirmed network source only (not CRU), allow 10 minutes or one reel buffering before playback can start. I think the IMS1000 does not support it.

Sony R320 PlayWhileIngest (PWI) is a separate network ingest software module. It has a bunch of limitations and thus will only work for select features

The classic GDC servers will also live play from CRU. I know one cinema doing one-off art house screenings that way on a regular basis. Not a good idea, but it seems they rarely ran into trouble. Don't know it's limitations (like OV/VF issues).

- Carsten

Posted by Monte L Fullmer (Member # 2797) on 05-18-2017, 04:42 AM:

quote: Carsten Kurz
Don't know it's limitations (like OV/VF issues).

In my situations, as long as the key is stored in the server, the content, both OV and VF, are valid since the drive is mounted, then the server sees the drive mounted being as part of the system.

The content from the mounted HD shows up in the content list. Thus, the needed content can be added in the SPL.

Can do the same with a trailer drive if the operator wants to actually screen trailers for proper usage in the feature SPL: Mount the drive in the system so the content can be noticed in the trailers tab where this content can be added to the SPL build, SPL saved, then played back for viewing.