Film-Tech Forum ARCHIVE: GDC Ingestion "Exceptions"

This is topic GDC Ingestion "Exceptions" in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=002394

Posted by Jim Cassedy (Member # 4115) on 08-20-2015, 10:24 AM:

At one location, I deal with 4 GDC SX-2000A servers and a GDC TMS.
Occasionally, I find I cannot ingest a movie into the TMS and distribute it
to the 4 servers, without getting "exception" messages. Doing a file
verification gives me a 'hash mismatch' error on both the TMS and servers.

This happens only very infrequently, and usually with DCP's from independent
producers or distributors. But last night I found I cannot ingest "American Ultra"
into the TMS or distribute it to the auditoriums without getting 'hash mismatch'
exception errors.

(all ingestion is being done through the CRU drive bay)

So, this means that I will actually have to GO down to the theater at
some point later today and manually ingest "Ultra" into each server locally.
Ingesting locally into each auditorium's server always solves the problem.

As I said, this happens very infrequently, but whenever it does, it a big
P-I-T-A. I will either have to go down to the theater late tonight after
the last show, or go in around 4am 2moro morning to take care of this since
I have other commitments I need to work around and GDC's are friggin'
slower than molassas in January when attempting to mulitask ingest and
playback at the same time. So my best bet is to go do it when nothing
else is running at the theater, even though it's a complete drag on my time.

So, any ideas why this occasionally happens?

GDC Serverthingys are SX-2001A's, and as far as I know are up-to-date,
software-wise. The TMS was updated to latest version either late last
year or early Y2k15.

But this has always happened unpredictably since the system was
installed, and always when I have the least time to deal with it.

(I'm sure GDC designed it that way)

Posted by Carsten Kurz (Member # 5396) on 08-20-2015, 01:39 PM:

I would assume an issue with ext2/ext3 OR NTFS drivers of the TMS. Is the TMS running on a windows machine? You should try to check the disc format of these DCPs/discs causing these issues.

- Carsten

Posted by Jim Cassedy (Member # 4115) on 08-20-2015, 01:51 PM:

Hi Carsten-
I believe the TMS is running on Windows 7 Professional.
This only happens intermittently. Mabye once every 5 or 6 months.
But always (Murphy's Law) when I have the least time to deal with it.

Almost every other time it's happened, it has been with a drive from
somewhere other than a major studio or distributor. So I can understand that.
But "American Utlra" is from one of the 'big boys' so I'm surprised I'm having
trouble with it.

This morning, I've got one of 'the kids' physically moving the hard drive from
server-to-server and I'm ingesting and verifying the CPL's locally into each one
remotely from here at home. I will go down there later 2nite in person to check
everything out, since I have to deal with a curtain issue in one auditorium too.

(Thankfully the curtain got stuck in the "up" position, and not "down" - -
or I'd be down there right now trying to get it to open! hahaha)

Posted by Steve Guttag (Member # 268) on 08-20-2015, 03:21 PM:

I don't recall getting such complaints from my GDC TMS sites so I'm more inclined to believe it may be the TMS server itself and how it is set up, or is patched...etc. GDC switched over to TMS 4.0 in 2015 that is a place to start. Another thing about our TMS systems (anybody's) is that those servers are purpose built and NEVER update Microsoft anything. Once they are assembled and deemed stable, that is it. The only updates are to the TMS software itself. They don't surf the internet...all they do is run the TMS. We have the GDC TMS running on Win 7 Pro and Windows Server 2008.

Posted by Frank Cox (Member # 6258) on 08-20-2015, 05:18 PM:

I ingest directly onto my GDC server and the one and only time I got an "exception" turned out to be a bad CRU drive. They sent me another one and there were no problems with that (other than that I lost my Friday night show since I couldn't get the movie to load.)

Posted by Monte L Fullmer (Member # 2797) on 08-20-2015, 08:21 PM:

I once in a while get these exception ingests where I also have GDC, but with SX-3000 and also GDC TMS setup.

We use Windows 2008 Server OS.

I cancel the exception ingest and content, but before removing the drive, I push the the gray button until the blue drive light goes out (this parks the heads on the HD) then remove the drive.

To retry, I put the drive in another bay and restart the ingest where the next ingest is successful.

Once in a rare case, I'll have to do USB ingest if nothing else works.

-Monte

Posted by Mark Gulbrandsen (Member # 72) on 08-20-2015, 10:05 PM:

Exception's can occur if the server you are sending to or transferring from is not logged on to the TMS, say because you rebooted screen 2 a few minutes ago because of some projector issue. If you do that then TMS doesn't automatically log that screen back on when the projector and server comes up. You have to right click on the TV screen at the left and select "connect to screen" manually. It shows the exception because there is an error.

Say you start a transfer to screen 2 from the TMS Store and Screen 2 isn't logged on for what ever reason... TMS uses Filezilla as the FTP and FZ doesn't know IF Screen 2 is logged on or not and proceeds to transfer what you told it to. TMS does however get a signal back from the server acknowledging transfer and if it doesn't see that signal then you get the exception error.

A network error or funky switch could cause it as well but GDC TMS doesn't require a managed or sophisticated network to begin with. Simple GB switches work fine. SX-3000's only require one GB switch to handle both management and Media data thus simplifying the system even more.

If you log on the server and clear that exception then start the transfer over it should go fine.

Mark

Posted by Jim Cassedy (Member # 4115) on 08-23-2015, 10:09 AM:

Hi Mark-
Thanks (as always) for your comments.
But, whenever I've had this problem:
1) All the outlying servers HAVE been logged in.
2) Clearing and error & re-trying gives the same "exception" results.
3) As I said in my original post, this happens very infrequently, and
in the past, I've only had the problems with drives mastered by smaller
'independent' distributors, and was more prone to happen with material
ingested from mini USB drives. This was the first time I've had it happen
with a CRU drive from a major distributor.

I suspect that it has something to do with the network wiring, since not
only is it a very long run to two of the auditoriums, but I've never been
impressed with the quality of the net wiring work the installers did.

In fact on several occasions when I've had this problem, it has only
occurred at the two "distant" auditoriums. But last week I couldn't get
a clean transfer to even two "local" servers, one of which is located
probably less than 20 cable-feet from the TMS.

Here's a question for you: Do have any idea of what the 'average'
transfer speed between the TMS and servers is? (or is supposed to be?)

When nothing else is running, the process doesn't seem to take too long,
but if you are transferring while playing a show, the network slows down
to about the same speed as a turtle on tranquilizers.
(or a somewhat somnambulant snail on sleeping salts )

I know ALL systems will do this to some extent when multitasking,
but the difference on this one seems extreme. Or is it just the nature of
the GDCs?

Posted by Steve Guttag (Member # 268) on 08-23-2015, 10:14 AM:

The Dolby DSS line are the only servers I know that run pretty much at full speed when running a movie...particularly the DSS200 (The DSS220 does slow a bit but it is a software RAID and is generally slower on the ingest speed than the DSS200 too). GDC and Doremi do slow to a crawl, if not choke if there is a show going.

I don't have the exact speeds it normally runs. However in the GDC TMS, it does show the transfer speed so you could see it there.

Posted by Mark Gulbrandsen (Member # 72) on 08-23-2015, 05:25 PM:

Jim, the only other thing I have seen cause it is when someone fiddles with the settings and they point the media drive as another drive. That will also thrown an exception because the content is on the directory list but not on the "wrong" hard drive. Otherwise these systems are pretty foolproof.

The actual transfer speed is dependent on where the media is coming from. Move it between two server's and it can go quite fast. TMS to server is really dependent on how fast the TMS is capable of doling out the data. It's an app, so it's more CPU intensive than it is memory intensive. Oh, it IS best to transfer when servers are not playing back. The SX-3000 and it's hardware raid are a bit faster than the servers running software raid are at transfer during playback.

If you have managed switches those could also be a problem. I had two complex's full of local SMC managed switches and about half of them died a slow death. We pulled the rest and junked them and replaced them with standard gigabit Cisco switches. There is likely a gigabit media switch and a seperate management switch at your TMS so one or the other could also be the issue. Substituting in another switch is the best way to see if that's the issue.

Mark

Posted by Brad Miller (Member # 2) on 08-23-2015, 06:58 PM:

Be careful about screen to screen with GDCs.

GDC throttles incoming data during playback, but foolishly it doesn't throttle outgoing data, so if you have a GDC playing in screen 1 and a GDC in screen 2 is not playing and trying to transver a movie from screen 1, you're bound to have screen 1 start hiccuping and stuttering on it's playback.

Posted by Mark Gulbrandsen (Member # 72) on 08-23-2015, 07:55 PM:

Actually, after a certain build that no longer happens. The server playing back has it's ingest speed throttled now. The only time it might happen is if the array is above 90% full and even then it's going to throttle down to nothing.

Mark

Posted by Brad Miller (Member # 2) on 08-23-2015, 08:51 PM:

That's not the scenario I wrote above Mark.

Posted by Mark Gulbrandsen (Member # 72) on 08-24-2015, 02:02 PM:

That should not happen. Current firmware is designed to limit transfer speeds in or out during play back. It has been a zero issue for me.

Mark

Posted by Brad Miller (Member # 2) on 08-24-2015, 06:53 PM:

Well Mark that doesn't mean anything, because NOTHING is an issue for you when promoting GDC. So let's try this another way. Can you tell me what version of software the outgoing throttling during playback was incorporated in please? It would be quite helpful.

Posted by Mark Gulbrandsen (Member # 72) on 08-24-2015, 08:47 PM:

It was quite early on in the DCI build, they are at Build 218 now. I've never experienced any issues since probably just before the builds went over somewhere around a hundred. And even when it used to do that it was only if you were moving 4k content around while playing back. Build 7.8 final also doesn't do that. I still have non DCI servers on TMS as well. The GDC raid is actually Intel raid in disguise, Intel raid is also used in some cinedigm TMS servers,

Mark

Posted by Brad Miller (Member # 2) on 08-24-2015, 11:33 PM:

Mark that's ridiculous. The server doesn't choke on data because it is part of a 4K DCP. It's just data. That would be like saying the server would choke on any movie with a running time over 2.5 hours long. The data being transferred has nothing to do with throttling the outbound FTP speed to protect the server from stuttering.

Posted by Steve Guttag (Member # 268) on 08-25-2015, 09:48 AM:

Build 108 and above of 8.01 (conventional mediablock) has been stable for us. Build 186 has been very stable. Note, starting with build 200 (once it is loaded), the GDC will start checking warranty status and prevent future software updates if the server is no longer in warranty. GDC has a 2-tier system of warranty...hardware/software and software (with tech support) only. Looks like the free ride on server software is over except for USL of the primary 3rd party servers.

Posted by Carsten Kurz (Member # 5396) on 08-25-2015, 10:27 AM:

Anything about Dolbys policy towards Doremi server software upgrades?

Since 2.6.4, not much happened on the Doremi sites. Are there more recent versions available though Dolbys partner portal? Will Doremi users have to pay for updates in the future?

- Carsten

Posted by Dennis Benjamin (Member # 1137) on 08-25-2015, 11:22 AM:

I have a particular server this happens to at one of my locations. I am 99% certain it has to do with the network cable that was run. (I did not run this particular run of cable.)

It happens every once in a while. Since I do the work at this location remotely, I simply re-boot the G.D.C. server and it usually fixes the issue. It's every couple of months or so that it happens...

Posted by Mark Gulbrandsen (Member # 72) on 08-25-2015, 12:16 PM:

So far all of my customers have jumped on the GDC warranty deal. They just about cut the price in half from what it used to be and the warranty now includes the media block, all server parts and support. Hard drives have not been included for some time in the extended warranty programs but if you re-up they will sell you new drives at a big discount.

Mark

Posted by Steve Guttag (Member # 268) on 08-27-2015, 12:16 PM:

As I hear it, on the Doremi stuff...PATCHes that fix particular problems will remain free but actual version updates there will be a fee for the update of out of warranty products. One thing Doremi (and GDC) could do was just issue patches whereas the DSS line from Dolby was an all or nothing thing.