Film-Tech Forum ARCHIVE: GDC Cue fails to fire

This is topic GDC Cue fails to fire in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=002265

Posted by Dave Young (Member # 8390) on 04-04-2015, 07:18 AM:

This is a "have you seen this" kind of question.
At a location, I'd set up a new device and cue on each GDC system. In a test playlist, the cues fire 100% of the time. In a production playlist, the cues are failing to fire almost 100% of the time. I've been attaching them to video clips.

I've been attaching the cues in production instead of to an MOS. That's pretty much the only difference. This is one of those rare illogical cases of "works while I'm there" and that doesn't sit will with me AT ALL.

Any thoughts?

Posted by Nguyen Jack (Member # 7530) on 04-04-2015, 10:29 AM:

What is the new device ?

Posted by Harold Hallikainen (Member # 5405) on 04-04-2015, 10:58 AM:

Time for WireShark? I do know that GDC sets up a TCP connection to a device and leaves it open. Sometimes the first cue of a day can be missed because the command is sent right after the connection is set up. On this "miss 100% of cues," I wonder if you're hitting some limit as to how many sockets can be open at once. Summarizing, I don't really know what the problem is, but if I were debugging it, I'd use WireShark.

Good luck!

Harold

Posted by Steve Guttag (Member # 268) on 04-04-2015, 01:59 PM:

Try this with GDC...double fire all cues. that is, fire the same cue twice in a row...there is no penalty for that. Harold is correct that GDC opens a port and LEAVES it open and often the very first cue that opens the port is a sacrificial cue that opens the port...double-firing cues will take care of that. I normally space my cues apart with a wait...depending on the device, I might space a cue out by a second. I've said it before and I'll say it again, I prefer a dedicated automation...missed cues are VERY rare for us (pretty much nonexistent).

Now, on any server, we normally recommend that all cues reside on a "house" CPL like a blank or policy strip...the exception being the "CREDITS" cue since that HAS to be in the feature. In this way, they can set up templates and just adjust Ads, Trailers and Feature without having to recue everything...just the Credits.

Posted by Justin Hamaker (Member # 2165) on 04-04-2015, 03:44 PM:

It may just be that you're not sending the right name to the GDC. If you're sending cues from an external source, you need to make sure the name is exactly the same as on the GDC. If your playlist is sending "Flat", but the GDC has "MPEG_FLAT" for the flat format, the GDC won't know what to do with the "Flat" cue.

Posted by Dave Young (Member # 8390) on 04-04-2015, 09:19 PM:

I'll try the double fire. I can execute the cue manually in the control panel and it works. I also added the cue to a house CPL (MOS). I added some logging to my listening device and see lots of connections but not any socket closes!

Posted by Mark Gulbrandsen (Member # 72) on 04-05-2015, 10:23 PM:

If these are the first cues of the day then this is a known issue with certain outboard pieces of ancillary gear. In short, power up all ancillary gear firsthand and then the server so they can handshake. Also be sure your firmware is at the latest build for your model server.

I have close to 240 GDC servers I look after and cue problems are so far down the list as to be non existent.

Mark

Posted by Dave Young (Member # 8390) on 04-05-2015, 10:36 PM:

Quick update. I dug through my logs and saw socket connection at exactly the right times. I had been seeing them come in from a single IP so suspected a single rogue server but it was a false positive. I modified the logging to track open sockets and saw that each attempt was leaving a socket open. I had a chance to scoot over to the theater and modify the device to add a CRLF. On the next cue attempt, things worked perfectly.

So tomorrow I will modify the others to just add a CRLF On the device and I suspect that will fix the "problem". If it doesn't, I'll double-fire the cue in each action needed.

I'm very glad the puzzle is sorting itself out. Hate illogical clues!

Posted by Steve Guttag (Member # 268) on 04-06-2015, 05:57 AM:

Again, the GDC automation feature DOES leave the socket open...always...there is no choice in it. It would have been nice if it was a choice. On some older devices like the JSD80 or CP650...there is just the one port to open so if the server (or any device) grabs ahold of it...you are locked out from any other device.

The CR/LF thing could help some things though it won't help leaving the socket open. Though if you precede your cue with a CR/LF you can help since the CR/LF will open the socket. I've found that When the Dolby server fires its first serial cue after wake-up that cue can be missed (at least to an eCNA automation)...but if there is a CR/LF preceding the cue (just put an /0D/0A in the cue that is firing), then that does the handshake and doesn't compromise the cue at all.

Posted by Harold Hallikainen (Member # 5405) on 04-06-2015, 07:53 AM:

Does most stuff require CRLF, or just CR? Everything I designed just uses CR as a terminator. And, I believe the escape sequence should use a back slash (ie \0D on Dolby for CR).

Harold

Posted by Steve Guttag (Member # 268) on 04-06-2015, 10:29 AM:

Yes...it should have been a black slash (\). It all depends on the device if they want an LF after the CR. In fact, I've seen all sorts of stuff use both a leading character as well as termination characters. Some, like Extron, don't use any delimiters...if the right sequence of characters show up...it does the command!

Posted by Dave Young (Member # 8390) on 04-06-2015, 01:59 PM:

I've tried just adding a CRLF and it's spotty at best.
Next I'll try adding the code to the beginning of the string and see how that goes

Posted by Dave Young (Member # 8390) on 04-06-2015, 10:00 PM:

Guess which key is missing from the GDC touchscreen keyboard?
Added an additional action with some dummy chars as in "init" and so far it seems to be working. Will run all day tomorrow and see.

Posted by Harold Hallikainen (Member # 5405) on 04-06-2015, 10:38 PM:

On the missing keys on the GDC touchscreen, luckily you can plug in a USB keyboard to get them.

Harold

Posted by Mark Gulbrandsen (Member # 72) on 04-07-2015, 01:37 PM:

Yea, you ALWAYS want a keyboard on a GDC server or any other for that matter. The touch screen is really for building up shows and operating the server.

Posted by Steve Guttag (Member # 268) on 04-07-2015, 04:43 PM:

You can also just VNC to the GDC server and use your laptop/computer's keyboard too. The only time I drag a physical keyboard on to the GDC is on initial commissioning and when I make a periodic "save" of everything where you need to catch it on boot up so it stores an image (in case something goes awry down the road...kinda like Windows does with its System Restore. Cheap insurance, if you are thinking about it at the time.

Posted by Dave Young (Member # 8390) on 04-07-2015, 09:32 PM:

I've tried double-firing, all variety of CRLF, tried adding \0D\0A with a keyboard. It goes in just fine, but gets sent as strings and not escaped chars. I've also tried adding dummy commands before and after the desired command.

The only thing consistent is that the GDCs open the socket and keep it open. That will be an awkward burden to bear with hundreds or thousands of screens.

I setup a few of each type tonight, will remove error handling on my socket server, and see what happens tomorrow.

Otherwise, I might just end up using a JNIOR because I know it works. Just trying to avoid the extra costs.

Posted by Steve Guttag (Member # 268) on 04-08-2015, 05:34 AM:

When you double-fire...do you put any delays between each cue? And don't be afraid to stack even more...I've done up to 4 (with 1 second spaces between them) to get a GDC to convince an NEC projector to change Presets or move a douser.

But honestly...it shouldn't be too difficult to get things to fire reliably with the GDC though I strongly advise the use of a proper automation. My preference is the eCNA line.

Posted by Mike Blakesley (Member # 26) on 04-08-2015, 03:14 PM:

One quirk I have noticed about the NEC is, if I close the douser manually (either on the control panel or using the NEC or GDC interfaces), the next time a playlist runs it won't open the douser. Every other function works fine no matter what manual step have happened.

Posted by Dave Young (Member # 8390) on 04-08-2015, 09:08 PM:

Steve, the combination that seems to work:

Dummy Call
Timer (500)
Actual Call

Thanks for the timer suggestion. It looks like it's going to work. I am definitely going to be looking into dedicated automation (I have worked with JNIOR). It complicates the setup but also works 100%. Also don't need to worry about open sockets on my server for thousands of screens that way.

I appreciate all your help!

Posted by Ian Freer (Member # 1940) on 04-08-2015, 10:13 PM:

A few notes if you're going to use the Jnior AND the Cinema.Jnior application to run Macros etc... This might be specific to GDC, I am unsure, as in my case it is with a GDC...

Long story short: I am just trialling Cinema.Jnior 2.34.212.1338 as we had significant dropped cues with an earlier version. I don't know if it's up on their website yet, but it's been running for a few days with no issues so far.

For more details read on...

I have several sites with GDC, Jnior and the Jnior manual control panel, so running Cinema.Jnior is needed to make the control panel work.

I had been running Cinema.Jnior 2.22.1115.1501 for a while, but every so often a site would let me know the Jnior had become unresponsive and needed a reboot.

The guys at Integ looked into it and found an issue and a fix, so gave me version 2.30.939.1430, which I installed into one nearby site, connected to a GDC, and it has not since missed a beat.

So feeling confident, I recently updated another 'occasional reboot required' site to 2.30... (and updated the GDC to Build331 at the same time). It broke the Jnior automation. i.e. other than a very occasional reboot required, it had been functioning well, now it was missing many cues per day.

After lots of back and forth with Integ looking through logs, they found an issue where some characters were being dropped meaning two messages were running together as they were received from the GDC, resulting in invalid commands and missed cues. It seemed only to affect multi-cue macros from the GDC, i.e. a simple House Lights Up command was fine, but a SCOPE FEATURE Cue, with Lights Down, Masking Open, Volume x.x all trying to fire from the GDC to the Jnior and the last message or two got dropped, but only sometimes...

I also run duplicate commands and a 500ms gap between commands to the same device in a setup like this to ensure things run smoothly, but was finding I had to really space things out to 2000ms, and while that improved things, it still missed many cues.

Luckily one site had reported this to Integ a couple of weeks prior to me, so a fix had already been developed but not tested in the field... But it's all looking OK at the moment. No complaints from my site since the upgrade to Cinema.Jnior 2.34...

Cheers,
Ian

Posted by Steve Guttag (Member # 268) on 04-09-2015, 06:23 AM:

See generally when I use an automation...I only have the GDC fire ONE cue to the automation and if there are multiple tasks, then the AUTOMATION handles the Macro or sequence of events. With the eCNA line of automations, there are currently 50 macros one can set up, 9 programs as well as direct out commands (and RDI commands).

So I'll have Start Show 2D Flat (or words to that effect) and everything one needs to happen to get the show going will be handled by the automation once it receives that command. Projector, curtain, masking, lights...what have you. And it will do it without fail and you can trace the command coming from the server (anybody's) as well as the macro being executed.

It doesn't really complicate anything either, it actually greatly simplifies it. Your commands become VERY uniform regardless of projector or server company you are integrating with. The commonality is the automation itself. The same macros can be used throughout the chain and even if Christie is used on one site and NEC is used on another and Barco is used on another (you can do the same with the sound processors too), the whole operation remains very uniform. Via the eCNA's RDIs, if you are uniform in how you create them (function 1 is Lamp on, Function 2 is lamp off...and on down) such that when you switch out the RDI file based on the projector or sound processor...you don't have to reprogram anything other than the IP addresses.

Even the "greatest" DCinema system out there (Film Tech's DCS) uses the AUTOMATION as the central control point (The FT-21) such that it is equipment agnostic.

Having an automation just makes much more sense to me and honestly, our cue related complaints are effectively zero. It is nice that servers have some capability to control stuff out of the box and in the most basic of systems it might be sufficient but it isn't what I would want to design anything around.

Posted by Dave Young (Member # 8390) on 04-09-2015, 07:06 AM:

That's a fantastic design. I'm a 3rd party service provider integrating with existing setups for second-screen automation, so I don't get to make decisions, only recommendations. Additionally, because my commands hit an externally located server, I need special handling on a case by case basis.

I prefer to use a colocated JNIOR taking a single command in, and sending multiple commands out. Sometimes I have to go through an ACT (GDC->ACT->JNIOR->Cloud) to do that. Rick from Integ has been extremely responsive in answering my myriad questions, and this kind of setup runs for months with no hiccups.

Either way, it usually requires someone to do some network and firewall config though. Getting some bandwidth from those kinds of resources is extremely difficult. For this reason I do what I can to get up and running, I'll try anything once. I'm now thinking about doing what I need to do by monitoring a TMS.

I have tremendous respect for what you guys do. It's deeply fascinating

Posted by Ken Lackner (Member # 1002) on 04-09-2015, 07:54 AM:

Dave, I'm curious why you would use an ACT and a JNIOR. They are both automations. Why have two automations?

Posted by Steve Guttag (Member # 268) on 04-09-2015, 10:27 AM:

Yeah...just what does the JNIOR do that the ACT doesn't? I think on of the ACTs biggest flaws it what some would call its greatest strength. That is, it is complicated and designed around running scripts. Once you get your scripts the way you want them (and there are LOTS of canned ones), you then are down to changing IPs for specific sites...but before that, you have a bit of overhead getting the scripts to suit your types of theatres.

The JNIOR is definitely a cost-effective device and I can appreciate why it is favored by many.

At some point, we standardized on the eCNA (Eprad) and they too have been quite responsive to our requests. The eCNA is VERY fast to set up and configure as needed...no scripts...just a decent web browser interface for setting up what you want and super-easy to save things as you desire.

Note, we too are a 3rd party service company (and a dealer) so I do understand the difficulty of sometimes convincing customers to buy things. We do have a decent track record we can reference of how our systems perform. That can often help.

Posted by Dave Young (Member # 8390) on 04-09-2015, 11:22 AM:

The only reason for running through an extra ACT is network topology and security restrictions on the IT level. In the one case I've had to do this, the theater was willing to open up my JNIOR to net out on the firewall, but not their Christie ACT itsel, and certainly not the projector network. Although a firewall means you can safely restrict traffic to outgoing on specific ports, some feel safer with lockout topology.

Posted by Dennis Benjamin (Member # 1137) on 04-09-2015, 07:35 PM:

If a theatre is under a V.P.F. agreement, like with a company such as Christie, they lock up their network pretty tight. Making changes to the firewall or to the network configuration could void the contract that the theatre chain has with the monitoring company. This risks thousands of dollars and could void the V.P.F. contract. So I'd be very careful.

Posted by Steve Guttag (Member # 268) on 04-10-2015, 05:56 AM:

Generally, the firewall is the router and only the NOC will have that password. Likewise with any managed switch which could also limit which ports can be used and what devices can "see" what other device.

Posted by Dave Young (Member # 8390) on 04-10-2015, 09:10 AM:

Dennis, I am following the rules in regards to modifications thanks to your advice when we spoke on the phone in early January.