Film-Tech Forum ARCHIVE: CP650 sometimes not accepting commands from Doreim DP2K4

This is topic CP650 sometimes not accepting commands from Doreim DP2K4 in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=003555

Posted by Michal Matys (Member # 8614) on 01-18-2019, 07:56 AM:

Strange issue started happening in one of our cinemas recently. We updated the servers to 2.8.20 version. We have also changed some macros recently, but what we did is just copied the old ones via web interface and created duplicate with different names.

Some commands for the sound are not being executed. It is completely random. The macros themselves are working when I test them via macro execution.

In the logs we are getting following error:

[Thu Jan 17 14:40:01 2019]: [PLAYER]: Macro "LAMP ON" executed with status: PlaybackElementStatusExecuted
[Thu Jan 17 14:40:04 2019]: [PLAYER]: Executing macro "2D FLAT"
[Thu Jan 17 14:40:06 2019]: [PLAYER]: Macro "2D FLAT" executed with status: PlaybackElementStatusExecuted
[Thu Jan 17 14:40:15 2019]: [PLAYER]: Executing macro "LIGHTS 50"
[Thu Jan 17 14:40:15 2019]: [PLAYER]: Macro "LIGHTS 50" executed with status: PlaybackElementStatusExecuted
[Thu Jan 17 14:40:18 2019]: [PLAYER]: Executing macro "DIGITAL SOUND"
[Thu Jan 17 14:40:21 2019]: [PLAYER]: Macro "DIGITAL SOUND" executed with status: PlaybackElementStatusExecutionFailed
[Thu Jan 17 14:40:21 2019]: [PLAYER]: Executing macro "VOLUME 40"
[Thu Jan 17 14:40:24 2019]: [PLAYER]: Macro "VOLUME 40" executed with status: PlaybackElementStatusExecutionFailed

Tried to replace network switch, swapped the eth 1 eth 0 on doremi.
I have also swapped CP650 with another hall and the issue moved with this CP650, so I suspected this could be the processor issue, so I replaced it for spare one, but recently another hall started showing the same issue, seems too much of a coincidence to me.

Anyone had something similar before? Doremi was claiming that the device / macro configuration is incorrect, obviously it is not since I am sending the commands via macro execution and it works fine.

Thank you
Michal

Posted by Dave Macaulay (Member # 813) on 01-18-2019, 08:44 AM:

Looks like you're doing everything right, and if the commands work intermittently they should work all the time.
I haven't seen this come up with 2.8.20 yet but we have few 650s.
I have had lots of weird problems with the 650 and network automation. The CP650 network hardware is just not very good, the design is very old as these things go using the early components available then. You definitely need to delay between commands, it looks like you did that ok.
There are a few sites where I used CP650 serial control either directly or via Jnior TCP/IP to serial conversion, because network automation would not work reliably. Serial is super reliable in my experience.

Posted by Sean McKinnon (Member # 612) on 01-18-2019, 08:50 AM:

I have found with the Doremi that it sends commands very fast after opening the TCP port though I have not had a specific issue with the CP650. It may be worth adding a \w to the beginning of your macro's text string this will add a very short pause between opening the port and sending the text string.

Posted by Steve Guttag (Member # 268) on 01-18-2019, 12:55 PM:

Sean is correct. The Doremi (and their Dolby decedents) behave poorly, Ethernet wise. It is advisable to always use a \w command (wait). They misfire their cues at FAR too great a frequency and putting in the wait command helps a lot.

Note, the CP650's Ethernet port is "weak"...it barely can do its job and gets flooded EASILY. Do not rapid fire stuff into it or it will lock up and the only way to wake it up again is to reboot.

In a related topic, we were having problems with Doremi servers talking to eCNA automations. In addition to the wait commands, the eCNA has to be switched to full-duplex before we found the commands to be at or near 100% reliable. The CP650's port is half-duplex, I'm reasonably sure. You can also double up your commands on the Doremi, if need be. That can better you odds (particularly with things like NEC projectors are that get fussy about listening for such commands.

Posted by Michal Matys (Member # 8614) on 01-18-2019, 01:24 PM:

Just got call from the cinema that same thing happened on hall with CP750. Same error in logs, three shows today went okay and this one failed, note that it started happening randomly after the server update as well as Macro change.

What I will do is completely delete the devices and macro xmls, ingest clone from another cinema and sdjust it accordingly, I suspect some glitch in the system, could have somethig to do with ScreenWriter TMS which we are using, maybe the macro packs got confused after the macro changes, hard to say.

So strange though.

Posted by Carsten Kurz (Member # 5396) on 01-19-2019, 06:09 AM:

A while ago I had someone reporting failed CP750 macros to me, and claimed it happened out of the blue with no changes to the system. Well, after some days it turned out that someone had indeed worked on the configuration and screwed it up.

But this could happen due to a bad switch port or cable as well, so, one should probably always have a cheap small replacement switch on site for immediate testing.

- Carsten

Posted by Bruce Cloutier (Member # 9615) on 01-21-2019, 07:46 AM:

We see the following happen all too often (and not long ago with Doremi I believe) so I want to post it here for all of you programming types that may be lurking. The symptom of it is that communications that appear to be working sometimes fail seemingly at random times but sometimes correlated with levels of network activity.

When you program your network interface you need to remember that one TCP packet DOES NOT equal one command or message. So when your driver reads from the network using a fread() or something, the data you receive might indeed frequently contain a whole command or message but sometimes it will contain a partial message or it might have a whole message and part of the next. This depends on network and server loads. This is because TCP is a stream (not UDP) simulating a serial cable. Protocols need to be designed with headers that detail the count of bytes to follow in the message. You then use this count to collect that amount of data before you can assume that you have an entire message to process. Otherwise things get out of sync and depending on software design may or may not ever fall back into place. That guarantees loss of messages (i.e. the macro didn't work). Often it leads you to reboot.

I am not saying that this is necessarily the issue behind this thread but, this is such a rookie programming mistake and it is so unbelievable how often we have to explain it to people. It is worth preaching about.

Posted by Steve Guttag (Member # 268) on 01-21-2019, 10:39 AM:

My two most problematic companies for Ethernet commands are Doremi (servers) and NEC (projectors). The NEC projectors just don't want to deal with incoming commands if it is "busy" doing something like opening a douser. You'd think it would have a buffer that could execute subsequent commands and it probably does but it is unreliable.

Doremi just is unreliable with enough devices to point the finger at it. They are the only ones that need to put a "wait" command into the command string too. It is also an admission of guilt, to me.