Film-Tech Forum ARCHIVE: Weird Network Problem in the Booth

This is topic Weird Network Problem in the Booth in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=003740

Posted by Dennis Benjamin (Member # 1137) on 07-15-2019, 06:27 PM:

I have a 7 screen site that is a mix of equipment - Christie/GDC.

I have two CP-2000zx's that are actng funky on the network.

I have an Ubuntu box that is being used to monitor the equipment in the booth. The Ubuntu box cannot see the 2 projectors on the network.

All of the projectors, the Ubuntu box, the theatre's TMS/LMS, the GDC servers, are all plugged into the same switch and are programmed to be on the management network (correct IP, subnet, and gateway)

I can ping the projectors I.P. addresses from the TMS/LMS and they answer back. I can ping the TMS/LMS from Ubuntu box, as well as every other piece of equipment on the mangement network and get an answer back, EXCEPT those two projectors.

I've checked all network settings on everything.

I have other locations with the same set up with CP-2000zx's. Everything works.

These are the only two machines in the entire company with this issue.

Any ideas?

Posted by Leo Enticknap (Member # 534) on 07-15-2019, 06:38 PM:

Is there a DHCP server on the management network, and could it have allocated the static addresses of the Christies as DHCP clients, thereby creating an IP address conflict?

Posted by Mark Louis (Member # 10632) on 07-15-2019, 06:54 PM:

IP conflict would be my first guess as well, as long as you've verified there's no issue when connecting directly to the CP-2000 NIC's with your laptop. Only other thing you might want to make sure of is that your switch is not managed and you just happen to have the two CP-2000's in the wrong ports.

Posted by Leo Enticknap (Member # 534) on 07-15-2019, 08:41 PM:

Unlikely, I'd guess, if communication was previously OK, nothing was changed or messed with, and now, for no apparent reason, communication is no longer OK. That is why I thought of a DHCP server: it will allocate addresses without any human interaction, other than connecting a device that then asks for one. If the management LAN is also the house LAN for Internet access (not recommended per DCI, but is the reality in a lot of theaters), all it takes is for an usher to connect their iPhone to the house wifi, and, if the DHCP range is not set outside the static range, this can happen.

For this reason I would always separate the DHCP range from the static range: for example, configure a router's DHCP server to give out addresses in the 172.16.1.200 to 253 range, and use 2 through 199 for static addresses (with 1 for the gateway/router). But if you don't do this, and allow static and DHCP-allocated addresses to co-exist on the same subnet, you are at risk of address conflicts happening.

You could try scanning the subnet using an IP scanning program (Advanced IP Scanner is my favorite, and works in Ubuntu under WINE; Angry IP Scanner is available in a native Debian version for Ubuntu) but there are many to choose from), and if it reveals a device with the same address as your Christie that is not your Christie (if it has no hostname, check the MAC address), then you know that you have an IP address conflict on your hands.

The only other thing I can think of is that major updates to Ubuntu can sometimes mess with the IP address settings; so it might be worth double checking that the IP settings in the Ubuntu PC are correct, especially if you have a static address, subnet mask and gateway configured. In particular, this happened the last time I updated between LTS versions (I think it was 16.04 to 18.04, but can't remember for sure). For this and several other reasons, I prefer not to do major updates to Ubuntu, period, but to nuke the partition and do a clean install of the new version.

Posted by Scott Norwood (Member # 30) on 07-15-2019, 10:24 PM:

Assuming that it's not an IP conflict, as discussed above, I would look at the switch configuration. Are you using vlan tagging anywhere? Is it a layer 3 switch? If so, is "ip routing" enabled? Are there any access lists that would prevent things from talking to other things?

Also, does anything here have multiple network cards and/or multiple IP addresses assigned to it?

I know that you said that you checked it, but are all of the netmasks correct?

If you connect the Ubuntu box directly to one of the projectors in question using a crossover cable (no switch in the path), can you ping one from the other?

Is firewalld disabled on the Ubuntu box?

Edit: are you pinging by host name or IP address? Be sure to do it by IP address only in order to eliminate (or identify) any potential DNS problems.

Posted by Marcel Birgelen (Member # 6801) on 07-15-2019, 11:48 PM:

I'd recheck the subnet mask on both the Ubuntu box and the projectors. If you're absolutely sure they're correct, then I'd say resume with the following step.

You didn't indicate it, but I guess both Ubuntu box and the projectors are in the same broadcast domain/IP range, so the Ubuntu box should use ARP to communicate to the projector. Check if it resolves to the right MAC address:

First, on the Ubuntu box do a quick ping:

ping <IP address of one of the projectors>

When nothing was magically healed, this should give time-outs, you can kill the command with Control-C. This step is important for the next one, because without any reason to communicate with the host, the ARP table will not be populated with its MAC address.

Continue with:

arp -an | grep <IP address of one of the projectors>

It should return a MAC address, if it doesn't, there is a Layer 2 or lower communication problem, most likely switch related. If it does return a MAC address, check if it is the MAC address of the projector and not something else. If it is something else, then there is an IP conflict. To get a clue about what kind of device it is that is interfering, you could check the vendor of the MAC address online. This often gives an indication in the domain to be searched. Alternatively, if you have a managed switch, you could log-in onto it and possibly check via that route, on which port a certain MAC address is located. The syntax differs per switch brand, unfortunately.

In order to debug your Layer 2 problem, one thing you can still do is doing a tcpdump. Make sure tcpdump is installed on your Ubuntu machine, otherwise install it via:

sudo apt-get install tcpdump

Then, open a second terminal or secondary SSH session. In the first, do something like:

ping <ip address of a projector>

in the second, something like:

sudo tcpdump host <ip address of a projector>

You can kill both commands with Control-C, but the tcpdump window should give you an indication of the traffic flow and if anything is answering on your ARP requests and if something is giving you an ICMP ECHO reply on your pings.

If there is clearly no response, then I'd proceed to look at the logs of the switch, if there are any and maybe I'd schedule a reboot of the thing. I've seen weird stuff happening with switches before...

Posted by Steve Guttag (Member # 268) on 07-16-2019, 07:27 AM:

In addition to the other suggestions (so I won't repeat their excellent advice), since the ZX is a series 1 projector, you do know you have TWO IPs to that projector? One is to the TI boards and one is to the projector management. Is there any chance that someone got confused on the setup of those and is pointing your Ubuntu box to the TI boards instead of the projector?

The DLP IP address has to be on the same subnet (but not the same IP, of course). The server cares about the DLP IP for security reasons but control cares about the projector IP.

Posted by Ken Lackner (Member # 1002) on 07-16-2019, 01:03 PM:

In addition to Steve's addition, most Christie projectors have a setting on the network settings page that can be set to "Login Required," where the device trying to connect to the projector must supply login credentials, or "Free Access". I have never seen "Login Required" used. In the projector's Web GUI, go to the Admin tab along the top, the the Network tab along the bottom. About half way down the page on the left you will see a section called Ethernet Remote Access Control. Try changing this to Free Access if it is not already.

Posted by John Thomas (Member # 6509) on 07-17-2019, 09:17 AM:

It would be worth re-configuring the VLAN settings for those ports on the switch. Sometimes switches will lose settings after a power-outage.

ZX PCM boards can have communication issues like this. Have the batteries on the PCM boards been changed on time?

If you don't have the program already, ask Christie (or someone on here) for the ZX Log Cleaner, run that, and then reboot.