IMS3000 network switch and Internet access – any concerns

[Steve Guttag]

What are you doing to allow the IMS to get a good NTP time source? Many IMS clocks drift, considerably.

[Bruce Cloutier]

Just as aside, we get telemetry from about 22% of the population of JNIORs. About 20% of those routinely have direct access to us as opposed to only when being serviced. Now not all are in cinemas of course. We basically just get a packet with the serial number (and host name) which allows us to count and map them here. I have had to do that to quantify things relating to our customer support liabilities and to monitor our performance. You know, since we provide support for free and that.

There is a big difference between devices having access to the Internet and the Internet having access to devices. Most of you do an excellent job managing this. But even if you are air-gapped it is absolutely imperative that you define passwords and not leave the defaults. The vast majority of JNIORs still have the defaults (Ugh!). Also, you need to check the user database on all of your systems. While you might define a unique password and overwrite the default, there may be other users defined wherein the default is still active. Did you know that the JNIOR has an 'admin' user with administrative permissions? These account credentials are ultimately your last defense against an external attack. Each JNIOR does not need a unique password (or user). But you should use a custom password at least for all of those that you are responsible for.

Obviously, using a browser to access random sites or clicking on links in emails are huge risks. Logically, I hope, that doesn't happen on your production network. What they do in the office, well, who knows?

Also, your systems use general operating systems like Linux. No one really knows everything they are doing. There is all kinds of network activity that you can observe. A lot of it I can't even guess at what it is doing. I have a QSC core, and when we power it, that thing is noisy, busy communicating to who knows what? So out of paranoia, an air-gap strategy is commendable. At least I know everything that my OS does, why and when.

Full disclosure, the JNIORs phone home for the reasons I noted, but in that process we created a structure that we could use to setup a NOC. Since the population is huge there is possibly value in that. Those features have to be specifically enabled by you on each JNIOR and require the entry of a cryptic activation code. No JNIORs have been activated and we have not built any NOC. I think all of you have stacked up more than enough monthly billings. You can chat us up if you have concerns or interest in the concept.

I also have two JNIORs directly connected to the Internet each with their own public IP address (and domain names). I monitor network traffic daily. They survive just fine and with all of the ports open (Telnet, FTP, etc.). Of course, there are no default credentials. JANOS also uses a technique that makes it essentially invisible to perhaps over 90% of malicious bots. It is interesting because those JNIORs are not publicized. All of the network traffic that I see comes from crawlers, bots probing, worms, and all of that. There should otherwise be silence.

There are screens displaying network sniffer output in my office all day long. Reminds everyone of The Matrix. You do know that JNIORs have their own network sniffer (try NETSTAT -sd) at the command line. A lot of you do need to update JANOS to get all of the diagnostic tools. I lose sleep over those of you happy with JNIORs running JANOS prior to v2.0 (Series 3s as well). We went through our own certification process and cleansing, eliminating almost all remaining issues in going to JANOS v2.x. I get it. It works so don't fix it. The problem is that when it doesn't work you aren't noticing.

[Marcel Birgelen]

What are you doing to allow the IMS to get a good NTP time source? Many IMS clocks drift, considerably.

You can configure the Ubiquity EdgeRouter as an NTP server for your local network, the router itself pulling the time from the known pool of public NTP servers.

[Steve Guttag]

Sonicwall (with 7.2.x) can also serve as an NTP time source...switches and routers can do that, depending on make/model and how they are configured. But it isn't a "given."

As to default passwords...it depends. I come from an era where guest-technicians were a common thing in the areas I work and they have legitimate reasons for needing access to things like the projector and sound processor (going back to the film days, even). I don't go out of my way to get rid of the defaults. I go by the theory, if the person you are trying to keep out is physically in your booth and trying to connect to your equipment, the security breach was allowing them physical access to the booth.

For the most part, what is the worst a "bad actor" can do to your system? Delete content? With most any reasonable backup scheme, you should be able to get out of most other tampering relatively quickly. I'm just not worried about people accessing the server or projector or sound. I already make it a need to be on the physical network to do it. Some systems depend on default passwords for content transfers to work (I've found out). Normally, I do provide the end users their own unique passwords for the site and I tend to make them fast/easy because, again, they are physically connected to the network when using them...no remoting in.

The one thing that is completely locked down is the router. That is where one can keep bad people out and set up the rules of what can go where anyway.

Again, you can go and change every user/password...require a known MAC Address for equipment (everything) and really clamp the system down. I just don't think the benefits outweigh the downsides of maintaining a system like that. If they are standing in your booth, something else has gone wrong.

[Leo Enticknap]

The other issue with changing default passwords is that a record then has to be kept as to what they have been changed to. If the equipment is used for months on end without an administrator or root level login ever being needed, during which time whoever changed the password left the organization without succession planning that included passing this information along to anyone who might need it, then the next time such a login does need to be done, you're buggered. In almost a decade as a service tech I have had to abandon maybe half a dozen service calls because a factory default password had been changed and the equipment's owner did not know what it was, and was unable to find out. In one case we had to replace an NEC CPU board in order to get back in, in another I had to nuke and factory reinstall a DSS200 software image, having first left the thing for several hours backing up DCPs onto a CRU drive that the customer could not tolerate losing, etc. etc.

Whenever I install new equipment now, I tell the customer that I have not changed any default passwords, and that it's their decision, as owners of the equipment, as to whether they do so or not, weighing up the pros and cons ... but ... that if they do, it is up to them to maintain records, and that if they don't, we won't be able to help them if they call us back for troubleshooting in a few years' time and cannot tell us what the changed credentials are. I also note that this warning was delivered on the install report.

[Bruce Cloutier]

The JNIOR has the Safe Mode that temporarily enables default login credentials. That requires physical access. But I agree with you. Maybe everything should use 2FA? Haha. No! But at least ask you to identify the photos with bicycles!

To Steve's points, the threat is less likely to come from outside. It is more that the disgruntled individual that left might seek to cause some problems. I should not have suggested this given the date.

But they would know the password even if changed. And it would be pretty bad if they disrupt the event schedule or, worse, an ongoing show. Depending, it might be possible to damage equipment, blow speakers, or something.

A location sticking to a common password (or some rolling sequence) would still be better than the defaults.

Well, we have 20+ years experience with the same defaults and I do not know of any issues coming from that.

[Harold Hallikainen]

For devices with a display, if the user has not set a password, the device could generate a random password on power up and display that password. Once the user sets a password, it would be nonvolatile and would not be displayed. If a user forgets the password, a restore button can generate a new random password and display it. The user can then set a new password and save it. This is a bit like the random factory password with a sticker on the device, but avoids the need for a sticker.

[Marcel Birgelen]

Sonicwall (with 7.2.x) can also serve as an NTP time source...switches and routers can do that, depending on make/model and how they are configured. But it isn't a "given."

Running an NTP server on your gateway is probably one of the better solutions. It avoids the firewalling issues of NTP if you use DNS names like the "pool.ntp.org" addresses. Those are notoriously hard to firewall, as they constantly change. Also, it's not such a good idea to statically configure NTP servers on IP addresses into your servers, as those NTP servers have the tendency to move to other IP addresses or simply cease to exist. If your gateway handles NTP via one of the pool.ntp.org addresses, you're pretty much ensured there will always be a functioning NTP server and by connecting locally to the gateway you avoid the NTP firewall headaches.

As to default passwords...it depends. I come from an era where guest-technicians were a common thing in the areas I work and they have legitimate reasons for needing access to things like the projector and sound processor (going back to the film days, even). I don't go out of my way to get rid of the defaults. I go by the theory, if the person you are trying to keep out is physically in your booth and trying to connect to your equipment, the security breach was allowing them physical access to the booth.

Policies around here are getting stricter and stricter. stuff like GDPR compliance, NIS2, etc. is requiring companies to re-evaluate their risk exposure and not resetting default passwords is one of those practices that's a big no-no. Obviously, customers are notoriously bad at archiving and retrieving such information, especially when it's the most necessary. That's why I nowadays tend to store a backup copy of many of those passwords in my password vault...

For the most part, what is the worst a "bad actor" can do to your system? Delete content? With most any reasonable backup scheme, you should be able to get out of most other tampering relatively quickly. I'm just not worried about people accessing the server or projector or sound. I already make it a need to be on the physical network to do it. Some systems depend on default passwords for content transfers to work (I've found out). Normally, I do provide the end users their own unique passwords for the site and I tend to make them fast/easy because, again, they are physically connected to the network when using them...no remoting in.

I guess the booth networks of a cinema are a sufficient niche that it's not on malware producer's radar. But I've seen quite a few targeted ransomware attacks on high-profile targets, some of them pretty dedicated.

Hypothetical scenario: Let's say you're running a medium to big cinema chain, your booths are all of a standardized design. Now, somebody managed to get into your theater management networks via the central office branch and they crypto-lock all your IMSes and ICMPs for example, because the password on all of them, was the same or simply the default password. Now they want some god-awful amount of money in their favorite cryptyocoin, otherwise you won't be showing a single show the coming days.

While this might sound a bit like science fiction, it's, unfortunately, entirely plausible nowadays that such a thing could happen and similar things are happening each and every day. I just yesterday got a "nice" e-mail from one of the biggest telecom providers around here that they've leaked a ton of my personal information after a large hack of their ENTIRE customer database. They're still figuring out what was actually impacted and what not.

[Steve Guttag]

I'll take my chances. Getting an SMS server back up with a clean install is what, 15 minutes per? A Doremi could run off of a USB stick. Again, the slowest return would be the loss of the content (movies themselves) but getting a cinema device back to factory is normally pretty brisk.

If you are going to run everything via cloud with a centralized system, then you do have to due diligence in how you secure that pipeline.

Then again, you probably let Windows update when IT wants to and get all of the bugs it has to deliver (January 2026 was a bad month). I think security comes with locking your door at the firewall and the lock on the booth door.

[James Gardiner]

What do you exactly mean with non-routable? Do you imply they don't share the same broadcast domain, so they're technically split below layer 2, or do you simply imply they don't share the same IP subnet, whereas they still are in the same broadcast domain?

What I mean is isolation at Layer 3, and in practice also separate Layer-2 domains.

The media network has no default gateway, no router, and no L3 device capable of forwarding traffic to any adjacent network. Media devices can only communicate with other media devices on that network.

In addition, the media network is not bridged into any other broadcast domain — it is either on physically separate switching or on a dedicated VLAN with no L2 or L3 path to management or front-of-house networks.

So while I describe it as “air-gapped at Layer 3,” the intent is that there is no routable or bridged path out of the media network at all.

[Steve Guttag]

James, what "media device" (thinking catch servers, like DCDC) don't also have a connection to the outside? Since they are on the media network and have access to world, there is a bridging path there. Now, one could use some layer-3 switching to put those devices on their own VLAN and grant them access to the media network and, even better, set up rules such that they are limited to just FTP transfers to the media network to a specified address (the LMS). However, since the TMS and LMS are often one and the same...that could still be the breach in the system. Once again, the TMS/LMS is the one place that touches every network in the booth (except things like Q-SYS or traditional A/V stuff, and the like).

At some level, you need a degree of trust and dependence on the content servers to have protected their systems well enough not to be the source of a breach.

[Marcel Birgelen]

What I mean is isolation at Layer 3, and in practice also separate Layer-2 domains.

The media network has no default gateway, no router, and no L3 device capable of forwarding traffic to any adjacent network. Media devices can only communicate with other media devices on that network.

In addition, the media network is not bridged into any other broadcast domain — it is either on physically separate switching or on a dedicated VLAN with no L2 or L3 path to management or front-of-house networks.

So while I describe it as “air-gapped at Layer 3,” the intent is that there is no routable or bridged path out of the media network at all.

I don't think I need to tell you this, but the reason I asked is because I've seen many times that people simply run different IP subnets on the same broadcast domain and deem themselves secure. The thing is, you only need one privileged device on such a network and you can easily sniff out what the other IP ranges are by listening to broadcast traffic and then claim an IP address in that range yourselves.

So, unless there is separation at layer 2, via physical separated switches, VLANs or fancy SDN, there is no security at all.

The term "air gapped", like Steve indicated, might be misleading, as it literally implies an "air gap" between stuff, so not even a connection on "Layer 0". Maybe "virtual air gap" is the term to go, it fits neatly into all those other things virtual today... Virtual LAN, Virtual Machine, Virtual Private Network, Virtual Print Fee, Virtual Reality....

[James Gardiner]

At some level, you need a degree of trust and dependence on the content servers to have protected their systems well enough not to be the source of a breach.

Yes, all delivery systems are directly on the media/management network so the TMS or, when needed, a player can directly ingest from them. But no equipment could have them set at a gateway. So lateral movement would be problematic.

The Media delivery device could be doing some NAT tricks to make the projection network NATed into a global delivery network implementation. Allowing them to directly connect to all players and do things like monitor them and download playback logs. (Which some times I wonder if thats why they give it away for free.. most of the time. These companies do't do anything for free.)

But in general, I would expect those implementing these devices likely have suitable firewall rules to block any mischif. But yes, these delivery devices are a vector in that a cinema could be hacked from external. But then again, POS vendors and likly others are doing this type of thing all the time too. Comes back to security cost over reward again.

[Steve Guttag]

FWIW, I don't allow catch servers on the management network (why do they need to be on that network?). They get to be on the media network...that's it. I don't even provide them internet. If I was required, I would probably put them on a separate DMZ type LAN.

[Marcel Birgelen]

FWIW, I don't allow catch servers on the management network (why do they need to be on that network?). They get to be on the media network...that's it. I don't even provide them internet. If I was required, I would probably put them on a separate DMZ type LAN.

They need Internet for software updates and I think the stuff from IDC also pulls configuration and index data via the Internet. I'm not familiar with DCDC, which seems to be more prevalent in the U.S.

Around here, Internet distribution via e.g. Gofilex/Deluxe is far more common than satellite distribution nowadays. I don't put those machines in the Media nor Management network. They need open Internet access, as their CDN endpoints are almost impossible to firewall. They're using common cloud providers like AWS, which change IP endpoints all the time. I put them into a separate zone, which is more akin to a DMZ. That means that all traffic between the catch server and cinema servers/TMS will go through the firewall, but when sizing the firewall correctly, that should not be a problem. Gofilex/Deluxe hates me for doing this, but I don't trust their stuff.