NOC - Monitoring For Cinemas

[Abdulaziz Saleh]

Hi ,

I'm tying to program something for Cinemas special for NOC, allowing them to get reports before things gets out of hands,
Here are my point so far, can you guys help me to add more things feature before kicking off

• ? CRITICAL — Projector (via SNMP): Projector UNREACHABLE (no SNMP response), Projector REBOOTED (uptime resets), HIGH TEMPERATURE (ambient temp ≥ 29°C), Dowser CLOSED (closed > 20s while show running)
• ✅ RECOVERY — Projector: Projector back ONLINE (previously unreachable projector responding again)
• ? CRITICAL — TMS: TMS OFFLINE for X min (unreachable for ≥ 10 minutes)
• ✅ RECOVERY — TMS: TMS back ONLINE after X min (recovers after offline ≥ 10 minutes, silent if < 10 min)
• ⚠️ WARNING — Shows: SHOW STARTING LATE (not playing ≥ 2 minutes past scheduled start), SHOW PAUSED for X min (paused ≥ 1 minute)
• ✅ RECOVERY — Shows: Show RESUMED (previously paused show is now playing again)

THank you

[James Gardiner]

I have not seen any software like thi that you can purchase.
Typically the NOC service entity develop it themselves or develop plug ins for (typically open source) tools like zabbix, nagios or other enterprise level monitoring tools. They are quite complex as they do so much. Even setting them up requires a high IT understanding as Databases and running/administroing your own hosts (Typically linux based) is required.
They also need a log of labour as every time an alert is fired, and you get a lot of false posatives, you need to check and clear the alerts. You need staff on hand to update any modify the system as the environment changes. (Equipment changes or added/removed)

[Marcel Birgelen]

Keep in mind that there are quite a few components involved with a successful show, depending on auditorium configuration.

- Projector
- Server or IMS
- Audio Processor
- Amplification systems
- Automation systems (for lightning, masking, etc.)

And this is only the beginning, there are much more complex setups in many booths. Most of those components, especially the more modern ones, can be monitored via IP.

As such, there are many potential failure scenarios, some of which come to mind:

- Did the lamp actually strike?
- Did all the audio equipment engage?
- Is all the audio equipment configured to the correct settings?
- Has the screen been masked to the right setting?
- Is there actually any picture on screen?
- Has the right macro been run to zoom/stretch/mingle the picture into position on the hopefully masked screen?
- Did the house lights dim?
- Did the auditorium doors close? (some theaters use auto-releasing door magnets when the show starts, an open door to the lobby, especially in a multiplex, can be VERY annoying)

I could go on for a while... not all of it is trivial to monitor.

How are you going to check if the show started? Are you going to interface with the API of every server? Do you even have access to that API?

[Bruce Cloutier]

We considered providing NOC-like services monitoring JNIORs. They could be enlisted to independently monitor many different events that are expected in a cinema showing. They are able to phone home - so to speak.

The first issue is that we can get telemetry back from only about 20% of product placed out in the field. Of that, maybe only 20% make themselves known to us directly. The rest we get indirect information about though support mechanisms and tools.

Secondly, what is that monitoring worth anyways? It does not make sense for us to create a small revenue stream nor look for ways to drain revenue from you. Although when the economy and international relations go down the toilet as they have, we could really use the added revenue sources.

Instead the JNIOR can utilize SYSLOG servers and SNMP servers. You can set up your own. To our knowledge very few have done that. Heck, very few synchronize their clocks with NTP (which the do by default). As you all have pointed out, the JNIOR is really under-utilized.

Worse, most macros are not closed-loop. By this I mean you instruct the JNIOR to make something happen but there is nothing verifying that what you ask ever gets done. A failure of that is worth notifying you about. But we haven't been asked for any kind of SMS (or better yet Signal) failure notifications. You can already do that by emailing your phone number at Verizon if that is still available. Nobody does. Email is wholly unreliable these days. We have been debating running a Signal server here allowing JNIORs to deliver immediate (and secure) messaging to those who need it. But that runs into the limited access to the Internet that JNIORs have. I also can't see the utility in adding free services when I have to question the cost of those we already provide.

If it were my cinema chain, I would want some kind of real time monitoring. I would want the opportunity to immediately address any failure. I would lose sleep over it. I would not like to think that stuff fails all of the time and no one notices. Okay, typically no one is paid to notice or, at least, do anything about anything that doesn't seem to be working perfectly. It is why we get the general degradation that everyone wants to whine about.

Yeah. Just adding my skepticism here. There is the old thing about leading a horse to water but... I guess those old idioms pass right over everyone's heads these days. We need a more modern version. Maybe something about the utility of doing a Windows update...

[Marcel Birgelen]

Although when the economy and international relations go down the toilet as they have, we could really use the added revenue sources.

Hey, borders over here are still (mostly) open for the "right people", depending a bit on the color of the hat you wear I suppose. Also, over here on the "Dark Side", we do have delicious cookies, made from REAL sugar.

But yeah, selling monitoring business is a volume business and you really need some considerable volume to get to a decent turnover, so you can actually pay some FTEs to "do the monitoring". It's easier to get to volumes if your customer is a corporate client with 500 endpoints than a cinema with maybe 4 JNIORs endpoint?

Something to keep in mind: you're on the hook for everything you didn't monitor, whether or not that was in the contract or not, it's an endless discussion with the customer. "Yesterday the leaf blower blew up inside the warehouse and we were half a day out of commission cleaning up the mess, why didn't you foresee the leaf blower blowing up? What am I paying you for?!" - I did have similar discussions... and I never sold one leaf blower, let alone accessed one via SNMP or similar.

I've sold "monitoring" in the past, but we put it into the complete package, not really sold as monitoring at all, but "proactive support". It's always great if you can tell the customer that he/she has a problem before he/she noticed it or that if they call you can say... yeah, we noticed the Ultimulti is down, our team is already working hard on it!

[Bruce Cloutier]

It is one thing to jump ship and relocate. It is another matter to completely cut ties and not still be obligated to the homeland (that you are trying to escape). Besides, at this point, I would be looking for the right kind of pubs within walking distance and populated with friendly opinionated old farts like myself bound by a fellowship founded a good brew.

On the monitoring side, I would pick one or two of the things that he thinks are CRITICAL and establish some way to log that, and get notification to whomever would care about it. Once you have that working then you can increase the things being monitored. It is not the monitoring that is hard, it is what to do with the information and about any issue when it should occur. The logging really helps. Maybe then if you find the place in a mess you can not only decide that the leaf blower blew up, but who filled it with jet fuel and yanked the cord.

But, really, is it about blame or being able to avoid the issue in the first place? So the signals are that an explosion is imminent, but when (if?) you get a hold of someone they dismiss you in thinking that you are just trying to sell them something else. I mean, you have to do that a dozen times a day as it is. Why should our call specifically get attention?

Boy, I am in a mood today.

[Harold Hallikainen]

A simple monitoring system was the LSS-200 ( http://ftp.uslinc.com/Products/LSS-200/ ). USL offered centralized software ( http://ftp.uslinc.com/Products/LSS-DB/ ). The LSS measured screen luminance, chromaticity, and SPL. The local web interface showed the readings plus graphs of the readings over time. The LSS would also post XML formed data to the LSS-DB where multiple theaters and auditoriums could be viewed. GDC offered a Quality Management System that used LSS sensors ( https://www.gdc-tech.com/cinema-solu...gement-system/ ).

[Elia Orselli]

I have not seen any software like thi that you can purchase.

Arts Alliance Media Lifeguard is made for that: https://www.artsalliancemedia.com/product?id=3 and it's based on Cielo https://cielocinema.com/

[James Gardiner]

Arts Alliance Media Lifeguard is made for that: https://www.artsalliancemedia.com/product?id=3 and it's based on Cielo https://cielocinema.com/

Whats the average costs per screen monitored?
In general, a cinema support site has highly skilled techs and some times programmers, and they implement a lot themselves as to saves costs. But at some point it makes more sense to go with a services solution.

For example in my region, a NOC company was created by the local NATO equivalent to do the NOC. They purchased all the tooling, and after a number of years was sold to Christie for $1 to take on the debt and to ensure the VPF didn't fail due to that key entity failing. I would imagine the high costs of externally services for a lot of the tech required to run a NOC, made it impossible to reach profitability. It would explain why a lot of the VPF NOCs all went away after the contracts ended, or was partially consumed by the largest integrators in that region. Became more about securing sales channels and direct customer relationships for the next projector purchase and service contracts. The number of NOC staff contracted significantly.

Its actually very bad in my region as trained and good technical staff have left the industry and none are coming up to replace them. The only ones left are in their late 50 or much older. (Too old to want to learn new tricks) It does not pay well and if you have the right skill,s there are other industries that will pay a lot more.

[Steve Guttag]

I'm adding that GDC does offer a NOC software. I have not worked with it. But as James has previously pointed out. One of our clients uses Cielo...which augments you own in-house by altering your designated people when a known issue arises. It seems to work reasonably well as I have been alerted to issues without being flooded. It also alerts the theatre, typically, before they know of the issue.

A key to a good NOC system is filtering to know what issues are worthy of escalation and which are mere static. That is, do you really want alerts that the douser has been opened/closed? Software that just reports what SNMP is reporting will flood you with worthless information so the key is to find when something needs attention (including things like lamps that may need to be ordered). To that end, a NOC software has to keep up with all of the hardware changes that may happen (fortunately, I've noticed that SNMP does not seem to change very often, even with product revamps).