Dolby CP950 freezing

[Gronw Roberts]

We replaced all our equipment last year so I'm still learning all the weird gremlins that pop up occasionally. One that's happened a few times is that during live performances of the National Theatre/Royal Opera House the Dolby CP950 will freeze and the sound disappears (but the picture continues to pass through).

This never happens during DCP playback but when streaming a live through our Unique Rosetta Live box over HDMI it does happen. Not every time but enough times now that it's become apparent that a fix is needed.

Do any of you have an idea what might be happening to cause this freezing? I have to hard reset it when it happens.

Thanks.

[Marco Giustini]

I'd expect the acceptable number of crashes on a cinema sound processor to be equal or less than ZERO. So you're right in wanting that fixed.
I'm not aware of that issue myself. One thing Dolby will ask - besides the usual "did you try turning it off and on again" - is going to be the software version. Which one are you running?

You can also save a set of logs and upload it to the log analyser, it might find a thing or two - and it will also check your software version.

Did you ask your integrator to look into this issue? They would probably get logs and forward them to Dolby.

[Leo Enticknap]

The same thing happened to one of my customers recently: the fix was changing the HDMI version in the macro from 1.4 to 2.0 (I think - definitely from the lower number to the higher number).

[Stefan Scholz]

Your enemy in a cinema environment is HDMI. And even worse, cheap "streaming boxes" for alt content, supplied by content vendors or such.
So as Leo pointed out, the first question is, are you on current firmware? Is your HDMI setting set to the most recent protocol version implemented. Are you in legacy HDMI for an older equipment...
Questions most operators in theatres do not even understand.
HDMI is a home technology, is unfortunately encrypted transport, and cinema equipment, unlike Smart TVs is not connected to the internet to do automated upgrading and bugfixing.
So, even though crashes are not really acceptable, it eventually is something to accept when using signals outside the professional interconnections.

[Leo Enticknap]

Your enemy in a cinema environment is HDMI.

+1, and to extend that analogy, an entire armaments industry has emerged to support that war, from cheap Ukrainian drones (HD Fury boxes) to multi-million dollar missiles (networked video infrastructure solutions from Netgear, Visionary Solutions, etc.). Once we start to get into the weeds of EDID exchanges and HDCP, and especially if, in the case of a CP950, we're passing an HDMI signal through a device and disembedding the audio en route, the potential causes of trouble are enormous.

But as a first step, make sure that the CP950 is on its current software/firmware bundle (2.3.3.7), and try swapping between HDMI versions in the macro setup. Those are the easiest fixes to try.

[Marco Giustini]

I agree on HDMI being a "can of worms" which should be avoided or at least treated with gloves (no cheapo stuff). But always remember HDMI was never designed to run with splitters, extenders etc.

What I disagree on is accepting that HDMI can cause the sound processor to freeze (assuming that's what the OP meant: does it become unresponsive and requires a reboot?). That should not happen regardless of what you do with it.

[Leo Enticknap]

I'm with you on that. A bug of that magnitude could cause lost shows. If and when it becomes confirmed and reproducible under test conditions, it should be absolute top priority for the OEM to identify the root cause of, and either produce a software fix or a low to no impact workaround for.

[Caleb Williams]

Earlier versions of the CP950 firmware have memory leaks that cause loss of stability and slowly degrade the performance of the CP950. If a reboot solves the problem then I would bet your firmware needs upgraded.

I have noticed that new equipment sent from Dolby can have old firmware versions. If the integrators that installed the new CP950's last year did not run upgrades that would be the first thing to try.

[Marcel Birgelen]

If it's freezing, you should get it closer to a cosy fireplace, the poor thing!

Don't know if it's relevant, but I guess there is some software overlap between the IMS3000 and the CP950. We've had similar issues with older firmwares running audio trough HDMI on our IMS3000. Since we're operating a screening room that also sometimes is being used as a mixing stage, we've plenty of options to bypass the IMS3000/projector for sound, so we started doing that. I just recently hooked up a BluRay player directly, trying to get Dolby Atmos for pleps working this way and was surprised to see that 1. Dolby Atmos for Pleps (aka consumers) on HDMI is not just an advertised feature anymore, but actually working. 2. We've now done multiple presentations without sound dropping out this way...

[Gronw Roberts]

Thanks all.

I've had a word the installers, they've said we are on one version below the latest software but there's nothing in the update about HDMI bug fixes or mentions of HDMI crashing on our current version.

They saw this in the logs around the time it happened, the audio engine the monitors and controls the internal audio stream had an issue.

• DEBUG BASE FabricInterface - fabint.AudioDrv underflow flywheel: count=127, recover=1, status=0x00
• DEBUG BASE Thread.AudioManager - CPU intensive audio task no longer in progress
• DEBUG BASE System Rebooted ---------------------------------

They also mentioned we should be doing a hard restart every week. This seems rather strange for a professional (or any, to be honest) piece of equipment. They say others have been having audio engine issues/stuttering if the reboot is not done weekly. We never had to do this with our old CP750.

They are currently checking things with Dolby.

I had a look for the HDMI setting, it's set to 2.0.

Software is 2.3.2.5-64619.

The hardware version is 4E.

[Marco Giustini]

Unfortunately a periodic reboot is really required with modern stuff. Things have got more complex. In my honest opinion a reboot is required if software is not great. But everybody seems to be accepting that computers need a periodic reboot.
You'd expect a sound processor to be a simpler device, less prone to those quirkiness but in the end the CP950 runs a web interface and I guess it runs Linux (not sure about what CPU it sports but it's a microcomputer).

Maybe weekly is overkilled but 2 weeks is probably a good idea. Same for your server and projector.

That said, why not turning the sound processor off at night? It doesn't have hard drives inside (HDDs don't like being turned on and off all the time) and unless you run 12 hours of shows a day, I think it might be a good idea?

[Leo Enticknap]

There is a known issue related to RAM and HDMI usage in the CP850. I had one site at which, typically after about a month, the CP850 would stop accepting automation commands from the Crestron integration system. After having ruled out obvious causes, I opened a support case with Dolby. About their first question after looking at the log was if the HDMI passthrough was in regular use. It likely accounted for around 90% of the CP850's activity: this was in a college lecture theater, where HDMI was in use eight hours a day, five days a week, but they likely only played four or five DCPs a quarter. Dolby suggested scheduling a weekly reboot in the middle of the night, explaining that intensive HDMI usage can fill up the CP850's RAM and make it unstable. We did this via the Crestron, and had no problems with that CP850 for three years after that, until the unit was upgraded to a 950A (that college has a policy of replacing everything it considers to be a piece of IT equipment every five years). Likewise, I've had CP850s that only ever play AES3 or Atmos (as in, they don't even have anything plugged in to the HDMI jacks) that have gone for years without a reboot, without any trouble.

Maybe the CP950 handles HDMI in a similar way?

[Harold Hallikainen]

That sounds like a classic memory leak where memory is allocated when HDMI operation starts, but the memory is never released. In my MUCH SIMPLER microcontroller code, I rely on local variables in functions for temporary RAM (on the stack) which is automatically released on a function return. In other stuff where I'm using new or malloc, it is done during initialization (instead of on each call of a function) and stays in memory until power down. This can result in a bunch of memory being allocated and going unused for long periods of time, but avoids the issue of allocating and not deallocating. Another trick I use on microcontrollers is to fill the memory between the heap and the stack with a repeating pattern during initialization. Later, I can look at that area of memory and make sure the pattern is still present indicating we have not had a stack and heap collision which can occur due to overallocation of the heap or a memory leak. Finally, it's common in microcontroller products to include a watchdog timer that resets the system on a crash. Ideally it never happens, but before I added a watchdog timer to a product, I had to drive 100 miles to push the reset button.

I know what they are doing is MUCH more complex than my projects. The power cycle is a good workaround until the problem is resolved. It would be interesting to have the system periodically log RAM usage so leaks can be detected. The IRC IR emitter periodically logs the usage of each of its buffers.

[Marcel Birgelen]

Alternative solution:

1. Invent Time Machine.
2. Go back to 1972.
3. Kill everybody that was involved with the invention of "C" or its predecessor "B"... or to be sure anybody involved with "UNIX" or even better: "Computers".
(Since you are a programmer at heart AND have a Time Machine, you may try to iterate on it a few times to look which solution works best.)

Maybe I watched too many movies?

[Harold Hallikainen]

I really like C since it is "close to the machine." I first hand coded MC6800 code (still remember 0x7e is JMP, 0xbd is GOSUB, 0x3f is SWI, etc.). Then 6800 assembly, FORTRAN, COBOL, PDP-8 assembly, PDP-11 assembly, and C. But, as with any language, the machine does what you tell it to do, not what you want it to do.