Unhelpful communication from Dolby about media block certificate extensions

[Frank Cox]

Without that I might have strolled in an hour before doors like normal and discovered it then.

That's a seriously bad practice. If you consider what could have happened in this situation you can see why.

And maybe it'll happen again tomorrow, or next week or next year.....

[Ryan Gallagher]

That's a seriously bad practice. If you consider what could have happened in this situation you can see why.

And maybe it'll happen again tomorrow, or next week or next year.....

For sure, I should clarify that is not the norm, ingesting the key right before the show that is. But when you only do one off screenings, the keys often arrive close enough to the screening that they don’t align with any production shifts in the building.

We try to stay ahead of it by confirming key details against the CPL from home, as at least the DCP tend to have arrived early enough to already be ingested.

I would like VPN access to the booth, or some kind of key ingest automation, but none of that is implemented yet.

But otherwise the projectionist arrival time tends to be an hour ahead for digital singles, and add another half hour for every additional screening. The server stays on if we are in a run of films, but we shutdown for long gaps.

If all we were responsible for was running film, we'd probably make an 8 hour shift out of them and come in earlier. But we tend to be booked well past 40 hours for the week already with our other live events. We are hourly with minimums, so popping in top of the day just to boot up the booth and ingest a key doesn't quite jive with our arrangement. When the building is in summer film mode and film is the person's ONLY responsibility, and they can self-manage their time a bit more flexibly.

In a perfect world there would always be a shift the day before or morning of any run of screenings, to do all the things. But it doesn't always shake out naturally that way, it would have to be a choice to permit/add them.

[Ron Graham]

One of our IMS 2000 boards, despite running the cert extension package and getting the nagging error to go away, apparently the cert did not actually extend and Dolby is just shrugging their shoulders. Super cool.

[Ryan Gallagher]

One of our IMS 2000 boards, despite running the cert extension package and getting the nagging error to go away, apparently the cert did not actually extend and Dolby is just shrugging their shoulders. Super cool.

I assume you sent Dolby logs? It was pretty easy to confirm it was extended in the diagnostic tool interface... that 2038 date etc. Maybe a step was missed on your end? Or you've uncovered a bug in their roll out?

[Ron Graham]

unfortunately there are no logs as the system SSD chips on both of the ims2000s at this particular site died about a week after the cert extensions packages were run, so both don’t show anything other than the system software that is flashed onto all the chips at the factory.

But the more I’ve thought about it the more I think this unit just had underlying issues with the security functions that lead to the package installer showing complete, and removing the nag screen, but not actually uploading the new certain. The secure clock on this server always had very bad drift and eventually stopped allowing even the 6 minutes of adjustment per year. So I’m guessing whatever part of the SM that needed to be open to do time management is also what was needed to upload the new cert?

This unit has also always had a busted info pane for the certificate valid dates, so I think the only actual way to have verified this would have been to open the cert files themselves and put them in one of those online cert readers?

I assume you sent Dolby logs? It was pretty easy to confirm it was extended in the diagnostic tool interface... that 2038 date etc. Maybe a step was missed on your end? Or you've uncovered a bug in their roll out?

[Ryan Gallagher]

unfortunately there are no logs as the system SSD chips on both of the ims2000s at this particular site died about a week after the cert extensions packages were run, so both don’t show anything other than the system software that is flashed onto all the chips at the factory.

But the more I’ve thought about it the more I think this unit just had underlying issues with the security functions that lead to the package installer showing complete, and removing the nag screen, but not actually uploading the new certain. The secure clock on this server always had very bad drift and eventually stopped allowing even the 6 minutes of adjustment per year. So I’m guessing whatever part of the SM that needed to be open to do time management is also what was needed to upload the new cert?

This unit has also always had a busted info pane for the certificate valid dates, so I think the only actual way to have verified this would have been to open the cert files themselves and put them in one of those online cert readers?

Perhaps the drift was bad enough that it WAS doing 6 min of adjustment each year, but that stopped being enough to cover the sync needs. Our 2K4 was out by almost an hour until i got a massive quota adjustment package from Dolby and temporarily set up NTP. Worth mentioning I did our cert extension while it was in that almost an hour incorrect state, the quota fix was later for us. So kinda doubt a drifted clock alone would fail the extension packages, but your note about the info pane never rendering the cert correctly might be more on point in this case?

[Ron Graham]

Perhaps the drift was bad enough that it WAS doing 6 min of adjustment each year, but that stopped being enough to cover the sync needs. Our 2K4 was out by almost an hour until i got a massive quota adjustment package from Dolby and temporarily set up NTP. Worth mentioning I did our cert extension while it was in that almost an hour incorrect state, the quota fix was later for us. So kinda doubt a drifted clock alone would fail the extension packages, but your note about the info pane never rendering the cert correctly might be more on point in this case?

To be more clear, when utilizing the time adjustment function, even doing tiny adjustments like a couple of seconds would eventually result in an error and the time wouldn't change at all. This has been the case on that unit since roughly 2018. So I am guessing that there has long been an SM issue and it ultimately resulted in an inability to actually extend the cert. But who knows, and with the dead SSD chip not even allowing us to go back and view the logs from when I ran the extension package, we will never know and Dolby will just say to go buy an IMS 3000 instead.

And despite my preference for the IMS 2000 and 3000 over GDC on a use interface level, this and sending me a repaired and still faulty IMS 3000 on a recent RMA has me ready to shift fully to GDC. Only issue I've ever experienced with an SR1000 is there super annoying warranty nag every time you log in. Lame and customer un-friendly, but the product has otherwise been rock solid for me.

[Ron Graham]

A quick follow up: I recalled that I had a log dump from earlier in the year, as well as a few from previous years. After spending some time with the Dolby techs again, it does indeed appear there has been an issue with the SM since we initially received the IMS 2000 some 8 years ago. The inability to slip the clock forward or backward was directly linked to this, and there really was no way this device was ever going to get its certs extended because there was no way for Dolby to access the part of the system that it would need to in order to do so. Dolby and our main vendor of equipment are trying to work out a deal for a discounted IMS 3000 as there is actually documentation of this issue on this unit going all the way back to when it was under warranty, but Dolby at the time didn't think it warranted and RMA because the unit was otherwise functioning properly. Annoying situation, but it seems like maybe they will try and do right by us. Stay tuned for the exciting conclusion.

[Colm Nolan]

One of our IMS 2000 boards, despite running the cert extension package and getting the nagging error to go away, apparently the cert did not actually extend and Dolby is just shrugging their shoulders. Super cool.

That could be down to the previous cert hanging around in the background

[Leo Enticknap]

I hear Ron's point about Dolby vs. GDC, and in response I think it would be fair to acknowledge that the IMS3000 has had significant hardware reliability issues over the years, the big one being the boot flash drives failing prematurely. However, it is worth flagging up that Dolby did offer the certificate extensions free of charge, even for the very early generation of Doremi rack servers that are now coming up to two decades old. The same applies to software and firmware updates: the purchase price of a Dolby server or audio processor includes software and firmware updates for its entire service life. When GDC went through their certificate expiration issue in 2024, you had at least to renew a server's software warranty in order to be able to extend them. The purchase price of a GDC only includes software and firmware updates for the bundled warranty period.

The bottom line is that both product lines have their pros and cons.

[Marco Giustini]

it is worth flagging up that Dolby did offer the certificate extensions free of charge, even for the very early generation of Doremi rack servers that are now coming up to two decades old

So to get this enthusiastic feeling about a product I make, all I need to do is to code a SOFTWARE timebomb in the machine and 20 years later be kind enough to issue a FREE package to extend it a further 20 years in the future?

[Ryan Gallagher]

So to get this enthusiastic feeling about a product I make, all I need to do is to code a SOFTWARE timebomb in the machine and 20 years later be kind enough to issue a FREE package to extend it a further 20 years in the future?

I expect the issue is more complicated than that. Certificates come from issuing authorities typically, and one has to choose an expiration, I don’t believe you can get an open ended one unless you are self generating. They may have selected the longest lived option available to them. If they had the option to go longer, since they were basically just computers of the era, I can’t blame them for expecting 20y to be more than plenty of time, which turned out to be a serious under calculation, if it was even an option.

[Marco Giustini]

I see your point, Ryan, but the bottom line is that there was (still is) a software time-lock in those boxes - for a reason or another - and I don't think we should try to normalise that.
If never-expiring certificates are not possible, then something must be implemented so that they can be extended when they expire, without having to feel grateful that it's even an option.

[Leo Enticknap]

Marco's point taken and acknowledged. Mine is that whatever their faults, one of Dolby's strengths is that, in spite of becoming more ruthless about ending parts inventory support for certain models in recent years, they remain among the best in the business for software and knowledge-based support for their older product lines. A couple of months ago I opened a support case asking for suggestions for dealing with a CP65 (not a typo: CP65 with no zero on the end!) that is not much younger than I am, and that had developed a ground hum. I fully expected a response along the lines of, "Leo, would you expect your local Honda dealership to be able to fix a malfunctioning carburetor on an '85 Civic?" But instead, they provided a lot of detailed, knowledgeable advice that enabled me to fix the fault. And personally, I'm more comfortable with their business model of a higher initial purchase price that covers device life updates, and a good chance of help from someone who understands the device deeply, even several decades later, than something that is 15% cheaper to buy at the outset, but after a couple of years the support will cost you.

[Steve Guttag]

Leo, just know, since Dolby went public, they have more than flirted with paid-for software updates. In fact, officially, once your product is out of warranty, you are supposed to pay for them. The DSS servers had to pay for software updates. The CP650, for Surround 7.1 was a paid-for software update (not tied to a serial number so you could "forget" to pay them for future ones) but you may note, they never posted 2.3.6.6.

Somehow, they rationalized making IMS and CP750, CP850, CP950 available to all though officially, it is just for those in warranty. In a twist, GDC which used to be free software for life, switched and embraced the software as a paid update (for out of warranty) and, even more aggressive than Microsoft, nags the crap out of you for having expired software. Thus far, Barco and Christie have also figured out how to have software not be a paid-for thing (except for Post Production, Barco does charge for that one and one needs that if you want to use their hokus-pokus metameric correction for color.

But I do agree, Dolby does tend to make a best-effort at support even their legacy equipment. That has to be a challenge given that much of the talent from those eras have moved on (in some fashion or another).