Film-Tech Forum ARCHIVE: My first deceased ICP certificate battery

This is topic My first deceased ICP certificate battery in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=003789

Posted by Leo Enticknap (Member # 534) on 09-05-2019, 09:41 PM:

Thought this was worth mentioning, as the battery being soldered onto the board means that when this happens, a new ICP is needed; which in turn means a big ticket part and potentially significant down time while it is in transit.

This was in an NEC NC3240S-A, that had been manufactured in August 2011. It had only done 18,642 hours before the ICP failure, meaning that it has spent a lot more time powered down than a projector would in a typical multiplex pattern of usage. Given that these ICP batteries are claimed to have an average lifetime of 10 years, and this one has likely lasted 8-9 (I'm assuming that the ICP was manufactured by TI a few months before final assembly of the projector), this sounds about right.

The reported errors when I found it were:

120 : DLP Ack Fail
301 : System Error
302 : Self Test Error
305 : Key Error
306 : Certificate Error
336 : Red Satellite Serial Link Error
337 : Red Satellite Firmware Configuration Error
343 : Green Satellite Serial Link Error
344 : Green Satellite Firmware Configuration Error
350 : Blue Satellite Serial Link Error
351 : Blue Satellite Firmware Configuration Error
370 : ICP Frame Memory Test Result Fail
372 : ICP Data Path Signature Test Result Fail

Obviously, the 306 means that the ICP has gone out, whatever else might be wrong. I did try a firmware reflash just in case, but no luck. Replaced the ICP with a new one today, and after flashing it to 4.54 and restoring the settings backup, all was good.

Just for giggles, I measured the voltage on the cert battery of the dead ICP - 2.48, so that was the problem, alright.

IMHO, there are two takeaways from this. Firstly, owners and operators of early Series 2 projectors, especially those in arthouse and other low hours theaters (that have spent a larger proportion of their life powered down, and therefore with the battery draining), should be prepared to replace their ICP at any moment (a bill likely to be in the mid-four figures). Service techs should be prepared for ICPs to start going out in significant numbers.

Posted by Steve Guttag (Member # 268) on 09-05-2019, 11:17 PM:

If the OEMs had any class, they'd be proactive about this and get RX units prepared since every S2 projector is going to suffer this fate and since most went in between 2010 and 2013, they are all going to start dropping next year.

Rather than having to buy a new ICP in order to obtain the battery, if they did an RX program, it could be a much smaller nominal fee for a new battery and certificate.

As Leo indicated, the ICP isn't cheap.

Posted by Marcel Birgelen (Member # 6801) on 09-06-2019, 01:19 AM:

Aren't there even laws in place in certain areas of the world limiting the amount that can be charged for the replacement of obvious consumable items, like batteries? I know there are such laws in place in Europe for consumers regarding mobile phones, but they're probably not applicable to business contracts.

There have to be some applicable precedents about stuff that's essentially engineered to fail and who's responsible for the repairs in such cases.

We're talking about the same companies that are eager to point out that the spare part we bought from them is a consumable and henceforth only enjoys a very limited warranty...

Posted by Carsten Kurz (Member # 5396) on 09-06-2019, 03:59 AM:

Leo, now that you have a dead ICP lying around - what about analyzing the battery circuit to see wether there are options to replace the battery on a living device? It can't be too complicated to apply a temporary backup source? There is a pin header next to the cert battery - any chances the battery contacts can be measured through around it?
Any way the adjacent clock battery could be used as a temporary backup source?

I know this is not for the faint-hearted.

Would be better if the OEMs would offer a preliminary exchange program, so those near-EOL ICPs could be salvaged in a smooth transition.

- Carsten

Posted by Marcel Birgelen (Member # 6801) on 09-06-2019, 04:13 AM:

There will be a way, but even if you do find a way, when are you going to do this the first time on a borderline-but-still-operating ICP? [Smile]

Posted by Carsten Kurz (Member # 5396) on 09-06-2019, 04:44 AM:

If you feel safe enough doing it, just consider the manufacturing date, or measure the battery voltage (carefully). There's just one inch distance from the positive of the clock keeper battery to the positive of the cert battery. Wondering if they share a common GND/- ? That would make it easy-peasy...

I would probably set a bridge wire between the two '+', then clip the '+' contact of the cert battery. Then one would have to get rid of the '-' beneath somehow (wiggling...?). I have no pictures of the ICP bottom PCB side around the batteries. Leo, can you supply one now?

BTW - 18642 hrs in 8 years is 6-7 hrs per day. I know many systems with fewer hours than that. And, series 2 deployment started in late 2009/early 2010. So, the first ICPs have been manufactured in 2009.

- Carsten

Posted by Marco Giustini (Member # 4544) on 09-06-2019, 07:09 AM:

What's the manufacturer's (TI) explanation for soldering the certs battery on the PCB? I'm curious.

Posted by Mark Gulbrandsen (Member # 72) on 09-06-2019, 07:14 AM:

I have locations nine years old with Gen 1 ICP's and over 30K hours and the battery is still fine. I was told by TI that the expected life is between 10 and 12 years, which they consider to be the life span of a typical projector. I have no idea HOW they manage to get more than shelf life out of a standard battery, unless the board is also charging it. I did have an ICP crap out when I was changing the clock battery. NEC claims there is a super capacitor across that battery and that it was probably bad.

I had an ICP fail in a similar way to Leo's last year, also in a 3240, but at a drive in. They were pretty sure there was a lightening strike very close to the booth and it may have caused this failure.

Note that changing that battery should be a breeze. You just need to jumper the same power across the one there and then change it using an isolated tip soldering iron. Similar to way you change the batteries on earlier GDC media blocks by placing the same batteries across the terminals while changing them out.

Mark

Posted by Marcel Birgelen (Member # 6801) on 09-06-2019, 07:48 AM:

quote: Carsten Kurz
I would probably set a bridge wire between the two '+', then clip the '+' contact of the cert battery. Then one would have to get rid of the '-' beneath somehow (wiggling...?). I have no pictures of the ICP bottom PCB side around the batteries. Leo, can you supply one now?

There must be some other points on the backup power bus on that board, which can be connected to a temporary, parallel power source.

quote: Marco Giustini
What's the manufacturer's (TI) explanation for soldering the certs battery on the PCB? I'm curious.

I guess it's their way to say: Don't remove this or you're in deep shit. [Wink]

Posted by Bruce Cloutier (Member # 9615) on 09-06-2019, 08:38 AM:

Look to see if a soldered-in battery appears a little bloated as if there was a little internal pressure that has taken it away from being flat. From experience with the JNIOR Series 3 batteries, they may get shorted by the wave soldering process (depending on the orientation) and that ends up reducing the battery life. Also, if a tech sets the board down on a metal surface and there are (+) and (-) pins that may get shorted. Again, this impacts the life.

This is one of the reasons that we went to the replaceable battery in the Series 4. Partly for your convenience in replacing the battery but partly to insure that the battery you get lives for a long as it can. Of course, there is nothing critical in our case like certificates being saved (they are in flash).

Maybe if you see a bloated battery you might be a little more proactive in getting it replaced in case it is in fact going to die on you before its time.

Posted by Leo Enticknap (Member # 534) on 09-06-2019, 09:04 AM:

Agreed that replacing the battery is theoretically possible. There are two contacts on the underside of the board, covered by a blob of plasticky stuff (presumably to insure against accidental shorting when the board is being handled) - this was what I peeled off on the dead one to measure the voltage. I see no reason why you couldn't solder a 3v source onto those temporarily.

The problem is that if you attempt to do that surgery on a living device, you could kill it if anything goes wrong, which will leave you having to provide the customer with a new, replacement ICP. The only point at which you have nothing to lose is after it's died anyways, at which point it cannot be resurrected.

quote: Marco Giustini
What's the manufacturer's (TI) explanation for soldering the certs battery on the PCB? I'm curious.

It isn't a DCI requirement, because Barco's new ICP-X (the ICP you can buy for the SP4K range, if you don't want to put an Alchemy in it) has user-replaceable batteries: the same dual CR2477N holder as is in the Alchemy, in fact.

When I asked this same question, but in the context of why Dolby were replacing the battery holders in the cat745 with a soldered on battery whenever one is sent in for re-certing, someone (can't remember who - could have been Mark or Harold) suggested that the resistance that can build up on spring-loaded battery terminals can make the contact unreliable over time, especially when the current draw is very low. That is the only justification I can think of besides blatant profiteering.

Posted by Mark Gulbrandsen (Member # 72) on 09-06-2019, 09:24 AM:

I think that's nonsense Leo. GDC has a battery holder in all their servers and never had contact issues. Knowing Dolbly it was to save a few bucks and eliminate parts from stock. Dolbly is now run by a BOD, not the highly smart individual that used to. The TI battery is a very critical part and they likely did not want the unknowing pulling it out and destroying the board.

Mark

Posted by Leo Enticknap (Member # 534) on 09-06-2019, 09:34 AM:

The cynic in me agrees with you; though media block manufacturers have addressed the risk of someone accidentally pulling the battery with plastic covers over the holder and labels with words to the effect of "Don't f*** with this, pal, unless you know what you're doing."

As you pointed out in an earlier post, TI likely figured that very few projectors are likely to be in service for more than a decade, and therefore the number of projectors outliving their ICPs would be small enough to be a non-issue. The reality, however, is that many early Series 2 machines are still operating reliably, approaching a decade after their introduction; and their owners are likely waiting on the sidelines until the xenon to laser transition shakes down and beds in, and an investment in laser can be made with real confidence.

Posted by Steve Guttag (Member # 268) on 09-06-2019, 10:24 AM:

No...the reason for the soldered in battery is exactly as Leo stated. For critical things like the certificate, it was the only way to guarantee that things like contact issues wouldn't cause it to fail.

I just checked on a dead ICP I have here, the two negative terminals are together. The header pins to the right indicate, at some level both batteries feed some of them via diodes, since the two batteries are absolutely NOT tied to each other so perhaps the battery change battery is already cooked in so long as the RTC battery is good, one could change the certificate battery...just a thought. I didn't see where putting a jumper on any of those header pins tied the two together sans diode so that isn't it.

As for GDC...they have had their share of certificate failures. GDC's solution also was that they have two batteries in parallel so the odds that BOTH batteries have contact issues at the same time is minimal.

Posted by Bruce Cloutier (Member # 9615) on 09-06-2019, 10:27 AM:

The obvious question is why are the security certificates stored in battery backed memory and not in highly protected flash of some kind? I guess that is some level of protection against tampering. Maybe that's someone's requirement?

Posted by Steve Guttag (Member # 268) on 09-06-2019, 10:40 AM:

Oh yeah...every DCI compliant device is designed with a suicide mission too

Posted by Scott Norwood (Member # 30) on 09-06-2019, 11:00 AM:

This strikes me as a design failure. Would it have killed them to add a second battery or capacitor to power the memory when the primary battery is being replaced?

Posted by Leo Enticknap (Member # 534) on 09-06-2019, 11:24 AM:

Yes; decryption certificates have to be stored in volatile memory per DCI rules, as an anti-piracy measure. However, there is no rule which says that the battery that maintains that memory has to be installed at the factory such as to be irreplaceable in the field, combined with the OEM refusing to offer an RTB replacement service, thereby making the board a disposable device. As far as I know, the TI Series 2 ICP is the only digital cinema component in widespread use for which this is the case.

Posted by Harold Hallikainen (Member # 5405) on 09-06-2019, 11:29 AM:

We have found that battery holders are unreliable at the very low currents (about 10 uA) used to hold up the security chip (private key storage, real time clock, tamper detection, etc.). So, we now solder in a LARGE battery.

Harold

Posted by Steve Guttag (Member # 268) on 09-06-2019, 10:50 PM:

I like the super capacitor idea. Of the mediablocks we have dealt with, without a doubt, the CAT862 (Dolby's used in the DSS200) has been the most reliable on the certificate area. I can't remember ANY having lost their certificate in the ones we service and even those that have sat for extended periods of time).

Posted by Stephan Shelley (Member # 8587) on 09-06-2019, 11:19 PM:

I did have a Cat 862 loose its certificate. It would boot okay but the secure clock date was something in the 1980s. None of the keys would work. This was back when Dolby would fix them. It came back fixed and a bunch of glue around the super caps.

Posted by Steve Guttag (Member # 268) on 09-06-2019, 11:23 PM:

Did it die due to defect or being off for a protracted period?

I've had 3 or so CAT862 outright failures but none were certificate related. They would either not advance the show or put out nothing but white screen.

Posted by Leo Enticknap (Member # 534) on 09-06-2019, 11:34 PM:

The DSP100 also seems to be very reliable as far as the certificate goes.

I service a site that consists of an 11-plex theatre with a two-screen drive-in in the backyard. The main house used to have DSS100/DSP100 combos, but upgraded to DSS220/cat745 systems shortly before I started working for its service vendor. The drive-in still has DSS100/DSP100s. They have several of each on the shelf, formerly used in the main theater, and now being kept to replace the ones that die on the drive-in battlefield.

Twice, now, I have powered up a DSP100 that has not been powered up for at least one year and probably closer to two, and, after a firmware update to 4.7.8.7, has worked just fine.

Posted by Carsten Kurz (Member # 5396) on 09-07-2019, 07:51 AM:

quote: Leo Enticknap
The problem is that if you attempt to do that surgery on a living device, you could kill it if anything goes wrong, which will leave you having to provide the customer with a new, replacement ICP.

That depends on how you explain that job to your customer and prove the necessity to act in time (e.g. manufacturing date/cell voltage).

When the time comes, an experienced tech applying a pre-EOL battery swap on ICPs could make some nice money. This is not the typical electronic failure that MAY happen at some time or not, it WILL happen in masses. Every 'real' series 2 DLP uses an ICP, and tens of thousands have been deployed between 2009 and 2012, and more later.

When performing an ICP clock battery swap, or even a typical 'reseat', I guess I would start measuring the cert cell voltage as a routine job. If properly explained, the cinema will happily pay the little extra time...

Can someone supply a photo of the bottom PCB traces around the ICP batteries?

- Carsten

Posted by Mark Gulbrandsen (Member # 72) on 09-07-2019, 08:16 AM:

quote: Steve Guttag
As for GDC...they have had their share of certificate failures. GDC's solution also was that they have two batteries in parallel so the odds that BOTH batteries have contact issues at the same time is minimal.

All my certificate losses have been from dead batteries and no, the batteries are not in parallel. I have had no certificate losses on the full size servers. The 1.5 volt lithiums are in series. If you measure the output of the plug on battery used while you replace the on board batteries you will find that it's 3 volts. I have never seen anything in any D-Cinema projctor corrode. Also, lithium batteries do not outgas anything that is corrosive. So battery holder contacts are not going to corrode from the batteries themselves like they might from lead acid batteries.

P.S., There is, according to NEC, a super capacitor on the ICP. Just where it is at is not very obvious though.

Mark

Posted by Carsten Kurz (Member # 5396) on 09-07-2019, 08:28 AM:

If there is a supercap on an ICP, it is not for the cert, but for the clock battery. Having a supercap for a soldered-in cell makes no sense.

The photos that I have of various ICPs (mostly Barco), and that NEC I handle personally do not show anything like a supercap. Only classic SMD electrolytics.

Whatever, it should be relatively easy to save an ICP pre-EOL. Best is to learn the procedure on one that is dead-already. Then find a cinema owner willing to understand and take the risc.

I have seen more complicated operations than this to be performed by cinema field technicians in technical bulletins over time. Although soldering is rarely suggested. We do know that media blocks can go brain-dead about a battery-swap even if standard procedures are followed. We know that devices can die even during a 'simple' firmware upgrade. Agreed, most media-blocks can be revived at a lower cost than an ICP replacement. Still means lost shows.

- Carsten

Posted by Stephan Shelley (Member # 8587) on 09-07-2019, 11:51 AM:

This location that had the Cat 862 secure clock reset runs daily but shuts down the equipment fully at night. Worked fine the night before but next day secure clock was at 1980. No keys would work.

Posted by Steve Guttag (Member # 268) on 09-07-2019, 01:21 PM:

quote: Mark Gulbrandsen
All my certificate losses have been from dead batteries and no, the batteries are not in parallel.

They sure are...that is why you can change them but only 1 at a time. I would suspect there is a diode form each so the batteries can't see each other but via diodes, they are going to alternate who is keeping the certificate alive but will drain together. However, even this scheme would tend to keep a low-current issue on the battery holder from causing a certificate loss.

Posted by Mark Gulbrandsen (Member # 72) on 09-07-2019, 03:20 PM:

quote: Steve Guttag
They sure are...that is why you can change them but only 1 at a time.

I guess you haven't done any SX-2001 or similar servers. The only servers that you can pull batteries out of one at a time are the later AR servers or the SX-3000 where each actual battery is 3 volts. The earlier servers with internal media blocks as shown in the picture have two 1.5 volt batteries in series and require the backup battery supply be plugged into the media block before removing the internal batteries... or else.