IMS3000 Flash Card Storage Failure – Anyone Else Facing This Issue?

[Caleb Williams]

The newer model of replacement flash chips is Swissbit. As far as I can tell the only warnings come from when you put logs from the effected server into loganalyzer.customer.dolby
com. As long as the flash is at least as performant, the server should run without any issues.

I have a script I like to run periodically to check for errors on one of those flash partitions. It is below:

#!/bin/sh
grep -q 'lost page write due to I/O error' /doremi/log/kern.log

if [ $? -eq 0 ]
then
echo
echo "\e[31mPOTENTIAL BOOT FLASH ISSUE FOUND\e[0m"
echo
else
echo "No issues found"
fi

[Leo Enticknap]

Effective August 31, Dolby will require that defective IMS3000 boot drives be returned "as part of the advanced replacement process" :

image.png

I hope this doesn't mean that they won't ship a new one until they get the NFG drive back. The affected screen could be down for days if so.

[Harold Hallikainen]

Since it is still an "advanced replacement," I believe they will still immediately ship out the replacement. From their notice, they are just trying to see what's going wrong by analyzing ones that have failed.

[Leo Enticknap]

I also hope that they still really mean advanced when they say advanced, but I can't think of any other way that they can police this. Maybe they don't intend to, and if they get a few dozen back, then that's all they need. My point is that when one of these things fails, the screen is down, and so time is of the essence in replacing it. Most site staff don't have the knowledge to create a boot flash drive and then do everything that has to be done afterwards to get the IMS3000 running again (change IP addresses, restore backups, etc.), and so the replacement drive needs to be shipped ASAP to avoid the need for two service calls from a tech (and therefore double the labor cost for the end user).

One of the times I swapped out one of these drives, just for giggles, I hooked the one that the IMS3000 wouldn't see anymore to a Dupont to USB adapter, and thence to my laptop. With just that arrangement, the laptop wouldn't see it. I then connected the adapter through an externally powered USB hub, and the laptop read the drive without any problem. CrystalDiskInfo declared that the drive was in good health, too! So I wonder if there is a resistor on the board that increases in resistance over time, to the point at which the USB bus power on the IMS3000 no longer provides enough juice to power it up?

[Joseph Boutros]

Hi Leo,

Our RMA policy remains unchanged. When a failure is confirmed on an under warranty unit, we ship an advance replacement. With that said, the defective boot flash only needs to be returned when the IMS3000 is under warranty.

If you have any further questions, please do not hesitate to contact us at cinemasupport@dolby.com

Best regards,
Joseph
Dolby Cinema Support Services

[Caleb Williams]

Anyone actually have a in-warranty failure of the boot flash? Most of ours died literally months after the factory warranty lapsed. I've got a pile of flash I'm surprised they're not interested in.

[Steve Guttag]

All of our circa 2020 ones had the boot drive fail in-warranty.

[Caleb Williams]

All of our circa 2020 ones had the boot drive fail in-warranty.

Huh... Ours were from 2020 as well. Do you turn your servers off at night?

[Leo Enticknap]

They clearly had a bad batch around the time of covid because, like Steve, I had a crap ton of them fail that were new around that time within a year or two of going in to service.

Those account for the vast majority of the drives I've had fail, but there have been a few others, including more recently the drives in some in IMS2000s as well as in 3000s. That is why I wonder if my experiment described above is significant, it that it shows that there is something about those drives that increases the USB bus voltage they need to work as they age. The hypothesis would be that for whatever reason, this process happened more quickly in the circa 2020 batch, but that it's inherent to the design of the board and is still happening in all the others.

[Steve Guttag]

Yes, those servers (as almost all of our clients do) turn there servers off at night. I'd say that particularly bad batch of servers were sold from late 2019 through sometime in 2021 with 2020 being an understandably low-sales year.

A downside of the powering down at night is that the IMS3000 is can be flaky on boot up (temporary FIPS locks, SM doesn't come up). I have a Q-SYS module that just looks for some of those conditions.