IMS3000 Flash Card Storage Failure – Anyone Else Facing This Issue?

[Shibu Paul]

Hi all,

Recently, the flash card storage on our IMS3000 server failed for a large number of our screens. We are currently running the system using a bootable USB as a temporary solution.

I’d like to know if anyone else has experienced this issue at their sites. Also, Dolby has still not resolved the problem, and we are looking for any guidance or similar cases from the community.

Thanks in advance.

[Caleb Williams]

We have had about the expected number of failures for IMS2000's. However, the IMS3000 system flash card failure rate is abysmal. I have replaced almost half of the system flash cards for our IMS3000 fleet of 37 since the fall of 2024, and most of those IMS3000's are less than 3 years old. I believe the unofficial line from Dolby has been that there was a manufacturing defect or quality control issue with a couple batches of those cards used in the IMS3000's. There's not any way to prevent the failure from happening, and there isn't anything on the IMS3000 that is making the card more susceptible to the failure, (like absurd write rates on flash).

To combat this problem our in-house IT (me) obtained updated goldenmaster images for the IMS2000 and IMS3000. We purchase the replacement flash ourselves, and also flash and restore the server ourselves. I was able to source the newer Swissbit branded flash card from Digikey, however it looks like that part is end of life. We purchased enough stock to replace 3/4 of the flash of our IMS3000, although that same part works in an IMS2000 without issues.

We also had to purchase an adapter to allow a computer to communicate with the flash card using usb. A link to that adapter is here: https://www.amazon.com/BAIRONG-Dupon...s%2C152&sr=8-1

Both IMS2000 and IMS3000 now have a version requirement to maintain operation, so if you decide, like we did, to pre-flash some cards you'll want to make sure they are re-flashed with the updated software prior to installing them on a server to prevent bricking it. For example, if you try to use an IMS2000 goldenmaster using any software version lower than 2.8.52 you will undo the certificate extension previously completed and render the server useless. I don't think a similar result is quite as drastic with an IMS3000, but I'm not about to try it. Using the updated IMS2000 goldenmaster from Dolby we were able to recover an IMS2000 on Christmas Eve without any problems. Dolby will provide the updated goldenmasters on request if you send an email to cinema support.

I have manually copied config files onto new flash and successfully restored servers before, however the restore from backup is much more painless than doing it by hand. We've got it down to where I 1) flash the flash (heh), 2) manually set up the network config on the new flash (edit /etc/network/interfaces), 3) ship the card to the theater that needs it, 4) talk a trusted manager through removing the server and installing the flash card, 5) restore from backup, and 6) complete marriage. Of course we finish up the whole process by confirming that picture and sound are okay, because anytime something is disconnected from the backplane or network, there's a chance it didn't get put back together completely correctly.

While we try to stick as close as we can to OEM parts, my experience has been that anything that fits on that header and has enough storage will work, at least 4gb. If you're using something not "approved" by Dolby the log analyser will show an error specifically about that part, but it does not effect the operation of the server at all. The IMS2000 has quite a bit more space around that header so you can get away with a flash card that has larger components on the top and bottom. However, the IMS3000 requires a flash card that has small or no under-side components to prevent the flash card from interfering with main-board components and eventually breaking.

Happy to dive more into this issue if you have specific questions. I've done a lot of digging to make sure we're not doing anything that would brick any servers

[Steve Guttag]

It has been my experience that any IMS3000 that was manufactured in 2019-2020 will have a short-lived OS-SSD. And note, due to 2020 being a "special" year, it prolonged that batch of SSDs getting into the field.

One thing that will absolutely hasten the failure rate of the SSD, even in the IMS2000 is to not allow the server to properly shutdown before killing power. So, if you are in a theatre that is used to just powering off the projector...know, each time you do that, the SSD is taking a hit. Where I have IMS2000s that have been shutdown properly, the SSDs have had not had any problems and have achieved, at least 5+ years.

Thus far, I have not taken Caleb's approach of obtaining my own and flashing them though that sounds like a good idea, from a cost standpoint. I have sourced mine through Dolby. I do keep reasonably current (and certificate safe, to Caleb's point) versions of the DCP2000, IMS2000 and IMS3000 OS software for making bootable USBs.

If you have a Dolby Atmos system...and I cannot emphasize enough, that configuration is not part of the normal IMS3000 backup. Be sure to make your own back up of the audio processor portion as that will need to be reapplied. I also believe that the KDMs are stored on the SSD so they will need to be replenished too.

Depending on one's interaction with the server, you can get some hints that the SSD is going to die. If logging in becomes difficult or the web UI freezes. It is about done.

[Leo Enticknap]

I don't think a similar result is quite as drastic with an IMS3000, but I'm not about to try it.

When someone (I think it was you) flagged up the IMS2000 potential time bomb, I did an experiment while replacing the dead boot flash of an IMS3000 that was still in warranty and at an easily reachable site. It was at a post house that didn't need the theater for a few more days: therefore, no harm done if I bricked the IMS3000 and there was a delay of another 24 hours until the warranty replacement arrived.

This IMS3000 had previously been updated to 3.21.7 before the boot flash drive died. I installed a USB boot drive on 3.5.13: after boot up, the IMS3000's web UI complained of a version mismatch with the security manager. This entry was missing from the diagnostics page:

image.png

I then shut down the IMS3000, swapped the USB boot drive for one with a 3.21.7 image, and rebooted. The above entry was back again, and no version errors were reported. A Dolby Log Analyzer readout also gave no indication of a certificate problem. So based on that, I'm 99.9% sure that if the SM and firmware have been updated properly and you see the certificate extension in the diagnostic page, that subseqent booting of an IMS3000 from an emergency USB image that is 3.5.13 or earlier will not reverse the certificate update and brick the server.

Edit/Afterthought on the USB to Dupont adapters mentioned by Caleb: I've found that the Dolby boot flash cards need rather more power than a typical USB memory stick, with the result that, depending on the computer, it's sometimes necessary to connect that adapter via an externally powered USB hub. For example, my home desktop will see one of these drives directly through the adapter, but my field laptop needs one of these with the wall wart plugged in to do so.

[Caleb Williams]

Leo, thank you. Good to know one could update an IMS3000 back to functionality. Hopefully that will hold true after the new year.

To Steve, we should probably clarify that the failure of the system due to flash might not be directly from the gradual degradation of the physical medium of the card. If the IMS fails to boot, for example, it is possible that improper shutdowns caused file system corruption that rendered the IMS inoperable, but the storage medium of the flash card may still be in good condition. The improper shutdown will not generally cause flash failure by itself, but it absolutely increases the risk of the media block going into FIPS lockdown and should be avoided. That failure mode can be fixed by re-flashing the system card.

The degradation of the flash card is generally preceded by symptoms such as a 404 instead of the login page, the login page available but no other pages, or the web browser showing a "connection refused" as Steve mentioned previously. We've generally been able to catch flash failures early by checking the kernel log file and looking for a read/write failure on /dev/sdg5 or any other partition. You will see read/write failures way earlier than the 404 or other symptoms. We run the command below as root to check for the problem ahead of time, and that has saved us some downtime in a couple of cases.

cat /doremi/log/kern.log | grep -i "i/o error"

Also from a cost standpoint, we've found that Dolby marks up the replacement part by about $100 in our region, and they sometimes come with the wrong software on them. It is generally a good idea to confirm that you have the right software before installing the flash in a server. I've had a couple IMS2000's that suddenly thought they were ShowVaults with missing components. Although recoverable, probably not a good risk to take if it can be avoided.

[Steve Guttag]

Oh, I have no doubt that Dolby marks up the SSDs (and all parts) by quite a bit. Often manufactures will mark a part up on the order of 5x of their costs. That is not a Dolby thing. However, one also has to assign a value to the time of obtaining the right part, standing behind that part (I've had a replacement SSD fail and they replaced it, including covering shipping) as well as the time it takes to prepare the part so it is as close to drop-in as possible. There is value in all of those steps. There is also value in putting parts on the shelf (essentially putting money on the shelf). I recently obtained SSDs for another product where, as you noted, there was a substantial price difference between the OEM and what I could source in the open market. But that took time to source, even more time to obtain, and time to load them with the right firmware so that they would work. In a sense, the client is now paying US to do all of that rather than the OEM.

I have seen where on an IMS2000...where the client was just killing the projector power (Barco DP2K-12C) and that lead to the OS drive dying. Perhaps refreshing it was an option but if that took even an extra hour fussing with or having a repeat, the cost of replacing the SSD was cheaper than pursuing fooling with a multi-year old SSD. A USB drive was used to keep the system going until the new SSD arrived. The client was able to swap that and I was able to remote in to get everything set back up where either was. It was cheaper/more reliable, in my opinion, than fooling with the old drive. The client now will NEVER just slam the server off like that again.

[Caleb Williams]

I keep forgetting that you guys aren't in-house IT, and thus don't necessarily benefit from buying a part direct instead of from Dolby. Thanks for your correction there.

I have seen where on an IMS2000...where the client was just killing the projector power (Barco DP2K-12C) and that lead to the OS drive dying. Perhaps refreshing it was an option but if that took even an extra hour fussing with or having a repeat, the cost of replacing the SSD was cheaper than pursuing fooling with a multi-year old SSD. A USB drive was used to keep the system going until the new SSD arrived. The client was able to swap that and I was able to remote in to get everything set back up where either was. It was cheaper/more reliable, in my opinion, than fooling with the old drive. The client now will NEVER just slam the server off like that again.

On this point, do most of the theaters you all service manually shut down servers at the end of the night? Or is shutting down equipment at the end of the day even normal? We recently purchased a location that had Barco projectors with IMS3000's they left on 24/7 unless an issue required a power-cycle. A couple of the Barcos had almost 130k hours of on time, which is wild to me.

Our locations turn off breakers at night. After upgrading show servers from DSS to IMS, we found that closing managers were almost immediately untruthful when asked if they had shut things down as they were trained. To avoid issues, we created a simple application that listens for an automation cue, then shuts down the IMS and puts the projector in standby after the last show is completed. It is completely automated, and the only time we catch people not shutting things down correctly is if there is a no-show and they're trying to leave early. That scenario is a little harder to correct without convincing them that if they can't be trusted to care for thousands of dollars worth of equipment properly they won't be working with us anymore.

The reason for the dedicated application is a loop condition where if the IMS shutdown cue is hit inside an spl it will immediately shut the server down without completing playout. Then when the server is again booted it will automatically start playing the same spl previously loaded 10 or 20 seconds before the shutdown cue is hit again, causing the server to immediately shut down. The only way to break the loop is by breaking marriage, because playback will resume before the web ui is fully available. Using a separate application allows us to add in a timer to allow playout to complete, then gracefully shut down the server and projector. I think the original Dolby TMS had an answer for this, but we migrated to Screenwriter after the Dolby TMS project was killed off and we needed an alternative.

The application we built runs on top of nodejs (which is available on most platforms), and is small and easily modular. We've built it against an address library so that the relationship between the IMS that sent the cue and the projector address can be discerned. It would be easy to build your own address library, or modify the application to only shut down the IMS. If anyone is interested there is github repo I can send in a private message.

[Steve Guttag]

Caleb,

I'd say that 90% of our clients shut down at night. We use the Eprad eCNA automation in about 99% of our screens (there are a few odd screening rooms where we either didn't design it in or what we would have done with it was replicated by something like Crestron). As such, we automate the boot/shutdown process. We give them a System On/Off switch that tells the server to shutdown and then it will power down the projector. Depending on the venue we have it controlling the sound rack, exhaust/HVAC...etc.

We still have a large install base of DSS200 servers and are using the DSS "Show Manager" software in a mini-plex configuration with one or two that still have the DSL show library. For our non DSS systems and those that were on a VPF, once upon a time, the bulk are on ACE (and a couple remain on TCC...but I think by this time next year, they'll all be on ACE). I think a trick of automating a shutdown is getting that cue that indicates that it is at the end of the last show. And, on start up, one needs to have something other than the server being able to decide when it is time to wake things up. ACE, at the moment, does not have the ability to do these. As I recall, screenwriter does. Now, the eCNA does have its own clock so one can have an arbitrary schedule for power up (and power down).

Eprad has a real cool thing called the "Nano-Host." It gets the schedule from the POS (though it MUST be of the BLOAD format). With that, it knows the show times and the ticket sales. One unit works with the entire complex, regardless of screen count. We have one on a 12-screen complex, the largest we are currently supporting. The lamps only turn on if a ticket is sold, so it is a big savings on the all-too-frequent lean months. The owner remarked the saving on electricity alone paid for its implementation in less than a month (it isn't very expensive). Since the schedule is then known, automating the wake up/shutdown is cake too. We've augmented that to be aware of typical show transfer nights and delay shutdown on those nights by some hours to make sure even Avatar can make it to its destination

Now, if I have Q-SYS on a site, I have it checking with the server to determine if ingests are happening and if so, the shutdown is postponed until the ingest flag clears and then the system is allowed to shutdown. We've also utilized Q-SYS to aid in boot up, particularly of the IMS3000, which seems to have some bad mornings where not everything wakes up properly. If Q-SYS sees it in that condition, it will tell the automation to shutdown and try again. Mind you, we're talking about 0-5 days a year on a given server...so it isn't always easy to catch. What I've found is that those that haven't experienced the IMS3000's boot up issues, also don't shut them down at night.

Pumping dirt 24/7 through projectors is just nuts. I never gave it any thought on the box server days (DSS200, SX-2001, DCP2000)...they were built on 24/7 operation and often went out of their way to make shutdown more difficult. No mater the filtration on the projectors, dirt "will find a way" to get to where you don't want it. It is amazing how dirty the optical path gets over time.

[Marcel Birgelen]

I think that in the day and age of SSDs and laser light sources, it makes more sense to switch off projection equipment when not in use for a prolonged period than it used to be. The only reason to regularly power-on your equipment is to make sure none of those certificate-boobytrap batteries run dry. Still, for filesystem consistency, even on your IMSes, I think it's important to do a clean shut-down and not just to pull breakers. For hard power-cuts on equipment that doesn't allow automated power management, I prefer to use switchable PSUs. They're easy to automate and can also be switched remotely.

Regarding the boot SSD drama on the IMS3000: We've also had to replace ours a while back due to flash fatigue. We've always done clean shutdowns whenever possible. I suspect that the first generation of "SSDs" they put out weren't really SSDs, but just glorified flash disks with an internal USB interface. I guess at first they thought they could manage it with some firmware upgrades, reducing the amount of writes to said SSD, but like with many of those crappy flash drives, once you start to put some real-world load to it, it only takes a simple thing on your rather complex stack of OS components to put in some serious write-cycles, to put the flash disk over the edge. For me, this reeks of a design failure, but Dolby doesn't seem to see it that way.

[Steve Moore]

Just out of interest, are these the same boot SSD drives used on the Doremi DCP-2000s with the later motherboards (not the original DOM drives)? The only reason I ask is in our chain of cinemas had most of ours fail a few years back within weeks of each other (around 2019) in our Doremi servers for which we purchased MB upgrade kits which came with the new USB type SSDs. The "replacement replacements" we got sent from Dolby failed even quicker (some within days and one within hours of fitting.) In the end I bought some other brand blanks and cloned them with a home made cable I knocked up which looks simular to the Amazon one Caleb linked. So far (touch wood) these are still running fine (though Dolby Log analyser always flags them as not approved devices!) At the time they said it was a certain batch of drives that they encountered the issue with. I guess Dolby are in the hands of who ever they bought the drives from at the time.

[Marcel Birgelen]

They look essentially the same. Our replacement SSD is from a different make and model than the original one. But it's essentially an "internal USB DOM" in a "flat" format. I guess they've rotated trough multiple suppliers over the years.

It's essentially the same drama as with SD cards for e.g. a Raspberry Pi or other embedded PC running a full Linux OS. Some models work fine and also take a beating, while others blow up as soon as one service starts filling up your log-files...

Server suppliers like Dell have since moved away from those kind of solutions and swapped back to REAL SSDs as OS boot partitions.

[Leo Enticknap]

Oh, I have no doubt that Dolby marks up the SSDs (and all parts) by quite a bit. Often manufactures will mark a part up on the order of 5x of their costs. That is not a Dolby thing. However, one also has to assign a value to the time of obtaining the right part, standing behind that part (I've had a replacement SSD fail and they replaced it, including covering shipping) as well as the time it takes to prepare the part so it is as close to drop-in as possible. There is value in all of those steps. There is also value in putting parts on the shelf (essentially putting money on the shelf). I recently obtained SSDs for another product where, as you noted, there was a substantial price difference between the OEM and what I could source in the open market. But that took time to source, even more time to obtain, and time to load them with the right firmware so that they would work. In a sense, the client is now paying US to do all of that rather than the OEM.

Amen to that. In this case, the ready to install SSD from Dolby costs around $100 more than a blank one bought from a third party vendor (e.g. Digikey or Mouser: the last I time looked, they weren't available on Amazon). Having bought the blank one, you have then to burn the Dolby image to it and test it in an IMS before it can released to a customer. That process could easily take an hour from start to finish. The cost to a company to employ a digital cinema tech is well over $100 per hour (in California, at any rate). Add to that the fact that if the part bought from Dolby is DOA or fails within their warranty period, it's on them to make it right and it's their reputation that takes the hit. Once you start to consider those factors, the markup that cinema equipment manufacturers add to generic parts - especially computer parts - starts to look more like a bargain than a ripoff.

Of course you may be in the position whereby the OEM no longer has the part for sale, and you have no choice but to buy it from a third party (DSS220 power supply modules being a regular example in my case). There are also times when an end user decides to do that, believing the OEM's markup to be a ripoff. Mostly, they get away with it. But I've heard of one or two occasions when they didn't and lost shows as a result. In the case of one incident I was told of, someone tried to replace a Series 2 Barco liquid cooling pump with a generic alternative for about 15% of the Barco price. The endgame was sticky blue goo all over the light engine, which had to be replaced.

[Hong Sung Hyeok]

When using Dolby servers, I have to pray.
All I can do is just pray that the OS disk is healthy.

The USB DOM used as the OS SSD has poor durability.

If you have any warning signs, such as web access failure or missing automation commands,
it is best to replace it without delay.

The supplier is innodisk. Just in case, I purchased and am using a USB DOM from the same manufacturer.

[Marcel Birgelen]

I've been told that if you replace it with a DOM/USB SSD not provided by Dolby, it will still work, but it will be flagged as a non-Dolby qualified part. This might have implications if you run into warranty issues, even those that don't have anything to do with the OS boot disk. I was wondering if you buying the DOM from innodisk, who at least has been a supplier of said DOMs for Dolby at one time (and might still be), clears this warning.

Our boot SSD was replaced under warranty, but I'm not really confident with the new one we got either. I was thinking about buying a few of those as a spare, if I could buy a model that wouldn't be flagged as in-genuine, that would be a plus.

[Hong Sung Hyeok]

I've never heard of a non-authentication warning message appearing. but Could Dolby possibly revoke the warranty There is a possibility that it can be detected by analyzing the logs.
I was worried about that too, so I chose a product from the same manufacturer, but I couldn't find the exact same part number.

If your device is still under warranty, use a very fast SSD USB drive. There's even a manual on how to create an emergency boot disk.
When I used a regular USB drive as an emergency boot disk, the web UI was so slow it was frustrating.