IMS2000 Refusing to stay powered on

[Caleb Williams]

We have 2 IMS2000's in our fleet that have started to exhibit a troubling issue: They refuse to stay powered up after booting. They both show no issue booting up initially, but after some time (30 minutes to an hour) the IMS will go completely dark. No lights on at all. We've swapped these servers around to a few different projectors to eliminate the projector power as the cause of the issue, and the issue always follows the server.

We had an IMS2000 do this a couple years ago, and Dolby was no help at all, mostly because by that time the IMS2000 was out of support.

There is no easily noticable physical issue with the pcb or any of its components. I was hoping someone else here had seen this issue and knew if there was an easy resolution? I don't believe the system flash could cause this issue.

Edit: After removing one of the IMS2000's from service and installing it in the Dolby official charging cradle for monitoring, it did not stay powered on long enough to boot. I suspect this is a power issue on the board, but I have no clue what the next troubleshooting step may be

[Marco Giustini]

On the cradle, I would check whether it's the PSU stopping delivering the supply voltage - I suspect the board is overloading the PSU and the PSU is stopping to protect itself. Particularly because you say the IMS stays powered 30m in a projector but a few seconds in the cradle which I suspect uses a small PSU. I would probably use a current meter (either a small clamp or in series with a regular multimeter) and check how much power it draws. Or - even simpler - a Watt-meter on the PSU socket. Then compare with a working IMS2000 or similar server.

If the voltage is still there, then you have nothing to lose by poking around with your multimeter to see what rails are there and what are not.

If the voltage is disappearing, a thermal camera might be able to find the issue.

One thing you can also try is to disconnect everything you can to try to isolate where the issue is. Begin with the HDDs, then I'm not sure if sections of the IMS can be disconnected - I haven't handled an IMS in a while.

[Caleb Williams]

Thanks for the reply Marco. I'm reasonably certain the cradle isn't at fault here. The IMS will not stay on regardless if it's in a projector or in the cradle.

There are a few chips on the board that provide your standard 12v, 5v, and 3.3v downstream of the PCIe connector, and regardless of the state of the IMS they seem to be putting out the expected voltage. I had not thought to remove hard drives, but with the IMS we got little help on from Dolby, that didn't make a difference.

[Ioannis Syrogiannis]

I get the impression that Marco is not doubting the voltages of the cradle or the projector(s), but the amperage (drawn by the IMS). And that is why he is suggesting a multimeter in series or a clamp. Which makes sense.
I was wondering if you used the console out at all, at any point. I do understand that, since the IMS is turning off during boot on the cradle, it seems to be a hardware issue. If it was a full computer, it could have been a BIOS one. Yet, checking the console output might give a hint where logs haven't.

[Marco Giustini]

Ioannis is correct, I am not questioning the cradle.

I'm just suggesting that some overcurrent protection might step in on the power supply side because something on the IMS is drawing too much current. Maybe the projector is able to delivery more current before it gives up (or simply something is overheating in the IMS and the projector fans delay the inevitable) but the small PSU used for the cradle can only supply a few seconds before giving up.

I forgot about the console output: if I were dealing with IMS regularly I'd definitely have a suitable cable to inspect the boot process.

[Gordon McLeod]

I would see if one of the Hard Drives is drawing to much current and the projector and you craidle's power supply goes into protection. I have had Hard Drives fail by drawing to much power

[Caleb Williams]

I'm just suggesting that some overcurrent protection might step in on the power supply side because something on the IMS is drawing too much current. Maybe the projector is able to delivery more current before it gives up (or simply something is overheating in the IMS and the projector fans delay the inevitable) but the small PSU used for the cradle can only supply a few seconds before giving up.

I forgot about the console output: if I were dealing with IMS regularly I'd definitely have a suitable cable to inspect the boot process.

Thanks for this. Unfortunately, I wasn't able to glean anything useful from the console, since when the IMS shuts off there is no indication of error on the console output or in any logs prior to the off-state. In the case the IMS does not boot fully before shutting off, there is nothing different from a normal boot.

Side note - I did leave the unit booted into the BIOS, just sitting on the setup screen and it did not stay on for very long in that state either.

I would see if one of the Hard Drives is drawing to much current and the projector and you craidle's power supply goes into protection. I have had Hard Drives fail by drawing to much power

This is a good lead I will check up on it. I didn't know a shorted hard drive could take down the whole unit. Is it possible the drive starts normally but starts pulling more power after a few minutes? There were no RAID errors on this unit the last time it was able to boot fully

[Marco Giustini]

One thing you can also try is to disconnect everything you can to try to isolate where the issue is. Begin with the HDDs

Leaving the IMS on the BIOS was a good idea, it removed the software from the equation.

Wire yourself to ground and make sure the IMS is also grounded (or touch the chassis first) then power it on and touch the largest components you can see. You might be able to spot a pretty warm area somewhere which might indicate a partly-shorted component.

In the BIOS, can you check for system health and see voltages and temperatures? See if anything changes over time which might look suspicious.

[Leo Enticknap]

To confirm or rule out the RAID drives as the cause of the fault, maybe swap the entire set into another known good IMS2000, if you have access to one? There again, the risk could exist that a drive that has failed in such a way as to cause excessive current suck could damage whatever it's connected to, I guess. An alternative could be to put each drive into a USB to SATA receptacle in turn, and use a SMART parameter reading utility (e.g. CrystalDiskInfo) to see if there are any obvious red flags there.

[Ryan Gallagher]

I agree with others since this is now starting to sound like a board level or hardware fault... start removing anything removable one at a time (starting with drives), but daughter boards are good candidates too. Boot to bios every time since you've confirmed you can track the issue there.

Also look for any "healing type" fuses on the boards, maybe meter or scope those during or immediately after a failure?

Otherwise the touch test, alcohol evap test, or a thermal camera are a great resource at this stage if you can access everything.

I like Leo's thought of testing the drives separately, make sure they spin up and report SMART values connected to a separate system.

It could be something as dumb as a bad solder joint that is cycling to failure when loaded due to thermal changes. You can force some different thermal behaviors in an attempt to trace with some electronics freeze spray. If the subsequent power offs occur faster than a cold boot initial attempt, it very well could be thermally connected behavior.

[Marco Giustini]

If you have a way to check, I would begin with "total power drawn" using a wattmeter at the plug. Good IMS (idle in the BIOS for repeatability) and bad IMS. Compare. If there is no difference, then it might be trickier as it might be some logic shutting down the unit for whatever reason. But if you see the bad IMS is drawing more power, then it might be "easy" to find out what's going on

[Ioannis Syrogiannis]

[...]
Leaving the IMS on the BIOS was a good idea, it removed the software from the equation.
[...]

In the spirit of isolating probable causes, it makes sense. It would need to remove the USB SSD (system flash), besides/if not the HDDs.
For one, I have no experience with BIOS on IMS2000, and if there is any in the traditional form.

[Mark Gulbrandsen]

Yes, sub the drives first. If it's not them, then it is likely some sub-power supply on the IMS main board itself, or bad capacitors, which if memory serves, are all SMT (surface mount). Does Dolby even support them? The last info I found on line is dated 2019. Another tick if you can access it while it's powered it to use freeze spray on individual sub power supply related on board components... But hopefully it's the drives.

[Caleb Williams]

Thanks for all the help guys. Turns out when I leave all three drives in, the server crashes. If I take the drives out and let the server boot, it seems to stay on for much much longer. I'm really curious, since there is no issue with the RAID, which drive it is. If I operate the IMS2000 with only one of three spinning disks it will surely not be happy, but would it allow me to figure out which one is bad without further damaging the IMS? I understand that doing so would wipe out the RAID, and that would be okay, since this IMS has been removed from service.

[Leo Enticknap]

Assuming that only one is bad, if you leave two in, it should work (while complaining of a degraded RAID). It should take you no more than two swaps in that configuration to establish which drive is causing the problem. Start with 1 and 2. If the fault presents, try 1 and 3. If it still presents, try 2 and 3.