Film-Tech Forum ARCHIVE: Failed LMS/TMS

This is topic Failed LMS/TMS in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=003873

Posted by Aaron Mange (Member # 9671) on 12-09-2019, 05:00 PM:

Our IBM x3650M4 server running windows server 2008R2/ Cinedigm TCC has entered a reboot cycle. We are unable to boot from USB or disc, this included windows install, Linux live and Acrionis recovery discs and USB drives.

initially we received a BSOD APC index mismatch error. Nothing we have tried including outside support (sonic) has worked, before we drop an outrageous amount on a replacement , any last ideas I migh try?

Posted by Leo Enticknap (Member # 534) on 12-09-2019, 05:25 PM:

If it won't boot anything, this says to me that this is a hardware problem, not a hard drive or corrupted software problem.

Maybe replace the BIOS battery, and pull and reseat the memory boards as a first step?

Posted by Aaron Mange (Member # 9671) on 12-09-2019, 05:46 PM:

Thank you for the tips.
I was pretty sure it was a hardware issue as soon as it failed to booth any bootable disc or drive. Sonic has continued to suggest I re-image the device to resolve the issue. They fail to understand when I say "the device will not boot anything" and sent me the Acronis true image recovery to try anyway.

It is suggested I send the device in to be looked at, however I wonder how smart that would be given they cant even guess as to what the issue is. I feel wed be throwing money at a moving target.

Posted by Leo Enticknap (Member # 534) on 12-09-2019, 06:07 PM:

Shipping one of those things is a significant chunk of change straightaway, and if they'd charge you to look at it even if they fail to diagnose the problem, that doesn't strike me as a very sensible option.

However, might there be a local IT tech in town who could test the PSUs and memory for you? A guy from a PC repair store should be able to do that, and that would likely be more cost effective than shipping a 50lb server long haul. Unless the mobo itself is pooped, it probably is cost effective to repair the server, if the bad component can be identified.

If you want to try yourself, power supply testers are pretty cheap, though if you can't even boot as far as the BIOS, testing the RAM is more tricky. If the server has more than one RAM board, you could try removing each of them in turn, to see if the computer will boot OK with one of them out. If it will, that's your culprit.

Posted by Marcel Birgelen (Member # 6801) on 12-09-2019, 06:11 PM:

Hardware corruption can't be excluded, but I don't think this is a memory related problem, at least not if it's a server with proper parity-protected memory.

Booting Windows in Safe Mode also doesn't work?

Usually, if you press F8 right after the BIOS has finished its POST routine and Windows starts to load, you get a small boot menu that allows you to start Windows in safe mode.

If that works, then you can selectively start to disable devices in your system. I've seen this before on a desktop machine, where the sound driver was misbehaving.

Posted by Leo Enticknap (Member # 534) on 12-09-2019, 06:12 PM:

But would an NFG sound card prevent a Linux distro and an Acronis image recovery CD from booting as well?

Posted by Aaron Mange (Member # 9671) on 12-09-2019, 06:17 PM:

Safe mode, recovery mode, using an install disc to attempt a repair all failed. Ive decided to call in a local specialist to check out the system, thanks for the suggestion. I will however update everyone as to what the issue was, if its found and fixed.

Posted by Marcel Birgelen (Member # 6801) on 12-09-2019, 06:20 PM:

quote: Leo Enticknap
But would an NFG sound card prevent a Linux distro and an Acronis image recovery CD from booting as well?

No, it shouldn't. But booting stuff from USB stick or other devices often fails for multiple other reasons, especially on servers. Reasons like incorrect BIOS settings or incorrectly formatted drives or incompatible blocksizes. Sometimes, when a RAID is involved, the RAID controller will register itself into the BIOS and boot from the array anyway.

Aaron also didn't state where the boot fails. Does it start loading the Linux kernel and does it panic somewhere halfway thru? Or doesn't it load at all and just try to boot Windows?

If it fails halfway thru loading the Linux kernel for example, the location where it fails may provide a clue of what's wrong with the system.

quote: Aaron Mange
Safe mode, recovery mode, using an install disc to attempt a repair all failed. Ive decided to call in a local specialist to check out the system, thanks for the suggestion. I will however update everyone as to what the issue was, if its found and fixed.

On what error did the Safe Mode fail?

What is that Recovery Mode you're talking about?

And what error did you get from the Install Disc?

Posted by John Thomas (Member # 6509) on 12-09-2019, 09:11 PM:

If you can get to the point of attempting Windows safe mode, you can attempt to boot from external media.

Remove the RAID controller card and see if that changes anything.

Posted by Steve Guttag (Member # 268) on 12-10-2019, 07:02 AM:

2019 has been a very bad year for me and equipment failure. I have changed out more motherboards this year than all previous years combined. As that seems to have been my "hammer" I'm seeing you as having a "nail."

I've changed out Motherboards on multiple brand servers too (SMS servers like Dolby and GDC), not TMS/LMS systems, per say. However, their symptoms resemble yours...don't boot up well...hang at random times, self reboot.

So far, in every case, changing the motherboard resolved most or all of the issue. In one instance, the software also needed to be reloaded (I attribute that to bad hardware allowing software to be corrupted as well). All of those servers are running, at this point and we are talking about probably 6 or 7 now.

I'm wondering if the effects of ROHS assembly/soldering isn't showing itself in long-term stability more so than outright component failure though I could be the odd capacitor that manifests itself that way. In my MB story above, they are not all the same model motherboards either so it wasn't like it was just a bad run.

I know what TMSes that run the TCC cost so it is probably worth the effort to change the memory (I haven't had that problem yet but one server company always seems to suspect it so there is something there...if it is outright bad, most of the time the server will "yell" a bit...if it just has issues, you get random failures) and motherboard. All in, it is typically much less than $1000 plus your time.

Another technique I've used in troubleshooting computer hardware is removing cards, one at a time, to see if boot up will change. I've had a mediablock suck down the power supply such that it prevented boot up as well.

Posted by Marcel Birgelen (Member # 6801) on 12-10-2019, 08:41 AM:

quote: Steve Guttag
I'm wondering if the effects of ROHS assembly/soldering isn't showing itself in long-term stability more so than outright component failure though I could be the odd capacitor that manifests itself that way. In my MB story above, they are not all the same model motherboards either so it wasn't like it was just a bad run.

There has been some limited research into this and since we're now about 14 years later, I think we can conclude that RoHS did very much have a negative impact on equipment failure rates. Especially the first-generation solder replacements have proven to become brittle over time and others have severe corrosion issues. Also, we're still fighting the late effects of the capacitor plague, because some of those bad capacitors still made it into equipment which was produced way after 2007.

Posted by Mark Gulbrandsen (Member # 72) on 12-14-2019, 10:28 PM:

quote:
I know what TMS's that run the TCC cost so it is probably worth the effort to change the memory

Since 98% of my customers went with the GDC VPF program, because it was way easier to deal with, what I did for most of their racks because they are independents and looking to save a lot of $$$$ was to procure 2 year old server pulls. I found a reliable company in the South East to get them from. The reliability has been basically been at 99%, and they all spent their two year life in a server clean room... Have only lost two original Dell 2950 servers so far, and frankly because these servers are so inexpensive to buy, its not worth the time to trouble shoot what's wrong with the original unit. The minimum time they have lasted is 6.5 years and all but two of the 40 some odd systems I built with the Dell 2950's are still flying along just fine. Since they are NOT in a clean room they get cleaned out once a year. I've been installing Dell R-720 and R-730 (two so far) in place of the failed servers. They can be procured for $300 to $1000 with dual 6+ core processors and 20+ GB of ram depending on budget. I just move the OS drives over to an R-720 and add the foreign config to the PERC 6 and you got Windows back after a driver update. Then add the media drives as the second foreign config and that gets you 95% of the way back to on line status. I actually did the last one remotely with the help of another cinema tech at the other end who was in the area who moved the hard drives over for me.

Those that have large external IBM media storage arrays may find that those are not compatible with the Dell. I have not had an instance to try that yet... But since the model R-730 is also made in 16 and 24 drive versions you could also choose to go that route for more $$$$. Then you can toss the array in the trash along with the dead server.

I hated those IBM severs in the Cinedigm racks and fortunately only had three to deal with for about 5 years. You will find that most IT guys also hate IBM gear. The 8 minute long boot time for W-2008 is ludicrous. The Strong racks which used Dell's were way more responsive on boot up with W-2008.

Mark

Posted by Steve Guttag (Member # 268) on 12-15-2019, 11:15 AM:

Mark's math and estimates have always been a little off...2 out of 40 is 5%, not 2%...still low but he's off by over a factor of 2.

FWIW...all of my original Cinedigm systems are still operational, including three of the dreaded IBM/InSight systems, which I agree are more troublesome than the Dell ones put together by Strong. I attribute that to better hardware and Will York at Strong.

Posted by Mark Gulbrandsen (Member # 72) on 12-15-2019, 02:32 PM:

Well, it's actually two out of 46. I also installed a number of Dell 2900 towers at theaters that were only two or three screens.

Yes, Will York is an amazing IT guy...

Mark

Posted by Marco Giustini (Member # 4544) on 12-15-2019, 04:45 PM:

As Steve says, remove all the extra boards, all the drives. Try loading a linux distribution on USB. If it does not work, then it's MB or CPU or Memory (reseat everything). If it works, add one component at a time until it stops working.