Film-Tech Forum ARCHIVE: update ShowVault to 2.8.20 failed

This is topic update ShowVault to 2.8.20 failed - IMB not recognized in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=003426

Posted by Carsten Kurz (Member # 5396) on 09-05-2018, 02:25 PM:

I just updated a ShowVault running 2.6.4 to 2.8.10 - and after a reboot, the secure clock/IMB is no longer recognized. After the first regular reboot, the diagnostic messenger came up with a warning that the secure clock hasn't been replaced in 1344 days (it had been replaced 2 years ago). All was running smoothly before.

Any idea? Multiple shutdowns/power cycles didn't solve it.

- Carsten

Posted by Leo Enticknap (Member # 534) on 09-05-2018, 03:23 PM:

My guess would be one of three possibilities.

1 - You accidentally installed 2.8.20 for a DCP2000 or DCP2K4 (server with a Dolphin media block rather than an IMB), rather than 2.8.20 for a ShowVault, and it's trying to communicate with a Dolphin board and not finding it.

2 - 2.8.20 will only work if the firmware and security manager (both are flashed into the IMB) are updated to a newer version than the one they're on. I'd be surprised if this is the problem, because Dolby's instructions say to always update a ShowVault/IMB in the order server > security manager > firmware. Logically, therefore, a new version of the server software should not break communication with an IMB running an older SM or firmware, because it needs to see it in order to update it.

3 - The flash drive that holds the operating system image got corrupted during the upgrade. If that's happened, you may be able to rescue it by creating an emergency boot drive on a USB flash stick, from a working ShowVault, and then using that in the problem machine to get you going (but the permanent fix, I'm afraid, is to buy a replacement boot flash card from Dolby). I have an instruction sheet on how to do this, but this site won't let you upload PDFs directly. Please email me if you'd like it.

Also, apologies if this is so obvious as to insult your intelligence, but I take it that you have tried pulling and reseating the PCI-e cable, and checking that the IMB <> projector marriage is still OK?

Posted by Carsten Kurz (Member # 5396) on 09-05-2018, 04:27 PM:

The 2.8.20 packages for DCP2000/Dolphin 1.0, Dolphin 1.2 and ShowVault are bit identical. The download links also point to the same file:

http://doremi-techno-legacy-portal.dolbycustomer.com/images/Portal/Cinema_Players/DCP2000-DCP2K4/pkg/SW_2_ 6_4/dcp2000_install-2.8.20-0.pkg

???

I may try the emergency usb stick, have to dig one up somewhere. That PDF is probably on doremitechno, but I'd still like to see your version. You may contact me through pm or mail.

The instructions for creating an emergency USB drive on doremitechno also offer links for downloadable emergency images, but unfortunately only for versions 2.2.5 and 2.4.4.

- Carsten

Regards - Carsten

Posted by Juan Jose Vazquez (Member # 10496) on 09-05-2018, 10:39 PM:

ello everyone.

I have updated to version 2.8.20 in Doremi DCP2000 and ShowVault without problems and using the same file that doremi delivers in its FTP site, I have already installed the hotfix that Dolby recommends installing and the same without problems.

Regards

Juan José

Posted by Ioannis Syrogiannis (Member # 3235) on 09-06-2018, 01:56 AM:

Carsten, you can "create" an emergency USB stick by copying the SSD drive of another ShowVault, bit by bit (dd command via terminal as root - be extremely careful on what you write there, if you try to copy the wrong way around, you'll wipe the drive). If you do it while the server is on, the most possible, you will see an error that the machine did not close properly when you first use the USB drive for booting, since no power down commands were logged in your copy.
Other than that, you'll have to retrieve the certificates of your server, either online or by a diagnostics package you previously created and place them properly.
The actual placement procedure of the USB stick is described by doremi/dolby, so no reason for me to go by memory right now.
CAUTION: The motherboard of ShowVault, just like DCP2000 and the 2K4/SV4 series does not support USB3, use a USB2 stick, if you want the ampers to be proper. I tried once with a USB3 thumb drive and I wasn't able to get it to boot. Changing to USB2 did the trick. Also, you might find that SV3 and SV4 SSDs have different capacities. I am not certain if that has to do with the series (SV3/SV4) or the date of manufacturing. My estimation is that either 2GB or 4GB, the actual space used on both is the same, but it might need different USB sticks (they have to be equal or bigger capacity...).

If you don't have an SV3/4 handy, in order to create the emergency USB, PM me your e-mail and we'll try to retrieve a back-up for you to use, but -mind you- it'll be a v2.6.4 one.

Posted by Carsten Kurz (Member # 5396) on 09-06-2018, 04:00 AM:

On Site, we only have a DPC-2K4 (Dolphin). I don't know if an emergency USB stick created on that one woul also boot a ShowVault with the IMB working? I can see that our ShowVault says 'Dolphin card not found' during boot, so it may check for both Dolphin cards and IMBs? But I should be able to source an emergency USB stick from a colleagues ShowVault somewhere.

- Carsten

Posted by Danial Simmonds (Member # 4638) on 09-06-2018, 07:11 AM:

I have experienced this before, before updating make sure the server is on and IMB is connected.

We had few instances updating while the server was off and it didn’t recognise the IMB.

You can make an emergency boot usb, download the iso from legacy portal. Flash it into the drive. Plug it into the server, connect IMB etc and power up. Update the software gradually.

Posted by Carsten Kurz (Member # 5396) on 09-06-2018, 08:23 AM:

The mergency USB images I see on the legacy portal only seem to cover 2.2.5 and 2.4.4. We do have a 2K4 running 2.6.4 though.
Server and projector, incl. IMB were powered on during the upgrade.

NEC DCCS2 is showing the IMB active with Firmware 5.0.31-0
But the Doremi can not connect. We already reseated IMB, cable and PCIe card.

Service mentions a DCP2000 DDR memory upgrade package - never seen than before...

- Carsten

Posted by Dave Macaulay (Member # 813) on 09-06-2018, 09:19 AM:

You see that clock warning when it has failed to connect to the IMB, since it doesn't know the secure clock time it assumes it's 1970 or something.
Looks like a communication error somewhere. Try deleting the IMB from device manager and rebooting, it should look for it and maybe reconnect? That's what happens on a new installation anyway.
Also I would reseat the projector boards including the IMB, ICP, and CCB.

Posted by Carsten Kurz (Member # 5396) on 09-06-2018, 10:01 AM:

So... first, I created an emergency boot stick on a 2K4 (Dolphin). Creation works, the ShowVault starts with it, but then somewhere goes into a PCI/dolphin detection loop.
So, basically it works, but I can't get it to start the GUI to change the network config.

...interesting, after a while, it exits that loop and starts the GUI, now trying to reconfigure the proper IP addresses...

Will try to remove the IMB from device manager...

- Carsten

Posted by Ioannis Syrogiannis (Member # 3235) on 09-06-2018, 11:12 AM:

That's fine. It might save you some time on configuring.

Have in mind that, if you want to have the original innodisk memory drive checked outside the server, you can remove it from the USB-hub and place it in a USB 2 motherboard header. After all, that is what those Disk on Modules are made for...

Edit: That was a response for the previous post...

Posted by Carsten Kurz (Member # 5396) on 09-06-2018, 11:20 AM:

Hmm,. wow, that was weird - I was about to give up - and the final reboot actually saved us - suddenly, the secure clock warning was gone, KDM were there, diagnostic manager showed the media block/IMB present. I was able to play unencrypted and encrypted content.

Now, the system is running from the emergency boot stick created on our 2K4 running 2.6.4-0 - all I did was adding the DCP2000_DDR memory upgrade, and some IP reconfig. And many reboots... the first one added Certainty automatically. When it still didn't work, I removed all devices (projectors, Certainty, etc.), rebooted, added the NEC series 2 projector. After another reboot, it did NOT automatically add the IMB/Certainty. I was frustrated, power down both server and projector, restartet one more time - and now it works. Probably the projector/IMB needed another power kick as well...

We are now waiting for a new innodisk to arrive tomorrow. Maybe the old one was corrupt and didn't survive the update to 2.8.20...?

Glad, we were just about to start todays showings, so we didn't lose a single one...

- Carsten

Posted by Pete Naples (Member # 761) on 09-06-2018, 02:28 PM:

Carsten,

I can give you a bootflash ISO for nearly up to date OS if you need it. I got it from Dolby support, they said they were going to update the link on the portal, I guess they didn't.

Or just run the update on your eboot.

I had to do this recently when a bootflash died.

Posted by Carsten Kurz (Member # 5396) on 09-06-2018, 03:10 PM:

Pete, that would be great. I'm glad I'm up and running, but...

- Carsten

Posted by Pete Naples (Member # 761) on 09-06-2018, 03:39 PM:

OK I'm trying to find a way to get it to you, it's 2GB

Posted by Carsten Kurz (Member # 5396) on 09-06-2018, 04:52 PM:

Maybe it fits WeTransfer.

And by the way - thanx to everyone...

- Carsten

Posted by Marcel Birgelen (Member # 6801) on 09-07-2018, 08:17 AM:

Looks like you had a rough night. Do you already have a clue what triggered the SNAFU?

I somehow doubt it's a corrupt innodisk. It would somehow be pretty weird that a corrupt disk would allow you to boot the entire Linux distribution, including all modules and services, but not the necessary drivers and infrastructure for the IMB.

Posted by Carsten Kurz (Member # 5396) on 09-08-2018, 08:06 AM:

Dolby assumed that one of the many reboots/powercycles finally solved it, and that the IMB might be in a dicey condition. However, I created an emergency USB from the 2K4 running 2.6.4, and after adjusting it to the IPs, etc., the IMB is working flawlessly from that USB stick with 2.6.4. When I switch to the ill-fated 2.8.20 innodisc, the problem persists. When I switch back to the emergency USB, all is well. Tells me the software installation on the 2.8.20 innodisc is the reason.

I tried the same measures, removing all devices, full powercycle, etc., but that 2.8.20 simply won't work with the IMB. Maybe, although it shouldn't matter, that IMB needs to be running a later version of SM and/or firmware? I think it is currently on 5.0.31 and 30.07y (I need to dive into the detailed reports to see the correct numbers). After the experience with the 2.8.20, I am hesitating to update the SM and firmware, as that is more critical. Also, Dolby always recommends updating software first, then SM, then firmware. But as the 2.8.20 can't see the SM, I can't update anyway, I would need to have a working 2.8.20 installation first.

I haven't had a problem updating 2.6.4 to 2.8.20 to date. Maybe the specific SM/firmware was the reason, or the fact that the IMB/projector was powered up during the software update.

- Carsten

Posted by Ioannis Syrogiannis (Member # 3235) on 09-08-2018, 08:42 AM:

May be, now that you have the USB stick up and running, configured as it should, you could transfer the image from the USB thumb drive to the innodisk MoD. If the problem is software, and not hardware, the system will run properly.
I am wondering, though if there is any kind of dependencies on system software upgrading.
If you had to update from, say, 2.4.4, you'd have to install 2.6.4 first. Normally, from 2.6.x you can go straight to any 2.8.x, could it be that that rule was somehow broken?

Anyway, if you want to experiment, you can copy the USB thumb drive to the innodisk MoD and give the upgrade another go, with a newly downloaded .pkg...

Posted by Carsten Kurz (Member # 5396) on 09-09-2018, 05:13 PM:

While the creation of the emergency backup stick uses a dd on the block device level - the resulting copy seems to be no full copy of the innodisk operating system. I learned this the hard way - while the original 2.6.4 innodisk had the vnc package installed, it didn't work when booted from the USB stick. I had to reinstall the xvnc package after the copy to get it working.

Also, when I tried to upgrade to 2.8.20 from another 2.6.4 emergency boot stick - the update failed with the message, that the upgrade module was not available.

I was able though to reverse copy the running 2.6.4 backup stick to the unusable 2.8.20 internal innodisk by simply switchting source and target device in the command. This results in a running 2.6.4 system, but, it appears that some components are again missing. Can someone shed some light onto this? How do I get a 'full' OS installation back onto the innodisk? I can hardly believe that with dd it would at all be possible to miss components that reside on the source device?

- Carsten

Posted by Ioannis Syrogiannis (Member # 3235) on 09-12-2018, 10:11 AM:

I find it rather peculiar to randomly miss files or features.
I for one would troubleshoot the dd copy procedure.
One thing I would try is to create the same copy with the server NOT booted from the media that have to be copied.

I have used the dd if/of command several times for backing up my PC or laptop operating system, be it Linux, Windows or MacOS and, even though the procedure resulted in a HUGE file every time, I had no problem whatsoever when I had to restore such an image.
(I am not suggesting that particular way to back up, there are fine utilities out there that can do the trick, and MacOS has its own for free.)
The truth, though, is that in all such cases, I never had to clone the OS system while it was active. I usually use a live linux CD or USB to boot from and follow procedure.
When I had the chance to do so with a DCP2000/2K4 or a SV3/4 though, I had to follow procedure while the server was on and running. I hadn't the issue you describe in the occasion I used the back-up, but that might be just luck.
That factor, and, in addition to that, a possible physical corruption, may create issues.

I *guess* the actual copying procedure environment is to be considered here.
I 'd like, for the sake of showing the importance of it, to revisit the need for a USB 2 thumb drive for the whole task, so it can be sufficiently powered.

doremi (now dolby) was describing the procedure just the same, on its portal (at least some time ago), followed while the server was on.
Other than that, there is the option conv=sync,noerror that can be added after the command, for skipping bad blocks, but I wouldn't suggest it, just to be more loyal to the original doremi plan, not being in any case able to claim better understanding of the linux caveats than the doremi technicians.

Posted by Carsten Kurz (Member # 5396) on 09-14-2018, 05:28 AM:

quote: Ioannis Syrogiannis
I find it rather peculiar to randomly miss files or features.

Me too. Even if the dd is done from an actively running system, there should only be some small inconsistencies resulting in the copy - like log files, etc., but nothing serious.
The Doremis are very well documented, but the structure of the file system, mount points, possible RAM discs, etc., I haven't seen anything about this.

There is, of course, the possibility of imaging the innodisk separately, and, I may follow that route. I also guess it should be possible to run something like a GPARTED boot USB stick on a Doremi and do the imaging from there. When I copied the USB emergency boot stick back to the non-working 2.8.20 innodisk, I used a short USB extension to connect the innodisk to the front USB port instead of the internal USB6, where the boot stick needs to be. That way, I do not even have to take out the innodisk from the hub board.

- Carsten