Film-Tech Forum ARCHIVE: DSS 200s losing connection with Cat862s

This is topic DSS 200s losing connection with Cat862s in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=002604

Posted by Nate Anderson (Member # 2711) on 03-01-2016, 10:08 PM:

We have 7 DSS200s with Christie 2200 projectors. We're running Dolby version 4.8.6.13 and we have been having problems with the 200s losing connection to the Cat862s and movies not starting.

The movie with countdown to 00:00 and it was go green like its starting but the counter gets stuck at 00:00 and we need to do a hard reboot to bring the server back up.

They also get stuck on the Downloading projector files screen also requiring a hard reboot.

Any ideas? Thanks in advance.

Posted by Tony Bandiera Jr (Member # 2365) on 03-01-2016, 11:29 PM:

Two things:

One, replace that little blue ethernet jumper cable on the back of the DSS200..they are known to cause issues.

Two: If that doesn't work, on some DSS200s there is an internal link cable between the CAT 682 and the backplane, pull and reseat it. (This will require you to remove the mounting screws for the 682, and lift it up a bit to get to the connectors. Be careful not to subject the 682 to any hard shocks or you might "brick" it.)

Do a hard reboot (pull BOTH power supply cords, wait 15-30 second and reconnect them) afterwards and you should be o.k.

Posted by Leo Enticknap (Member # 534) on 03-01-2016, 11:56 PM:

Seconded on the 1-foot cat5e cable. I had a similar problem that was solved by replacing it.

When rebooting, rather than just doing a German reboot (i.e. pulling the power cords while the thing is booted up and fully running), there is a slightly kinder way to hard reboot a DSS2000:

ctrl+alt+F1
Enter the administration username and password that you'll find in the DSS200 manual (per forum rules, I'm not allowed to post them here)
Type "reboot" and press enter
When you here the fans in the PSUs surge, then pull both cords. At that point, it is guaranteed that you won't be cutting the power at the exact moment a hard drive write operation is taking place (because the BIOS will be initializing), thereby minimizing the chance of causing a drive error.

It is necessary to pull the cords, though, because if you just enter the reboot command and let it do it, you will only reboot the server, not the cat862. You need to cut the power to the cat862 in order for it to reboot as well.

Posted by Steve Guttag (Member # 268) on 03-02-2016, 06:30 AM:

First...ALL DSS200s have both an internal and external link cable. On the few times I've gone inside to resolve the problem, I change the cable out...that is, I bring a 2-foot cable, plug it into the CAT862 and then remove one of the ISA cover plates and run the cable out and into the link connector...thereby saving on 1 cable and two connectors plus what ever soldering/PCB work on the little board. I just don't want to revisit it over a $1.00 cable. I also don't like the way it is zip tied on the CAT862...the bend radius is WAY too small, in my opinion...yes it is a short cable but still.

I have seen CAT862 disconnections on System 4.8.6.13 though not as you describe. The CAT862 disconnections have been random and not necessarily when the timer gets to zero. It hasn't been epidemic in nature but enough for me to take notice. The ONLY trend I've seen in it is that the odds are MUCH higher when driving a series 1 projector. I think I've had 5 instances of the problem and 4 of them were on series 1 projectors.

I don't think it is the CAT862 but something up with the NIC on the motherboard or the software driving the NIC on the motherboard because a soft reboot (doesn't power cycle the CAT862 or really reboot it at all), clears the issue...if it was on the CAT862 side, a soft reboot shouldn't work.

Speaking of soft reboots:

Leo's way will work but requires a password and extra steps. Instead use:

<ctl><alt><F1> to get to the CLI interface

But then use the Bill Gates <CTL><ALT><DEL> and it will do a graceful reboot.

What is also different in your description is that you have problems on log downloads...something I haven't seen.

I would definitely pull logs from any of your servers that have had [any] issue and send to Dolby via Dolby Customer so the problem is on record. There is also a Dolby Log Analyzer to go through them and look for known problems.

If It were me, and on-site, I'd probably also reflash the CAT862 using the Software Update program and tick "reinstall all components" to wipe the CAT862 clean. Note, if you have Dolby 3D or use the CAT862 for audio delay, those settings will be wiped too so save them first.

If you haven't rebooted your servers in a long time, it could merely be that. Once a month is normally pretty good interval. I have also noted on System 4.8 a tendency for a server to come up "Disconnected". I'd say you have about a 1:50 chance.

4.8 has a lot going for it (quick show starts, intermissions, SMPTE package improvements) but it definitely picked up a few quirks that are not welcomed.

I don't recall seeing the CAT862 disconnections at this level before 4.8.6 so you may try rolling back to a previous version. 4.8.7.1 didn't resolve it for my sites and it is too soon to tell if 4.8.8.4 has any improvements. None are documented in the release notes.

Posted by Magnus Eriksson (Member # 6085) on 03-02-2016, 01:42 PM:

quote: Steve Guttag
I have also noted on System 4.8 a tendency for a server to come up "Disconnected".

So have I. Bot not disconnected from the media block or the projector, but disconnected from "itself". In this state I can´t see any content and the raid is "unknown".
Normally it connects after a while, sometimes it doesn´t. It then takes a cold reboot to wake it up.
DSS200:s with large drives are more prone to this behavior in my experience.
Something timing out somewhere?
/M.E.

Posted by Steve Guttag (Member # 268) on 03-02-2016, 03:36 PM:

Yes, the "Disconnected" state is with its own store. I don't mean the state that resolves itself but one that will stay there indefinitely. A soft reboot will bring it back but, depending on your access and where you issue the reboot, the script will be broken and not reboot the server. I've also had a couple servers hit me twice in a row on that one.

Posted by Stephan Shelley (Member # 8587) on 03-02-2016, 06:47 PM:

I have a site that does the same thing. show appears to start but stalls on 0:00:00. Dolby has admitted to a software issue on this and is suppose to be addressed in 4.9.X. Is this a miniplex setup? Is it a custom IP scheme? Is the control on the auditorium network instead of the theatre network? Those things make the problem worse. Those are the factors on the site I am referring to and it happens on a daily basis. They use the miniplex setup but they are on the Christie NOC so it is their IP scheme and control has to be on the auditorium network. They also have the issue of the disconnect lasting forever on bootup sometimes. The only way I have found to recover for the stall on 0:00:00 is a power cycle.

Posted by Steve Guttag (Member # 268) on 03-03-2016, 05:40 AM:

I'd say that 75% of our installations are "mini-plexes." /it is not a Dolby IP scheme on the Auditorium network; it is Dolbyesq on the Theatre Network and that retains the management for those sites. The only problems I have heard of with the show not advancing was with versions past System 4.8.4.10 and that rolling back to 4.8.4.10 will "fix" the problem such that you can re-upgrade to where you are now (current version is 4.8.8.4) without the issue.

Posted by Stephan Shelley (Member # 8587) on 03-03-2016, 01:55 PM:

I tried the rolling back as you mention Steve and then back to the current version with out it solving the problem. When you roll back you have to do a clean install and loose at content and SPLs.

I have another miniplex that I take care of which uses the stock Dolby IP scheme and they have only once had it stall on start of a show. The other location on the Christie NOC on a daily basis.

Posted by Nate Anderson (Member # 2711) on 03-03-2016, 03:46 PM:

I followed your advice, Steve, and am currently reflashing the 862s. I have one left to go. I'll keep my fingers crossed.

Thanks for the info.

Posted by Marco Giustini (Member # 4544) on 03-04-2016, 01:58 AM:

These very issues have been going on forever with dolby servers.
Why on earth dolby cannot fix them?
All versions are more or less affected. And the dreadful 'disconnected' message should be replaced with something more meaningful like "refreshing content, please wait" in order to avoid the customer restarting the projector and the server because "my server does not connect to the projector! "

Posted by Steve Guttag (Member # 268) on 03-04-2016, 07:14 AM:

It is my understanding that you DON'T have to wipe the server when rolling back to 4.8.4.10. Again, I have not be afflicted by this particular problem so I don't have first hand knowledge of it...just grapevine sort of stuff.

Note, the problem should be gone in 4.8.4.10...I'd leave the server there to verify that before rolling back up. I've never unconfiged a server staying within the same version range (e.g. all 4.8s are in the same version range). Honestly, I've rarely unconfiged moving major ranges (e.g. moving from 4.8 to 4.7). However, when moving major ranges I can understand why you might need to...there are features that will disappear when rolling back (in the case of 4.7 from 4.8...Intermission, instant start).

As for "Disconnected," that is a more "new feature" for me that seems to be closely tied to System 4.8...yes, you could get it on other versions but with 4.8 the likelihood jumped better than 100-fold. To the point that IF you reboot your sever EVERY WEEK, the odds are, you'll see it within a year.

With Dolby's servers, there are all of these subsystems (like with a projector...there is the brains of the projector...the "controller" then there is the ICP that effectively its own independent system residing within the projector and maybe an Enigma module also its own world within the projector)...with the DSS line, there is the server, the show store, the mediablock...etc. Disconnected effectively is saying it cant communicate with the show store within the server. I figure it has to be a timing problem on boot up since it is random and a version change affected it so drastically.

The not-advancing problem, I figure is likely related to when they made the transition to instant start that there were unintended consequences in how things process now. What I saw most after that happened was if you did a random-seek...you could get caught on a "Checking licenses" screen...a screen that shouldn't be there after the show starts...that problem was fixed in 4.8.5, as I recall.

As I've said before, System 4.8, like 4.5 and early 4.6(shudder), seems to be a version that has brought annoyances with some welcomed features (I really like instant start). That said, I'll take reliability over features EVERY TIME.

Posted by Leo Enticknap (Member # 534) on 03-04-2016, 12:33 PM:

There appears to be a spate of bugs with this software platform emerging at present, as evidenced by the two quickfire updates that were issued recently: 4.8.8.3 to correct a February 29 bug (risk of unpredictable behavior if stuff is ingested on Feb 29, or stuff timestamped Feb 29 is ingested thereafter), and then the discovery that this fix broke Atmos, and 4.8.8.4 to correct that.

Dolby don't usually publish an update that only has one or two fixes or feature additions, but there have now been three in relatively short order.

Hopefully 4.8.9 will embed the recent new functionality (quick start, intermission and full support for SMPTE subtitles) while achieving a fundamental reliability improvement.

Thankfully, I haven't experienced the "Disconnected" glitch on my DSS200/cat862 system yet (knock on wood), though I do do a "clean install" reflash thru USB of the cat862 immediately before and every time I update the main server's software image, which may have helped.

Posted by Steve Guttag (Member # 268) on 03-04-2016, 07:19 PM:

I haven't heard of a 4.8.9, at this point. If there is a bug fix only version, it would likely just increment the build number. The 4.6.0 went through some rapid fire stuff before finally settling on 4.6.1.4.

Posted by Marco Giustini (Member # 4544) on 03-05-2016, 03:33 AM:

quote: Steve Guttag
Disconnected effectively is saying it cant communicate with the show store within the server. I figure it has to be a timing problem on boot up since it is random and a version change affected it so drastically.

I've seen the forever DISCONNECTED problem - glad to know it's a known issue - but you usually see the DISCONNECTED message for a few seconds - minutes while the server is still indexing content (you can see that happen on the console screen).

DISCONNECTED makes sense for a developer, not for a customer!! For a customer that means the server is not connecting to something and they will tinker with cables and keep rebooting the server. The number of calls I've got over the years on this subject are proving it.

Why not changing that dreadful message with something more user-friendly like "waiting for ShowStore server, please wait" or "indexing content, please wait". Whatever would better than "DISCONNECTED". Some kind of progress bar would help too.

If the 862 does not connect, the ShowManager throws a specific error message. Same with the projector not connecting. It's just the ShowStore server.

Posted by Leo Enticknap (Member # 534) on 03-06-2016, 12:11 PM:

Sorry - I meant 4.9.

Posted by Steve Guttag (Member # 268) on 03-06-2016, 04:03 PM:

Oh...I'm sure that 4.9 will have everything that was in 4.8 (later versions always include the features of previous versions) and yes...I'm sure bunches of bug fixes.

But remember the software developer's credo. They are just one revision away from perfection.