|
|
This topic comprises 4 pages: 1 2 3 4
|
|
Author
|
Topic: Barco alchemy ICMP server or Dolby IMS3000?
|
|
|
|
|
Marcel Birgelen
Film God
Posts: 3357
From: Maastricht, Limburg, Netherlands
Registered: Feb 2012
|
posted 02-11-2019 08:18 AM
Steve,
I know, but that's no guarantee for glory. In the end, both of your switches will be part of the same network.
The following is a real deal scenario, it didn't involve QSys, neither a cinema, but still brought the network down:
Customer had two "core switches", his core infrastructure, running some VMWare stuff was neatly hooked up to both. So, every server, two uplinks, auto-failover correctly configured. Even everything was on two separate power feeds, connected to two seperate UPSes. Stuff that did only have one PSU was connected to STS.
Yet, one day, one of his expensive Dell/Force10 core switches decides to crap out. The so-called TCAM in the switch went bonkers (seemingly a common issue with the model) and the switch starts to put on bogus broadcast messages all over the network.
The entire network went down, because of the junk messages that were sent around, most likely at 10GE speeds or more.
When you connected to either of the core switches, they didn't report anything out of the ordinary, since the management engine was decoupled from the forwarding engine on those switches and the management engine had no clue that there was something wrong with one of those two switches.
This same situation can happen in a redundant switch setup featuring two core switches. Even if you have all "access switches" hooked up to both core switches, there will be no automatic failover, since the network is filled up to its brim with bogus broadcast traffic.
Many of the client devices you plugged into the network, failed instantly with high loads, because of the enormous load of broadcast traffic.
So, where do you start debugging this? Your core switches seem to be fine, so you think some of your access switches is putting bogus info on the network. Maybe spanning tree failed on one of them? So you start to reboot them one by one.
Depending on how big your network is, this takes quite some time. Many of those switches aren't the fastest "booters" on the planet. But after you've rebooted all of them, you notice no improvement. The network is still crapping out.
At last you arrive at the core switches, which again, pretend to operate fine according to their management engine. Then you reboot the culprit and everything starts to recover... By then, you're easily more than an hour in.
This is one of the many failures I've experienced in otherwise "redundant networks". It's not easy to see if the network "lives" or if it's dead. It may be easy to detect if a port is up or not, but that's no guarantee there is also connectivity to the rest of the network.
A network loop, somebody plugging in something into your network now claiming to be the new STP root bridge or some stupid vendor bug can all cause your network to come crashing down, even if you've got two of everything.
Depending on the quality of the switches, the monitoring and your knowledge of the equipment and the situation, a decent network SNAFU can be more easily recognized, but it will still take some time to fix.
| IP: Logged
|
|
Carsten Kurz
Film God
Posts: 4340
From: Cologne, NRW, Germany
Registered: Aug 2009
|
posted 02-11-2019 08:19 AM
quote:
If the primary CORE fails in a manner that the redundant core sees and takes over, you still have dumped sound on all 10-screens for at least 15-seconds until the redundant core is running the show and the amps/ I/O stuff are responding to the redundant core. 15-second is "forever" when something happens and a manager could be faced with an entire complex of readmits!
I would put the non-redundant screens the farthest away from the ticket counter or the managers booth...
'No sound? - Are you sure?'
One could use an external feedback loop or similar brownout detection, but, in general I guess it is better to let QSC 'simply' improve their own droput detection and increase software stability until statistics show that the risc is small enough. Once we're in the ballpark of typical power outage occurrences, I'd say it's stable 'enough'.
Marcel - unreliable equipment is not a specific risc for cinemas only, and mission-critical IT infrastructure is employed by many businesses. That is part of a system layout decision. QSC offers a set of recommended switches for QSYS, and the more QSYS is used (not only in cinema), the more statistics you get on typical failure modes and crappy equipment choices. After all, a cheap switch or other gear can also take out an old-style cinema processor one or the other way.
- Carsten
| IP: Logged
|
|
Steve Guttag
We forgot the crackers Gromit!!!
Posts: 12814
From: Annapolis, MD
Registered: Dec 1999
|
posted 02-11-2019 09:13 AM
Marcel,
Whereas the two QLANs are air-gapped islands and the switches are specifically configured for moving the audio between core and endpoints and whereas the QSYS system will flag any issue getting audio from here to there and whereas the number of "approved" switches are decreasing (QSC is now in the market of supplying an OEM). The odds of having such switch issues is on the order of having such issues on the projection network where if the network fails, so does the security between server/projector.
In short, I'm not too worried about losing two networks at once. I'm not saying it couldn't happen, I'm saying it is on the order of such a microscopic nature that I'm comfortable that won't be the issue.
Now, a CORE not noticing that the primary CORE has failed and therefore has to come on line, whereas I haven't had a CORE fail, I don't have the real-world experience to know how well that feature works. Sure, unplugging a CORE to simulate the problem will have the redundant core on line in 15-seconds or less but that isn't necessarily how it will fail (unless a UPS and transfer switch fail too!).
QSC is definitely confident in their system enough to say a single core to run a complex is safe. They already are running stadiums, airports and such things.
| IP: Logged
|
|
Marcel Birgelen
Film God
Posts: 3357
From: Maastricht, Limburg, Netherlands
Registered: Feb 2012
|
posted 02-11-2019 09:16 AM
quote: Carsten Kurz
Marcel - unreliable equipment is not a specific risc for cinemas only, and mission-critical IT infrastructure is employed by many businesses.
Since almost every business is nowadays an IT centered business, yes, it is.
But still, what are the chances of a network issue taking down your multiplex in the "old fashioned" way? Pretty slim, I'd say.
The worst that could happen is that you need to start the shows in every theater manually. If your automation is on the network, maybe you need to dim your lights manually and set your masking manually.
Do you have a bunch of CP650s dying because of network flooding? Well, disconnect them from the network and select the format and volume manually.
It might not be perfect, but beats refunding 10 rooms full of paying customers.
quote: Steve Guttag
Whereas the two QLANs are air-gapped islands and the switches are specifically configured for moving the audio between core and endpoints and whereas the QSYS system will flag any issue getting audio from here to there and whereas the number of "approved" switches are decreasing (QSC is now in the market of supplying an OEM). The odds of having such switch issues is on the order of having such issues on the projection network where if the network fails, so does the security between server/projector.
So, you keep a strict, let's say "pseudo-galvanic" separation between the "primary" and "secondary" Q-LAN?
What are you doing with clients that have just a single network connection?
quote: Steve Guttag
QSC is definitely confident in their system enough to say a single core to run a complex is safe. They already are running stadiums, airports and such things.
Besides those they put on a Dell server, I'm pretty confident it will be sufficiently stable, I'm more worried about the amount of crap around them. In many cases, those things aren't deployed on dedicated network infrastructure, but merely a separate VLAN on an existing network, if you're lucky.
| IP: Logged
|
|
|
|
|
|
Leo Enticknap
Film God
Posts: 7474
From: Loma Linda, CA
Registered: Jul 2000
|
posted 02-11-2019 11:06 AM
quote: Pete Naples
QSYS may be the most popular in the US, but not here in the UK. They have one dealer so unless the site happens to be a customer of that dealer, they won't use it.
I keep seeing stuff on Linkedin about how QSC has opened a shiny new European headquarters in Surrey (about 20 miles from where I was born, ironically!) and is running Q-Sys courses in Glasgow. From this I get the impression that Q-Sys might not have much market share in the UK now, but that QSC is focusing on it as a growth market.
However, these promotional pieces were mainly about conferencing solutions, corporate a/v and that sort of stuff, rather than cinema specifically.
It isn't in a huge number of cinemas here (as in, Southern California) as yet, but it is starting to gain significant traction.
| IP: Logged
|
|
Steve Guttag
We forgot the crackers Gromit!!!
Posts: 12814
From: Annapolis, MD
Registered: Dec 1999
|
posted 02-11-2019 11:33 AM
quote: Marcel Birgelen
So, you keep a strict, let's say "pseudo-galvanic" separation between the "primary" and "secondary" Q-LAN?
Yes, of course. They are on different networks, there are no cables attaching to either QLAN-A or QLAN-B that are not audio devices on those QLANs.
Once one gets into multi-theatres on one CORE, you move into the 510C and larger so you get a third NIC for automation. Every QSYS device is a dual-NIC device (look at the new CMS-5000 server, dual QLANs).
And Carsten, I'm with you on keeping theatres as much of islands as possible.
One can certainly go all CORE 110c and merely stream between cores for those pieces of audio that need to move about. It does complicate things some in terms of unifying controls.
From a cost standpoint, it is more expensive to do it that way but you buy constraining problems to one screen.
With one core running many/all screens, you save money on buying excessive processing as one CORE can handle a lot of channels moving about. You get the sound quality/processing of a full DSP system but reduce your per-screen cost down to a DCIO (less than a cinema processor) and some form of user interface, which can be a WEB-GUI (starting in the next release...current release is pre-beta web-uci) or a fancy touchpanel. Even without the WEB-UCI QSC has a free PC program called UCI-Viewer so the manger can open up and control any theatre from their office computer.
| IP: Logged
|
|
|
|
Marcel Birgelen
Film God
Posts: 3357
From: Maastricht, Limburg, Netherlands
Registered: Feb 2012
|
posted 02-11-2019 05:24 PM
quote: Steve Guttag
Yes, of course. They are on different networks, there are no cables attaching to either QLAN-A or QLAN-B that are not audio devices on those QLANs.
If you can maintain that kind of isolation, and the failover checking of QSC is actually any good (I haven't needed it yet), then I guess the thing should be considered good enough for anything non-life-threatening. ![[Wink]](wink.gif)
quote: Steve Guttag
With one core running many/all screens, you save money on buying excessive processing as one CORE can handle a lot of channels moving about. You get the sound quality/processing of a full DSP system but reduce your per-screen cost down to a DCIO (less than a cinema processor) and some form of user interface, which can be a WEB-GUI (starting in the next release...current release is pre-beta web-uci) or a fancy touchpanel. Even without the WEB-UCI QSC has a free PC program called UCI-Viewer so the manger can open up and control any theatre from their office computer.
Depending on who will "win" the game, we probably see less and less equipment in the booth, up to the point that there is just a projector left. (If we're not switching to LED screens by then...)
Servers have already largely been replaced by IMS systems and Dolby is now eliminating the sound processor with their IMS3000. Since Barco invested a lot into their own sound processing, I expect them to come up with an IMS that also integrates the sound processor in the future. Maybe we see finally some of the IOSONO technology they acquired about 3 years ago pop up.
While for now, you still need amplifiers, in the future I seriously expect active speakers with either an Atmos Connect and/or Q-Sys interface to start to take over...
| IP: Logged
|
|
|
|
|
|
Carsten Kurz
Film God
Posts: 4340
From: Cologne, NRW, Germany
Registered: Aug 2009
|
posted 02-12-2019 08:19 AM
Oh, I do believe that, no doubt. However, as we just learned here:
http://www.film-tech.com/ubb/f16/t003583.html
Paradigm changes can have very different effects, and centralizing equipment can have all sorts of effects, too.
Also, once the processor is in place and operating, what is the benefit for daily operations?
The more you abstract from a 'knob' or 'switch', the more dependent you become from external forces. Staff development of course is a key element. I know the manager of a large multiplex personally and we have a lot of technical exchange between us. He knows how to operate and bypass everything within his domain. But I feel that is not the norm.
- Carsten
| IP: Logged
|
|
|
|
All times are Central (GMT -6:00)
|
This topic comprises 4 pages: 1 2 3 4
|
Powered by Infopop Corporation
UBB.classicTM
6.3.1.2
The Film-Tech Forums are designed for various members related to the cinema industry to express their opinions, viewpoints and testimonials on various products, services and events based upon speculation, personal knowledge and factual information through use, therefore all views represented here allow no liability upon the publishers of this web site and the owners of said views assume no liability for any ill will resulting from these postings. The posts made here are for educational as well as entertainment purposes and as such anyone viewing this portion of the website must accept these views as statements of the author of that opinion
and agrees to release the authors from any and all liability.
|
Home
![[Smile]](smile.gif)
Printer-friendly view of this topic