Film-Tech Forum ARCHIVE: Live-playing all the time...viable?

This is topic Live-playing all the time...viable? in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=003035

Posted by Mike Blakesley (Member # 26) on 06-02-2017, 05:53 PM:

Right now we're in a situation where I will have to "live play" our movie from its shipping drive, and I had the thought: Why not just "live play" all the time? It would save wear and tear on our server drives, and would be a faster process than ingesting the content.

Your thoughts?

Posted by Frank Cox (Member # 6258) on 06-02-2017, 06:04 PM:

Unless your server works differently than mine I don't think you could set up a playlist with your trailers and everything and then just let 'er rip. (On further thought, maybe you can; I've never looked at the "feature" list on the contents menu when I've had a distribution drive plugged into the server so I don't know if the movie actually shows up there or not.

Plus I suspect that those distribution drives aren't intended for long term use, not to mention the beating around that they get during frequent shipping, so they might be a lot more failure-prone than a drive that's intended for continuous use and doesn't get banged around all the time.

But what do I know...

Posted by Monte L Fullmer (Member # 2797) on 06-02-2017, 07:23 PM:

I had three houses run for over a month with four shows each on Live Play due to my storage drives were acting up...and these were doing the USB sled option, not directly from the CRU bay since I don't have the 2000AR units, but the SX3000 IMB units.

Plus, I have to leave these units on 24/7 due to VPF and DCI requirements. Can't shut them down every night.

I just built the content from the added drives in the SPL and they ran fine.

One could could easily do live play for it's not hurting the system in any way.

Posted by Frank Angel (Member # 248) on 06-03-2017, 12:21 AM:

i would think playing "live" from a shipping drive would be adding a bit more of a higher risk factor...in film parlance, it would be akin to running film from batted-up shipping reels ...well sort of. I would trust my own server and hard drives that I run maintenance and assessment checks on regularly and can know if there is trouble brewing, while a shipping drive's condition is pretty much a crap shoot until it ingests properly.

These drives that DCPs are reused, yes? I mean, they don't use out-of-the-box new hard drives for each release, do they? I mean that would be nice -- when we do shows with wireless mics we ALWAYS put in a fresh battery EVERY show. Sure, chances are there is plenty of juicy to do 10 more shows, but are you going to be the tech who's is responsible for Mel Torme's mic dying for want of a new 99 cent battery?

Then again, given how much they claim they save on producing film prints and 35mm shipping costs, I would think that using a new hard drive would be a drop in the bucket compared to what it used to cost them for film distribution.

If I know I am getting the DCP on a spanking new hard drive, I would feel a little less apprehensive about playing "live." Just like when you get a print that you can see is on brand new shipping reels, you might feel comfortable playing it off those shipping reels and be reasonably certain is not going to bite you in the arse in the middle of a show. I guess it's just a risk assessment you make.

Might I ask, why would you consider doing it this way (assuming you can (I would assume DCi will all it's Big Brother demanding to make you run your booth the way they want you to, would have fairly strict protocol about how a film is to be ingested and played -- although they don't seem to have much to say about which way the sheet hits the fan in every direction when it comes to aspect ratio uniformity/presentation.

Posted by Brad Miller (Member # 2) on 06-03-2017, 12:32 AM:

quote: Frank Angel
Then again, given how much they claim they save on producing film prints and 35mm shipping costs, I would think that using a new hard drive would be a drop in the bucket compared to what it used to cost them for film distribution.

I think they are too busy spending FAR more money on each release by making a dozen different DCP versions, paying for KDM creation on each one for each screen plus the 80 different trailers each movie apparently now has to have for marketing. [Roll Eyes]

Posted by Frank Angel (Member # 248) on 06-03-2017, 01:49 AM:

Hey, they're the ones that made that bed that they have to sleep in!

Posted by John Thomas (Member # 6509) on 06-03-2017, 04:04 AM:

Even if you're looking at the SMART parameters for each distribution drive (distributors should be doing this prior to shipment) there is no telling what each drive has been through during shipment and otherwise.

Here is a link to a whitepaper from some folks at Google on the topic of disk failure:

https://www.usenix.org/legacy/event/fast07/tech/full_papers/pinheiro/pinheiro.pdf

While there are a few interesting takeaways from the study, I think the most crucial is that in the context of a single drive even SMART parameters will not reliably predict imminent failure. Therein lies the value of the redundancy of a healthy RAID.

Unless your RAID is in bad shape you are taking a risk with live play, in which case the responsible solution is to make your RAID healthy again.

Posted by Dennis Benjamin (Member # 1137) on 06-03-2017, 04:51 AM:

I guess everyone is having RAID issues now that we've hit that 5 year mark?

Posted by Steve Guttag (Member # 268) on 06-03-2017, 06:46 AM:

I've seen an up-tick on RAID issues. One of the things we do for our customers is check drive status periodically and I've noticed quite a bit of increase in reallocated sectors and errors listed in the SMART logs (regardless of server brand). So yes, 5-years of 24/7 seems to be about the useful life. As far as outright stop the show failures, that is still very rare.

Note, those LMS systems too are going to start dropping drives.

We are using this opportunity to offer larger drives since when these systems that are 5-years old had 2-3TB of storage can now have 4-6TB (or more, one sub run customer is at 12TB of storage per server since they have to "harvest" content while it is in release so they'll have it when they play it, including previews). In fact, getting 1TB Enterprise rated drives is getting more difficult. HGST, the best drive company I've come across, only has one 1TB drive and it is to support legacy systems. Western Digital "Gold" series has a viable 1TB drive too but it will throw out an error on SMART parameter 16, depending on your server brand (Doremi has a script to correct for it).

Posted by Leo Enticknap (Member # 534) on 06-03-2017, 09:05 AM:

quote: Mike Blakesley
It would save wear and tear on our server drives...

I believe that most if not all DCP server RAID controllers have them spinning all the time anyways, so probably not. Consumer or semi-professional RAID applications such as NAS boxes and software-based RAIDs (e.g. Windows 10 Storage Spaces) will spin the drives down after a timeout period in which no access is requested, but that's not the case for hardware RAIDs used in professional server applications.

Agreed with others above - the whole point of having a RAID in a DCP server is so that if one drive fails in action, the show can go on. The live play option is simply a "get out of trouble" option for, say, if the drive arrives two minutes before showtime. I can't see any reason to use it if you don't have to.

Posted by Marcel Birgelen (Member # 6801) on 06-03-2017, 09:13 AM:

I think the downsides to live playing are somewhat obvious and also already mentioned.

Obviously, if you program multiple movies in the same auditorium, live playing has the obvious limitation that you're constantly swapping drives around.

The load on a single drive during playout is far higher than on a RAID array, where the load is distributed across multiple drives. So, this increases the stress on the drive and therefore also the risk of failure. Also, most of those drives in distribution aren't spring chickens, they get reused after their engagement. You could look at the SMART values to see the "mileage" on those drives once you get them, but it will still just give you a marginally trustworthy indication of how well the drive will perform during an engagement.

The obvious risk is the drive failing during the engagement. It's already bad to lose a show, but you risk losing much more of your engagement than just a single show. So, I think you should make a backup of the drive before the engagement.

A possible alternative could be to buy a sufficiently big SSD drive and duplicate the contents of your distribution drive on them. Although all SSDs do have quasi-hard limits on the amount of writes they can handle, they support almost infinite read cycles. Also, a single SSD will easily outperform any normal "rotating rust" RAID, so you don't need to worry too much about performance issues. In the unlikely event of this SSD failing, you still have the original drive as a possible fallback.

quote: Steve Guttag
We are using this opportunity to offer larger drives since when these systems that are 5-years old had 2-3TB of storage can now have 4-6TB (or more, one sub run customer is at 12TB of storage per server since they have to "harvest" content while it is in release so they'll have it when they play it, including previews).

I think that once you go past the 1TB per drive mark, RAID5 actually isn't good enough anymore. You really need to go to a RAID level that offers more protection, like dual-parity RAID6. Unfortunately, the extra parity slice will cost you an additional disk.

I've had quite some RAID5 arrays fail during rebuild, especially on old arrays, where a lot of disks were roughly the same age. Also, with such large disks, the rebuild will take forever to complete, only increasing the timespan where your entire RAID is at risk of total failure and has impacted performance.

Luckily, most cinema servers usually can easily re-ingest their content from somewhere else.

quote: Leo Enticknap
I believe that most if not all DCP server RAID controllers have them spinning all the time anyways, so probably not. Consumer or semi-professional RAID applications such as NAS boxes and software-based RAIDs (e.g. Windows 10 Storage Spaces) will spin the drives down after a timeout period in which no access is requested, but that's not the case for hardware RAIDs used in professional server applications.

There's quite some debate on weather spinning up and down is doing disks more damage than just letting them spin.

My personal experience is that disks that spin 24/7 will usually last longer. Still, there's a lot of controversy in the broader industry at large.

There's another thing though. A busy drive, on average, will almost certainly wear out quicker than an idle drive. Besides letting the platters spin, there's also another aspect. A drive that needs to perform a lot of reading and/or writing also needs to move its heads much more often, which will wear out the actuator.

Also, a drive writing a lot, will need to demagnetize and magnetize the platter surfaces a lot more, which will also lead to wear.

Posted by Mike Blakesley (Member # 26) on 06-03-2017, 04:33 PM:

quote: Dennis Benjamin
I guess everyone is having RAID issues now that we've hit that 5 year mark?

We are indeed having a RAID issue but interestingly, the trouble is in the "box" rather than in one of the drives (apparently...they all show as healthy).

Posted by Adam Martin (Member # 641) on 06-03-2017, 05:52 PM:

In my experience, the GDC interface ALWAYS shows all drives as OK. Then you send GDC logs and they say "you have a bad hard drive".

Posted by Marcel Birgelen (Member # 6801) on 06-03-2017, 06:59 PM:

What kind of server have you got? A GDC machine?

The problem could very well be in the RAID controller, but I've also seen issues, were a single faulty drive causes trouble, but those problems aren't correctly being noticed by the controller.

Like Adam already indicated, often, there's something in the logs or in the SMART data of those drives.

The low-tech way to identify the affected drive would be to pull a disk, test the result, if the problem stays the same, put the disk back into the machine, wait until the RAID has been rebuilt, pull the next disk and repeat...

This procedure takes a lot of time and if there's anything on the RAID that's dear for you, you should definitely back it up.

Posted by Scott Norwood (Member # 30) on 06-03-2017, 07:02 PM:

Agreed with the others--it will work as long as the "shipping drive" is in good condition and does not need to repeatedly re-read information (causing pauses or stuttering) or fail (causing show stoppage), but there is really no advantage vs. loading the content onto the server's RAID. The "wear and tear" on hard disks from long, sequential reads (as in a cinema server) is minimal.

The RAID 5 issue is that disk capacities have increased, but the unrecoverable bit error rate has not. If a disk fails, the data will still be protected (and the show will continue), but the RAID may not be able to rebuild itself when the bad drive is replaced. This is because, statistically, the remaining drives will likely encounter an unrecoverable error in the process of rebuilding the array (which requires reading every bit on every disk successfully). The problem relates to the size of the RAID (in total), not the capacity of the individual disks, but a RAID5 over 12TB using normal SATA disks is asking for trouble. With a cinema server, this is not a big problem, since one can just replace the bad disk and wipe and reconstitute the array (and reload the content), rather than having to allow the RAID to rebuild. Yes, RAID6 is a better approach that allows for much larger disks to be used before RAID rebuilding becomes a problem.

Personally, I hope that future cinema servers avoid the RAID5/RAID6 thing entirely and just use larger disks (4TB or 6TB) in a RAID1 (maybe in a 3-way mirror, for extra safety), which is simpler, more reliable, and safer to implement in software (as opposed to requiring a hardware controller).

Posted by Mike Blakesley (Member # 26) on 06-03-2017, 11:05 PM:

quote: Marcel Birgelen
What kind of server have you got? A GDC machine?

The problem could very well be in the RAID controller, but I've also seen issues, were a single faulty drive causes trouble, but those problems aren't correctly being noticed by the controller.

Like Adam already indicated, often, there's something in the logs or in the SMART data of those drives.

We have a GDC SX3000 media block and an external RAID box.

We've already sent the logs to GDC twice and that's where the "drives are healthy" word is coming from. The machine is under warranty so they're sending a new box and a new set of drives, but to be safe I already ordered another set of drives too -- I'd been meaning to do that anyway to have a spare set but just kept forgetting to do it.

From the comments here I think I'll just keep doing our usual procedure of ingesting the content to the RAID.

Posted by Steve Guttag (Member # 268) on 06-04-2017, 07:35 AM:

Mike, are you using the PSD or the Enterprise plus storage?

Posted by Mark Gulbrandsen (Member # 72) on 06-04-2017, 11:22 AM:

PSD....

Posted by Steve Guttag (Member # 268) on 06-04-2017, 02:26 PM:

They seem to be more problematic than the Enterprise storage.

Posted by Mark Gulbrandsen (Member # 72) on 06-04-2017, 02:31 PM:

Out of close to 50 units installed this is the first failure of one.

Posted by Mike Blakesley (Member # 26) on 06-04-2017, 10:41 PM:

quote: Frank Cox
Unless your server works differently than mine I don't think you could set up a playlist with your trailers and everything and then just let 'er rip. (On further thought, maybe you can; I've never looked at the "feature" list on the contents menu when I've had a distribution drive plugged into the server so I don't know if the movie actually shows up there or not.

Frank - you can indeed "mix" the playlist for stuff that's on your server vs stuff on the shipping drive, if necessary. All the content shows up in the content list no matter where it's located. If the content you want to play is on both the mounted shipping drive and the server, the content will play from the server.

Posted by Frank Cox (Member # 6258) on 06-05-2017, 12:40 AM:

Well, whaddaya know. I never knew that before. It makes sense, though. I've never looked at the content list on the SMS screen when I have a distribution drive connected to the server.

Now I'm going to have to look at that the next time I'm ingesting something. [Smile]

Posted by John Roddy (Member # 7348) on 06-05-2017, 12:41 AM:

The big problem here is that digital storage technology has evolved too fast for the digital cinema industry to keep up with it. I can remember looking at the 3x1TB RAID5 storage configuration and thinking it was the bomb-diggity, but not anymore. Mechanical hard drives are slowly losing their relevance as flash storage becomes so much cheaper, and even RAID itself has become obsolete. The FreeBSD community has been working valiantly for years to pull the ZFS code from Oracle's "ridiculously useful stuff that we don't care about and won't share with anyone because screw all of you" portfolio, and that work is paying off. When FreeNAS Corral released a few months ago, it really looked like my fantasized UBERNAS (one storage system to drive an entire site) was finally feasible. Of course, the Corral release didn't end up working out very well, but the potential is still there.

Yeah, the bombshell dropped there was to just not care about the internal storage of the server and offload all of that to a purpose-built NAS solution instead. High-capacity NVMe storage servers are totally a thing now. In my mind, there would be a single server dedicated to content management and playback for one group of players, but still able to instantly take over for another group if that one goes down for any reason. That way, all players get the closest-to-bulletproof-possible storage they need, and the boatload of unreliable hard drives get replaced with just a few NAND modules.

Given the industry's eagerness to move forward with bold new ideas like this, I'd expect to see the so-called UBERNAS platform hitting the market at any point within the next 15 years or so.

Posted by Mike Blakesley (Member # 26) on 06-05-2017, 11:21 AM:

quote: Frank Cox
I'm going to have to look at that the next time I'm ingesting something.

Note: The content on the shipping drive doesn't just show up on the content list -- you need to mount the drive for "live playing" to make it appear.

Posted by Mark Gulbrandsen (Member # 72) on 06-05-2017, 03:55 PM:

Can one "Live PLay" On either of the Dolby servers? I'm curious about that since I never tried. I've had a couple occasions where that would have saved the weekend though.

Mark

Posted by Gordon McLeod (Member # 33) on 06-05-2017, 03:59 PM:

yes the dss 100 and 200 will

Posted by Leo Enticknap (Member # 534) on 06-05-2017, 11:04 PM:

But only from a CRU drive in the internal SATA-connected bay: a DSS200 at least will not live play from a drive connected by USB.

Posted by Steve Guttag (Member # 268) on 06-06-2017, 05:24 AM:

The DSS line has always been able to play directly from the CRU drive. The DSS220 can play from a CRU if connected via ESATA.

Posted by Mark Gulbrandsen (Member # 72) on 06-06-2017, 03:12 PM:

That's good to know. Thanks guys!

Posted by Carsten Kurz (Member # 5396) on 09-13-2017, 07:46 AM:

Barco's Alchemy/ICMP now has a direct play feature in the most recent software. I don't know yet wether it works equally well from Ethernet and USB3.0 sources, have to do some testing.

- Carsten

Posted by Marcel Birgelen (Member # 6801) on 09-13-2017, 05:25 PM:

Playing over Ethernet works fine for 2K 24 fps content as it neatly is capped to 250 Mbps. Playing 4K 60fps content over Ethernet is pretty much a no go though. Fortunately, that kind content is pretty rare as of now.

Still, the Alchemy ICMP could really benefit from a 10GBit upgrade in future revisions. It's not like 10 GBit NICs are as horrendously expensive like a year or 10 ago.

Posted by Jim Cassedy (Member # 4115) on 09-13-2017, 07:30 PM:

quote: Mark Gulbrandsen
Can one "Live PLay" On either of the Dolby servers?

You can, live play on the Dolbee DSS'es, as others have pointed out.
In my world of production, press, preview & QC screenings, I've had
to do it often as drives are often delivered at the last minute or even
hand carried by the director or client to the projection room minutes
prior to the scheduled screening start.

The clients don't always want to wait for ingestion, as they have other
meetings to go to or flights to catch. I explain the 'risks' and then just
pop the drive in & play it "cold".

But I'm not sure you can include the "live drive" in a playlist with trailers,
and cues, etc. I think I was told this once, but since I don't deal with
nonsense like trailers or automation cues, I've never tried it.

Maybe someone can clarify that point for sure. . .

Posted by Mark Gulbrandsen (Member # 72) on 09-13-2017, 09:45 PM:

I should place an update here on Mikes system as we solved this some time ago. It turned out not to be the PDP-3000 after all and either GDC missed the sector issues in the log of the HD's just didn't log them. Instead of messing around with the drives, trying to figure out which one had issues I just had Mike pop a new set of drives in and it has been fine since.

Mark

Posted by Carsten Kurz (Member # 5396) on 09-22-2017, 02:19 PM:

Looks as if Barco did a nice job on this recent ICMP update. Instant play, or 'Play Now' as Barco calls it, works nicely from a standard CRU ingest drive connected through USB3.0 (in this case, a USB3.0 MoveDock). I even tried it successfully through USB2.0. DCP had an average data rate of approx. 29MByte/s, which is not on the lower side of DCP data rates. Actually it was a 4k Scope feature with 7.1 sound (Hitman's bodyguard). The USB2.0 connection certainly was on the edge, but played without issues. During a test ingest of the same feature through both interface speeds, the ICMP ingest monitor indicated about 30Mbytes/s for USB2.0, and about 60MBytes/s through USB3.0.

When I turned off the ingest drive power during live play, it took around 15s for the image to freeze over USB3.0, and around 5s over USB2.0.

There is no visual crashing or such, image and sound just come to a halt. Barco say that if it happens due to a buffer underrun, playback would resume from the frozen position as soon as possible. Didn't happen once during my tests.
There is no significant delay between selecting a CPL for Play Now and the playout start, just a few seconds for the player to load the content and prepare playback.

I tried it with a german 7.1 VF. KDM needs to be ingested before Play Now is started from the ingest manager. You can't build playlists using this feature. Of course you could run your preshow and standard cues from your normal preshow playlist and then start the main feature manually.

I would never reccommend doing this on a regular basis, but when necessary, it seems to work. Wondering who would actually do this on a regular basis if you miss all playlist features.

Aside from using it for live play for a paying audience in an emergency, I noticed it is also quite useful that you can browse through your library or an ingest drive loaded with many CPLs, quickly looking into even encrypted DCPs, as long as the keys are loaded. We tend to get complicated Multi-CPL OV/VF packages here, and being able to watch short passages to identify them properly is a nice to have. Yes, you can cue to any feature position in stop mode during play now, just as with ingested features.

One important thing to note is that once a CPL is loaded in Play Now mode, that specific drive can no longer be seen in ingest manager. You have to eject the CPL from the player before you can see the content again and perform a regular ingest. There is no background ingest taking place during Play Now - after playout has finished, the CPL is still not ingested on the server.

- Carsten

Posted by Marcel Birgelen (Member # 6801) on 09-30-2017, 10:46 AM:

I recently had to use the "Play Now" feature. I didn't use it on an encrypted DCP though. It worked as expected, from an external USB 3.0 disk.

And yes, it's a nice last-resort feature and obviously implemented as such, since you cannot add "Play Now" CPLs to any playlist.

Posted by Tom Bert (Member # 5719) on 10-04-2017, 04:55 AM:

Glad you're enjoying Play Now :-)

And indeed: given the dependency on the external storage (R/W speed) and connection (cable/bandwidth), we can never guarantee the same performance as when playing from the internal storage. Hence: to be considered as a mitigation

Posted by Jim Cassedy (Member # 4115) on 10-04-2017, 11:19 AM:

quote: Carsten Kurz
Wondering who would actually do this on a regular basis
if you miss all playlist features.

Well, I can't really say I do this on a 'regular basis' but in my situation,
working mostly in several professional private screening rooms, I've
often had to resort to live play because material arrives at the last
minute, often hand-carried by a client who is too busy to wait around
for ingesting.

The lack of being able to build a playlist doesn't affect me anyway,
since my playlists, when I do build one, only consist a 10sec black
'slug' before whatever I'm playing and another 2secs of black at the end.
Everything (show start, lights, curtain, masking, picture & sound
formats, etc) are run manually.

(I'm not even sure the automation is connected or working in at least
one of the locations I work at. I've been there 5 years & never used it. [Cool]

)

Posted by Carsten Kurz (Member # 5396) on 12-22-2017, 06:02 AM:

Yesterday I tried 'PlayNow' from a (Gofilex) NAS connected to our ICMP, and it didn't work. ICMP ingest manager showed the content, I also ingested the KDMs. If offered the 'PlayNow' button in ingest manager, but the player gave an error 'could not load content into player'. It was a bit strange that even after ingesting the KDMs for this CPL, ingest manager continued to give the 'KDM needed' red flag for all versions. After I ingested the same CPL, I was able to play the feature from the internal RAID.
I also tried to play another, unencrypted DCP from the same NAS, same result.

Is 'PlayNow' only meant to work from USB drives? I understand it may be more complicated to implement through FTP.

ICMP is running 1.3.1.8 (most current release).

- Carsten

Posted by Tom Bert (Member # 5719) on 12-22-2017, 06:15 AM:

quote: Carsten Kurz
Yesterday I tried 'PlayNow' from a (Gofilex) NAS connected to our ICMP, and it didn't work. ICMP ingest manager showed the content, I also ingested the KDMs. If offered the 'PlayNow' button in ingest manager, but the player gave an error 'could not load content into player'. It was a bit strange that even after ingesting the KDMs for this CPL, ingest manager continued to give the 'KDM needed' red flag for all versions. After I ingested the same CPL, I was able to play the feature from the internal RAID.
I also tried to play another, unencrypted DCP from the same NAS, same result.

Is 'PlayNow' only meant to work from USB drives? I understand it may be more complicated to implement through FTP.

Play Now is intended to work also from network attached storage. However, the NAS needs to be configured as NFS; not as FTP storage. Did you check this?

Posted by Carsten Kurz (Member # 5396) on 12-22-2017, 04:54 PM:

Thanks Tom,

I may not have that option, as the Gofilex NAS is not managed by us. But I'll check with them.

Merry Christmas to everyone!

- Carsten

Posted by Carsten Kurz (Member # 5396) on 01-09-2018, 09:20 AM:

As it turns out, the Gofilex filmstore is not configured to support NFS due to security concerns, so, we're out of luck.

- Carsten