CRU rack mount USB3 readers and SSDs

[James Gardiner]

I’ve spoken with the CEO of DCDC over the years about this issue and repeatedly advised him that satellite delivery would, sooner rather than later, become uncompetitive compared to typical internet connections worldwide. My impression was that he resisted this transition because it dismantled DCDC’s strongest barrier to entry; once digital delivery is commoditised, the market inevitably shifts toward price competition, undermining the company’s core business model.

I suggested that the company either diversify or explicitly cater to the specialised needs of its key clients. This isn’t a difficult path, especially given that several of those clients are major studio stakeholders. Studios, with their unique technical and security requirements, naturally benefit from working with a large, established market leader like DCDC in the U.S. and similar players globally. Even so, it remains a marginal business — akin to trying to keep fax paper relevant in the age of email.

In contrast, independent filmmakers are far more likely to embrace the disruptive shift toward modern, flexible content-delivery models.

[Steve Guttag]

It would be nice if DCDC included KDMs in their delivery.

[Ryan Gallagher]

veering a bit off topic. But is the entire deluxe catalog available to be delivered via DCDC/CineSend? I'm just curious if it is a relevant solution to a classics house that spends 95% of it's time in the archives.

[Leo Enticknap]

My understanding is that when a theater books a title that Deluxe handles the distribution of, it is then pushed from Deluxe into your Kencast satellite rig or e-delivery download box (whichever your site has), and the theater manager then pulls it from there across the LAN into the TMS (or straight into the SMS, if it's a single screen). After the booking is over, it is automatically deleted from the Deluxe box. Presumably this happens with rep titles that Deluxe handles as well as first run.

When I've installed Deluxe boxes, I've noticed that a ton of trailers are also sitting on them at any given time. How it is decided which ones are pushed to any given theater I'm not sure (as in, if that is studio driven or on request from theaters).

[Ryan Gallagher]

When I've installed Deluxe boxes, I've noticed that a ton of trailers are also sitting on them at any given time. How it is decided which ones are pushed to any given theater I'm not sure (as in, if that is studio driven or on request from theaters).

I wonder if requesting zero trailers is a thing. lol. We still get TrailMix CRUs and just peel the label and send them right back. My impression is some effort has been put in trying to communicate a stop to them, but they still come.

[Leo Enticknap]

...and just peel the label and send them right back.

You must be the Marie Kondo of the movie theater industry! This is quite a common sight when I enter a booth for a service call...

image.png

[Ryan Gallagher]

You must be the Marie Kondo of the movie theater industry! This is quite a common sight when I enter a booth for a service call...

Jesus what a mess. Where did the actual CRUs go? I had to look up the Marie reference, good one!

[James Gardiner]

@steve, The KDM email system works extremely well and is effectively free for delivering KDMs to cinemas. I can’t imagine any content delivery network stepping in and providing the same service, with all the required technical overhead, at the same price point (i.e., free). So it’s unrealistic to expect such a shift to ever happen.

This is also why I believe independents will eventually move toward a similar “do-it-yourself” model for content delivery—essentially a server with FTP access. It’s no different to paying for commercial email hosting. Functionally, there’s very little difference.

You also don’t typically see piles of hard drives sitting around in my region, because the process has become highly commercialised. Missing CRUs are actively tracked down—it’s a major cost to any company managing this type of logistics. And returning drives is free; you just have to organise the pickup. The delivery provider invoices the distributor, the distributor passes those costs onto the producer, and in the end those costs are ultimately funded through ticket sales and the film rental you send back.

Cinemas often don’t worry about it, because the expense doesn’t appear to come directly out of their pocket. But it does come out of someone’s pocket, and eventually that cost structure feeds back into the overall viability of the industry.

[Leo Enticknap]

As noted above, I frequently walk into booths and find a significant backlog of CRU drives that the theater simply didn't bother to return. I've noticed it more in the last 2-3 years, which suggests to me that as theaters move to satellite or online delivery, Deluxe has not been as proactive about chasing drives that are shipped into the field and don't come back again. Rather than adding costs, maybe they now have a lot more drives than they need (because the proportion of customers needing physical media versus online delivery is diminishing all the time), and their thinking is that if the theater has to deal with the costs of final disposal of these things, rather them than Deluxe? The cash value of a heavily used, heavily shipped 3.5" spinning rust HDD must be near zero by now.

Edit, to answer Ryan's question: most of those boxes in that photo had drives in them. The boxes on the very top contained the drives that were recently used. I found another, large (as in, 50 to 100 units) pile of CRU cartridges, not in their shipping boxes, next to the TMS.

[Steve Guttag]

James, I still get quite a few KDM issues (perhaps due to the number of screens I support). in fact, I call it "KDM-Friday." This is when we find that the Thursday manager didn't verify they had the correct KDMs since everything looked good on Thursday (with keys expiring at 3-4am on Friday)...so the Friday person comes in to dead keys.

Some servers, notably the DSS200 with CAT862s, they don't automatically purge their keys so well and if one loads a bunch of unnecessary keys (for formats they aren't using), the frequency between manual purges steps up a lot. You know it is in this condition when it appears to not have many keys but it still can't take on any more.

I'd love to get a "how-to" on your KDM client when implementing in a Windows (where just about all of my TMS systems reside) based TMS. I suspect that it would greatly help with getting KDMs. But alas, if it doesn't have a windows installer, I'm going to have a hard time implementing it (or if there were clear instructions/video on how to get it installed).

[James Gardiner]

I'd love to get a "how-to" on your KDM client when implementing in a Windows (where just about all of my TMS systems reside) based TMS. I suspect that it would greatly help with getting KDMs. But alas, if it doesn't have a windows installer, I'm going to have a hard time implementing it (or if there were clear instructions/video on how to get it installed).

My tool is targeting Linux because Linux is the most reliable and less maintenance required typically. There is a reason all DCI-Players are Linux based.
I would suggest buying a small Dell, like a 7090 mini desktop. sit in the corner running Ubuntu headless linux server (No GUI) and install my tool if it suits. I don't expect it's for everyone. Tho, some smaller cinemas like it as it brings AutoKDM from Email. (Yes, it basically loads EVERY KDM targeting for a player onto that player). All players typically deal with that.
AutoKDM also produces a daily report that looks at the schedule on the player itself and warns you if a KDM is not present for a specific session so you know to chase it up a day or two before it's a dark session type problem. I suppose that's the main reason why I find KDMs so easy now. I have rarely had an issue of panic, trying to get a KDM for a session about to start. I have plenty or warning, and fire of emails for distributors to chase up a day before. Typically get, a sorry, be there in a moment. No stress.

DSS200 sound like a nightmare. There are NONE in my region at all. I hear there was a handful, but they have been decommissioned 7 years+ ago. I never integrated against them and never will. You should really have moved on from DSS200 from now, but I understand some don't have the choice.

The only time I have issue with KDMs is due to user error, typically on the distributor's part when they may get the dates wrong,, forget completely to place the booking of the KDMs, or select the wrong CPL in my region.
Lately I have seen VF problems more. Cinemas that play VF files are getting VF that are based on OV files from different regions. Audio or video essence mismatches. The players do not make it obvious why VF-CPLs fails to play. They ingest the OV, and 71 VF but still does not want to play.

Also, TMS's typically deal with TMS issues quite well. AAM has its own Auto Email ingest, while Comscore-ACE-TMS, the locations I know use my tool or similar. Its pretty set and forget. Problem is sites lose ability to understand what to do when things go wrong, and typically call me to investigate the exact cause. But my AutoKDM was designed to make it easy to discover the problem as the way it exposes the data.. I designed it that way. i.e search for a KDM targeting, a specific film, a specific player at a specific location. Not found, tell them to send again, typically a distributor end user error. If found exposes dates clearly etc. takes seconds to see all that data. Exposes KDM letter assigned to any KDM, and KDM source XML directly if you're so inclined. But ultimately it's fast to figure out the problem so you can chase up the person who has to fix it.

For example, I use a single instance of the tool to monitor Emails for 6 sites (Of a mini major in Australia), cloud based. It then, with a network overlay, can ftp KDMs targeting each projector to the specific site TMS that looks after it. The TMS will mark a session as RED and give the KDM as the cause days in advance. Typically, a site should look at the TMS status on Wed night and see ALL session for the following week are green, all assets and KDM and projector programmed for the next week.

Runs extremely well. ( I want to do some interface optimizations one day as it loads a huge amount of KDMs into the interface, making it a bit slow when looking them up. Fine for a single site. I page load it in a future release.)

[Steve Guttag]

James,

I'm not debating the pros/cons of Linux versus other OSes. I'm merely stating that my clients run on Windows and all of the TMSes I support are running some form of Windows (typically Windows Server on Dell based servers). Most of my larger systems are using Comscore/ACE so your tool definitely has an appeal though, again, they are all running some version of Windows Server. So, it is your suggesting to get a mini-PC with Ubuntu running on it and run your auto KDM on that? Or could a VM be created on the Windows server to run it that way? What do your clients with ACE do?

As to the DSS200 server. They've been reliable. There is no way my clients are going to throw away (particularly in these times) anything that works. I'd say that 75% of the DSS200s I installed (my predominate server during the main conversion to Cinema) are still on the job...but they are dropping off a little at a time. I just took a couple more off line a few weeks ago along with a pair of GDC SX2001. I inherited some SV/IMBs that I'd like to see go. Then again, I have no belief that the current crop of servers will yield as long a service life. They are also fussy about turning off; but I automate that. At least with GDC, we can sell them with a 10-year warranty to ensure that they go that long. Dolby tops out at 5-years on the IMS and Barco sort of ties the server to the projector but you can get up to 10-years on the ICMP series too.

In any event, I'd really like to explore getting your AutoKDM thing working on some of my sites, starting with the ACE ones.

[James Gardiner]

@steve, Yes, a ebay $500 USD Dell OptiPlex 7090 SFF i7-10700 1TB SSD 16GB RAM is perfect. Drop in a 8GB or bigger hard drive if you want it to cache DCPs. (As a LMS)
You could also run it in a virtual machine, or a machine that can run docker containers. (As you run it under docker under Linux or Linux VM).
The clients I typically server have ACE running bare metal on a Dell server under windows, as it's historically like that due to VPF. I know you can get it under Linux now. (Q: Which runs better, which is easier to support). it's a purpose created machine so, they don’t run anything else, i.e. VMs on it. To tell the truth, I am not sure how easy that would be? But it's definitely a possible path. But memory in the server would be a consideration.
In my experience with Comscore support, they don't recommend running ACE under a VM. Not sure why, recovery via snapshots would be far easier. I find ACE clunky and partially overengineered, and as such overly complex for most cinema owners.. But when you get your head around it. Pretty good.

I set up some local AutoKDM instances on small Optilex type machines but ended up also running a cloud instance that also takes care of all sites. "Redundancy". It runs on my infrastructure I have for my own cloud based autonomous cinemas I operate. Have some Dell servers in my office basement, with optic fibre to Equinix data centre. Cheapest path if you plan to host a lot of servers and shitloads of storage. Hosting at the data centre cost a fortune. Cloud computing is NOT CHEAP. If you can manage your infrastructure yourself, far, FAR cheaper to do that.

But otherwise, I would recommend you purchase an inexpensive VPS, and run it on that. If all its doing is AutoKDM, you don;t need much disk or processing power. You just need to install a network overlay so you can ftp directly into the ACE system. There is a plethora of tools to do that. Use the one you're most comfortable with. (Network knowhow needs to be medium to high)
If you want, I could walk you through how I would do it. You would need a VPS that has docker capabilities (Not just any VPS service, make sure you can run docker in it) latest Ubuntu LTS version. Go through the cinema-catcher-app manual for install. Set up a network overlay (This is likely the most difficult part as it concerns security practices. and discussion with the cinema.)

In terms of network overlays, I like slack-nebula (Self hosted) or netbird (An open source self-hosted Tailscale), or actually Tailscale is likely the easiest path. I like to self-host all core infrastructure, as to control my own destiny.
But hosting those services is a larger jump into networking knowhow. (Self hosted means you will need to set up another VPS to host those services for your own implementation. Major jump as you have to know how to manage those services too) So Tailscale is likely the easiest path at this proof of concept stage.

Try to get as far as you can then call me to arrange a TeamViewer type (I never use TeamViewer expensive system now) i.e rustdesk as its free. While on WhatsApp voice to talk.

[Caleb Williams]

veering a bit off topic. But is the entire deluxe catalog available to be delivered via DCDC/CineSend? I'm just curious if it is a relevant solution to a classics house that spends 95% of it's time in the archives.

Small note on this point: I was able to ask Deluxe about this when they commissioned our broadband for DCDC, and they confirmed that broadband delivery will allow them to transmit archive content that previously was shipped via hard drive. At the time of our commissioning, I was also told this would enable direct KDM delivery as well.

Satellite delivery uses a KenCast app called Fazzt. It's extensibility is rather limited for small files, and DCDC has a separate front-end that has publishing functionality. At any given moment your DCDC server may have more content on it than you can see to ingest. The "one-sender many-receivers" model means that to send KDM's would involve a lot more management overhead, and the Fazzt system was not really designed with those tiny files in mind.

My only concern with migrating to terrestrial broadband, is when they have live-streamed events. Our location recently had an extended internet outage with the same provider that DCDC uses. Had the only option been to receive the UFC fights on that connection, we would've had to issue a LOT of refunds. Because the broadcast was over satellite, we had no problems.

[Ryan Gallagher]

Small note on this point: I was able to ask Deluxe about this when they commissioned our broadband for DCDC, and they confirmed that broadband delivery will allow them to transmit archive content that previously was shipped via hard drive. At the time of our commissioning, I was also told this would enable direct KDM delivery as well.

Satellite delivery uses a KenCast app called Fazzt. It's extensibility is rather limited for small files, and DCDC has a separate front-end that has publishing functionality. At any given moment your DCDC server may have more content on it than you can see to ingest. The "one-sender many-receivers" model means that to send KDM's would involve a lot more management overhead, and the Fazzt system was not really designed with those tiny files in mind.

My only concern with migrating to terrestrial broadband, is when they have live-streamed events. Our location recently had an extended internet outage with the same provider that DCDC uses. Had the only option been to receive the UFC fights on that connection, we would've had to issue a LOT of refunds. Because the broadcast was over satellite, we had no problems.

Yeah we would have that livestream concern too (although outbound livestream of stage events), unless we add a dedicated provider. It would be nice if maybe broadband DCDC had a "pause/resume" delivery mechanism... other than say unplugging it's network jack? Or perhaps limit deliveries to overnight hours? I assume it's some kind of rsync on the backend, so the caveman version of just unplugging it would work, but might send folks into a tailspin on their end. I'm sure our IT person could limit it's bandwidth at the network level too, but a permanent limit feels a bit defeating of the concept.