ICMP-X Ingest Fail

[Sebastian Binz]

Hi,
I am seeing a very strange behaviour when trying to ingest into our ICMP-X via FTP from a Filezilla Server running on Windows.
One is a main feature with 136 GB, the other is a short clip with only 9GB.
Both come from the same studio via WeTransfer. We have tried differnet zip tools for unpacking without any change. According to the studio, the same file has been successfully used by various other cinemas.

Other files just work fine, but there are two which result in an ingest error. Logs of the ICMP show
SMS- Copy failed: (10501) source read error: curl error 36 Error
and a Curl 56 Error.

That would point towards a network issue, but as other files ingest fine from the same source, I don't really see that.



The transfer runs above 100%, continues to increase above the 9.59GB and then at some point completely fails with the huge negative numbers shown below.

Has anyone seen this behaviour before or any troubleshooting advise?
ingest_fail.png

[Leo Enticknap]

Does it ingest OK if you copy the DCP files onto physical media and then plug it in to one of the ICMP's USB jacks?

If so, that would suggest that either Filezilla Server or a network switch along the way is doing something unhelpful to them. In that case, maybe try another FTP server (e.g. the one built in to Windows) as the next troubleshooting step?

[Caleb Williams]

"Source read error" indicates an issue with the source files on the server. (Provided there are no network issues: since other content is transferring fine).

Can you confirm that the effected content is whole on the FileZilla server? I would start there. You may need to re-download content from WeTransfer then re-try an ingest.

[Sebastian Binz]

Some testing done with the smaller file:

-- Ingest via FTP from Windows PC running FileZilla Server ->Fail
-- Data Transfer from Windows PC via Windows Explorer onto a Synology NAS --> Fail
-- Data Transfer from Windows PC via FileZilla FTP client into Synology NAS --> Success!
-- Ingest via FTP from Synology NAS into ICMP: Success!

Now downloading again via a second WeTransfer link, that will take a wile....

[Caleb Williams]

Does WeTransfer provide a way to check file checksums? That would be a faster way to confirm that what you've downloaded from them is complete

[James Gardiner]

I discovered a few FTP bugs in the ICMP implementation a while ago. Reported and fixed by Barco. Make sure the version of the ICMP software you are using is not too old. It has a similar looking fault as you described.

[Marin Zorica]

I discovered a few FTP bugs in the ICMP implementation a while ago. Reported and fixed by Barco. Make sure the version of the ICMP software you are using is not too old. It has a similar looking fault as you described.

I remembered about two, or three year ago when distributors started to deliver content via internet, that I have set up many FTP for direct ingest from PC or NAS. And I remember like every NAS did work without problems on ICMP, while some PC didn't. In other hand i newer had problems for example with windows built in FTP and doremi. Now after some time, I did not have that issue any more.

[Carsten Kurz]

Sebastian - if these files come ZIPed, you absolutely need a later version of 7-ZIP to unpack them. ZIP files larger than 4GB are a known problem.

You may also, after unZIPing, run them through DCP-o-matic player/verifier. Are you sure your Filezilla is configured for binary transfer type?

[Ioannis Syrogiannis]

Does WeTransfer provide a way to check file checksums? That would be a faster way to confirm that what you've downloaded from them is complete

In regards to that, I will second Carsten's suggestion:
The DCPs themselves have hash checksums for most of the files. Namely, the audio, video, subtitle and composition playlist files.
One may run DCP-o-matic Verifier on a DCP and make sure that everything is according to the DCP author's intention. That is better than checking the hash checksum of WeTransfer, since it covers the case of something being wrongly uploaded there.

Besides that, talking about NAS, some new ones offer docker. If one is lucky in that sense, they might be able to run a check on NAS itself, by using docker-ized versions of Clairmeta or Digital Cinema Tools (dcp_inspect).
No docker version of DCP-o-matic Verifier yet. (...that I know of.)

[Caleb Williams]

In regards to that, I will second Carsten's suggestion:
The DCPs themselves have hash checksums for most of the files. Namely, the audio, video, subtitle and composition playlist files.
One may run DCP-o-matic Verifier on a DCP and make sure that everything is according to the DCP author's intention. That is better than checking the hash checksum of WeTransfer, since it covers the case of something being wrongly uploaded there.

Besides that, talking about NAS, some new ones offer docker. If one is lucky in that sense, they might be able to run a check on NAS itself, by using docker-ized versions of Clairmeta or Digital Cinema Tools (dcp_inspect).
No docker version of DCP-o-matic Verifier yet. (...that I know of.)

With WeTransfer specifically, we normally receive the content in a single *.zip directory. DCP spec requires hash checking for each asset, which for a feature with multiple assets, can be a pain to do by hand. If WeTransfer provides a single hash for the whole archive, that could save some time when verifying contents.

[Ioannis Syrogiannis]

Excuse me for not making it clear enough, Caleb: I was not suggesting (neither Carsten) hash checking by hand. (I suppose you mean to get the hash checksums from the packing list file and running hash checks against them.) There is a program, open source, running in Linux, Mac and Windows machines called DCP-o-matic Verifier. It's part of what became a suite of DCP authoring programs. One can point to a DCP, or a folder full of DCPs that program and verify those. Not only by hash checking, but by other means as well.

There are two other well known programs that work similarly, that I mention, ClairMeta and dcp_inspect. dcp_inspect may also check more than one DCPs at once, but that is not the case with ClairMeta, as far as I know. Those two last ones are a bit trickier to install, especially when not on linux, but either of three may prove indispensable for having some insight on DCPs. Especially because the info you get doesn't demand to ingest the DCP(s) in a cinema server.

The main benefit of using the DCP's own hash checksums, instead of the one for the .zip file one downloads from WeTransfer is that it helps troubleshoot the possible problems in the transfers before one downloads the WeTranfer file. Meaning, if there was a problem while uploading the DCP to WeTranfer, or in another copying that might have taken place between the authoring program and uploading it. Covering, thus, more ground to troubleshoot.

It seems you are a busy person, but you enjoy tinkering and figuring things out by yourself. For that reason, I believe you would find those programs interesting, to say the least.

[James Gardiner]

A tool that checks for files on a DCP level is the path to go. What if a broken DCP was send through WeTransfer, it will pass as it is exactly the same Broken DCP on the other end.
Integrating checks that check for DCP centric CRC checks, is a far better tool, in a DCP workflow model.

[Ryan Gallagher]

There are two points of interest here, checking the DCPs themselves, versus checking a large archive transferred completely. They both have value IMHO, especially if WeTransfer or whatever method used is not automatically decompressing whatever archives are provided. I'm not familiar so perhaps it does so and the zip is just the transport file that temporarily exists.

I for one wish every independent who zipped a DCP up for transferring via consumer cloud services would know enough to include a hash/checksum file too. It is a great troubleshooting tool even if you don't use before ingest or DCP verify steps. If your ingests are failing for completeness or integrity reasons you can immediately independently DCP verify them separately, as well as compare the transfer hashes, and at least know that the archive was as "as provided" and the download was not the culprit.

Considering downloads are often the most time consuming part of the process, in a time crunch it's a real benefit to know downloading a 2nd time would not produce different results, and move on to notifying parties something is wrong with the DCP.

[Frank Cox]

If your dcp is transferred to you as a zip file then that zip file already has a checksum built in; there's no need to generate one separately unless you really want to.

You can use 7za (or 7z I suppose) to view the CRC if you want.

This example zip file (not a dcp, just an archive that I have in front of me)has two files inside of it:

7za l -slt 2438_28798_01.zip |grep CRC
CRC = 94FA5D4B
CRC = 06DAE7EF

I think zip/unzip will do that too but I'm not entirely sure how.

[Ryan Gallagher]

If your dcp is transferred to you as a zip file then that zip file already has a checksum built in; there's no need to generate one separately unless you really want to.

You can use 7za (or 7z I suppose) to view the CRC if you want.

This example zip file (not a dcp, just an archive that I have in front of me)has two files inside of it:

7za l -slt 2438_28798_01.zip |grep CRC
CRC = 94FA5D4B
CRC = 06DAE7EF

I think zip/unzip will do that too but I'm not entirely sure how.

I was just using zip as an example archive format. But you are right, most compressed archives are gonna have some form of internal checking, so presumably if they extract without error they are complete. Though I don't know how well windows integrated zip tools behave in this regard. But it should be noted those are CRCs for EACH contained file... not a CRC of the archive itself. The old habit of including one in your download repository was really best practice for formats that lack internal integrity features such as ISOs.

Considering your note about Zips internal integrity, really an external hash/checksum's benefit is more accurately described to permit checking before extraction... cause extraction of very large archives is CPU/time intensive as well. No reason to start extracting if the checksum is invalid.