ICMP-X Ingest Fail

[Sebastian Binz]

After I had solved the initial trouble via ingesting directly from a harddrive, the issue now returned on another movie.

So as it currently stands:
- The problem was not related to the movie. First movie was about 120GB, second now about 60GB. Between the first and the last one, about half a dozen other movies ingested just fine.
- It appears not being caused by the type of delivery (first one came via WeTransfer and needed to be unzipped, second one now was a DCP exported from DaVinci Resolve) without any unzipping/download straight from a harddrive
- Ingest from a Windows PC via FTP, running Filezilla FTP Server fails, no matter how many attempts
- Ingest from a Ubuntu box using the FTP server provided by James Gardiner's Catcher fails also permanently
- Ingest from Synology NAS via FTP succeeds
- the transfer from the Catcher Ubunto to the Synology NAS (via Downloadstation from FTP) also got stuck twice before being successful at the third try.

All devices are within the same VLAN, connected to the same switch.

I tried upgrading the FileZilla FTP Server to the latest version, but quickly learned the hard way that that is no good, as also discovered in this thread (as the ICMP is using an odd LIST -a command) so I reverted to 0.9.60.

The barco SP4K-15 is on the almost latest version 1.9.11 already.

I am running out of ideas where to troubleshoot next.
It looks like there is something fishy going on in the network (as the MTU size was decreased to 1492 as discussed here, since most transfers run stable with high speed), but this appears to affect some transfers and client/server combinations more than others. And the ICMP's FTP client doesn't seem to help when it comes to.


Any suggestions on how to analyze the network stability?

[Ryan Gallagher]

After I had solved the initial trouble via ingesting directly from a harddrive, the issue now returned on another movie.

So as it currently stands:
- The problem was not related to the movie. First movie was about 120GB, second now about 60GB. Between the first and the last one, about half a dozen other movies ingested just fine.
- It appears not being caused by the type of delivery (first one came via WeTransfer and needed to be unzipped, second one now was a DCP exported from DaVinci Resolve) without any unzipping/download straight from a harddrive
- Ingest from a Windows PC via FTP, running Filezilla FTP Server fails, no matter how many attempts
- Ingest from a Ubuntu box using the FTP server provided by James Gardiner's Catcher fails also permanently
- Ingest from Synology NAS via FTP succeeds
- the transfer from the Catcher Ubunto to the Synology NAS (via Downloadstation from FTP) also got stuck twice before being successful at the third try.

All devices are within the same VLAN, connected to the same switch.

I tried upgrading the FileZilla FTP Server to the latest version, but quickly learned the hard way that that is no good, as also discovered in this thread (as the ICMP is using an odd LIST -a command) so I reverted to 0.9.60.

The barco SP4K-15 is on the almost latest version 1.9.11 already.

I am running out of ideas where to troubleshoot next.
It looks like there is something fishy going on in the network (as the MTU size was decreased to 1492 as discussed here, since most transfers run stable with high speed), but this appears to affect some transfers and client/server combinations more than others. And the ICMP's FTP client doesn't seem to help when it comes to.


Any suggestions on how to analyze the network stability?

This is starting to sound more and more like a network issue somewhere... maybe there are different retry thresholds at play between the different FTP servers in your testing... and the Synology is getting the luckiest with it's settings.

Is there IT support within your org you can tap into to get into the weeds on the networking topology and testing?

I've never used them but there are software based network analyzers in the paid software domains to consider, example DataDog. Probably based on things in the open source domain, but I'm not privy to what those are.
https://www.datadoghq.com/dg/monitor...network-device

Personally I've been drooling over these gig-bag sized testers:

https://netool.io/pro/

[James Gardiner]

Umm.. it was me. I would set up a FTP server that logs ALL low level commands to see what is happening. (On the server side). But then again, I think filezilla server does that as default? The filezilla client does.
The target devices, you can use filezilla to ftp from them yourself without a problem?
Its just the ICMP having the issues?
Can you ping the ICMP IP from the ftpserver without any issues?

[Carsten Kurz]

Sebastian - you use this system since almost two years. Can you identify when the trouble started?

[Marcel Birgelen]

Considering your note about Zips internal integrity, really an external hash/checksum's benefit is more accurately described to permit checking before extraction... cause extraction of very large archives is CPU/time intensive as well. No reason to start extracting if the checksum is invalid.

Both checksums serve a different purpose though. The checksums provided alongside downloads are two-fold:

- Check if the file transfer was without errors.
- Check if the file wasn't tampered with, especially if the file is being served by a CDN or download mirror. This is especially true if the checksum is provided by other means than the download site itself.

This checksum doesn't prevent you from downloading a file that was previously damaged.

The internal hashes are there to check if the compression/decompression process resulted in an undamaged file.

Windows' internal ZIP implementation is notoriously bad and doesn't check for CRC32 errors during decompression. It can't handle any ZIP files beyond the most vanilla ones. Add encryption or some special features, and stuff will also break. Some versions also don't handle 4GB+ files correctly, that even includes some 64-bit Windows releases, so for any large files, it's really essential to use tools like WinZip or 7Zip.

[Sebastian Binz]

Sebastian - you use this system since almost two years. Can you identify when the trouble started?

To some extent only.
We replaced the network infrastructure by a Unifi Setup in Feb25. This worked well for about 6 months without any flaws.
First issues arose when we changed the WAN connection to this setup from an external Router (FritzBox) to a PPPoE connection via a Draytec Modem. For some reason that impacted the local LAN traffic, but some MSS clamping by telling the router to limit the MTU to 1492 bytes solved that issue.

I am inclined that this new issue now has only happened since we updated the SP4K/ICMP from a version about 1,5 years old to the 1.9.11. However, I cannot completely rule out that there is still some funky network stuff going on.

Now, I did some log-watching today and start to understand better what is going on:

I initiated an ingest to the ICMP from the FileZilla FTP Server.
The transfer starts and both systems start to count up the progress percentage.
However, after a while (15-20 mins) I can see on the FTP server log that the connection is closed, re-established and a retry command is sent. The file transfer then starts again from 0% in the FTP server log, but the ICMP continues to count up. This can happen several times. So by the time the ICMP reports a progress of 50%, the FTP server is only telling me that the file has been transferred to 20%, because the transfer has been re-started at 30%.

I did increase all timeout values in the FileZilla FTP Server to >120min.
The only thing it did was that I no longer see the timeouts in the log, but the behaviour of the transfer being restarted several times still occurs without any obvious reason every 15 min. or so.

On the unify gear, pretty much every possible option that hints towards traffic inspection is disabled and the windows firewall is also disabled (and this also occurs when ingesting from the Ubuntu Box runing the catcher software).

This is the ICMP Log, the FTP server restarts from 0% every time the ICMP says its retrying. The ICMP just continues to count up until it stalls.

Oct 26 14:20:11 icmp user.warning SMS: SMS- curl error 56 - retrying 1...
Oct 26 14:36:05 icmp user.warning SMS: SMS- curl error 56 - retrying 2...
Oct 26 14:51:59 icmp user.warning SMS: SMS- curl error 56 - retrying 3...
Oct 26 15:07:37 icmp user.err SMS: SMS- Copy failed: (10501) source read error: curl error 36 Error
Oct 26 15:07:41 icmp user.info SMS: Ingest- Ingest job failed [CPL: OW130Min_FTR-1-25_F_DE-XX_51_4K_20250905_SMPTE_OV (JobId = ca205e75-c4cc-419e-9365-0a33382c4605)]: CPL copy failed
Oct 26 15:07:41 icmp user.debug SMS: SMS- Unmount: ftp://extHDD:***@192.168.20.160/ (/)

This could point in a direction that the ICMP in it's current version has a nasty FTP glitch that doesn't handle the RETR command well....
But then again this doesn't answer the question why the retries occur after all - there seems to be some other time-out I don't see yet.

[James Gardiner]

Can you plug the ICMP directly into the ftp server? Isolate the problem for everything but the ICMP and ftp server. But yes, it would tend to give the ICMP a higher likelihood its behaving poorly, as it's the main veriable that has changed.

Try a Debian vsftp server on the Ubuntu machine (With the help of ChatGPT)

Debian-based FTP server version (using vsftpd) with full debug logging and a guest/guest login, all running in Docker:

docker run -it --rm \
-p 21:21 -p 21000-21010:21000-21010 \
-e FTP_USER=guest \ -
e FTP_PASS=guest \
-e PASV_MIN_PORT=21000 \
-e PASV_MAX_PORT=21010 \
-v $(pwd)/ftpdata:/home/guest \
fauria/vsftpd:latest

Then, to enable full debug logs (i.e. every FTP command), you can pass vsftpd a custom configuration override like this:

1. Create a local config file vsftpd.conf:
   

   cat > vsftpd.conf <<'EOF'
   	listen=YES
   	anonymous_enable=NO
   	local_enable=YES
   	write_enable=YES
   	local_umask=022
   	dirmessage_enable=YES
   	xferlog_enable=YES
   	connect_from_port_20=YES
   	xferlog_std_format=NO
   	log_ftp_protocol=YES
   	dual_log_enable=YES
   	vsftpd_log_file=/var/log/vsftpd.log
   	pasv_enable=YES
   	pasv_min_port=21000
   	pasv_max_port=21010
   	user_sub_token=$USER
   	local_root=/home/$USER
   	debug_ssl=YES
   	EOF[*]

2. Then run:
   

   docker run -it --rm \
   	-p 21:21 -p 21000-21010:21000-21010 \
   	-e FTP_USER=guest \
   	-e FTP_PASS=guest \
   	-v $(pwd)/ftpdata:/home/guest \
   	-v $(pwd)/vsftpd.conf:/etc/vsftpd/vsftpd.conf \
   	fauria/vsftpd:latest[*]

3. Logs will appear inside the container (/var/log/vsftpd.log) or you can stream them live via Docker logs:
   

   docker logs -f <container_id>