Film-Tech Forum ARCHIVE: CRU vs. FTP ingest on Doremi

This is topic CRU vs. FTP ingest on Doremi in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=003299

Posted by Biraj Thakuri (Member # 10203) on 04-30-2018, 05:06 AM:

Hey everyone.

I have been working on doremi showvault server for a couple of months. I ingest new features every Thursday and I wanted to know which is the most fastest and reliable way of ingesting between CRU drive, FTP and USB drive. Also what is the maximum speed of them.

Any suggestions and info will be appreciated.

Posted by Carsten Kurz (Member # 5396) on 04-30-2018, 06:06 AM:

Depends on the scenario,

USB is usually the slowest method. However, recently ShowVaults come with a new 'X10' revision Mainboard that offers a USB3.0 connection. If you have such a recent ShowVault, your USB ingest speed may be on par with CRU or FTP.
You probably already experienced that ingest is extremely slow when the server is currently playing.

Regarding FTP - you don't get features on FTP-capable ingest media, so, if you ingest something though FTP, it will most likely have been copied to a network drive, NAS, another server, etc. That copy process takes time as well, so, even if FTP ingest as such would be faster, depending on the situation, you would need to factor in the preceeding copy operation. If you use multiple ShowVault servers, a mixed method is often used, CRU-ingest on one server, then FTP from there to the other servers.

- Carsten

Posted by Dave Macaulay (Member # 813) on 04-30-2018, 06:07 AM:

I don't have exact speeds for these ingest options, and rated speed of any link is never realized in actual data transfer speed - there must be a lot of extra data being transferred for error correction etc.
Slowest is USB, since the server probably only does USB2 (USB3 is pretty fast on servers that have USB3 ports like an ICMP, and Showvaults made in the last year or so do have USB3 BUT you need a CRU bay with USB3 - a blue jack).
My usual guess is that a feature will ingest with USB2 in roughly its playing time, it can be less or more of course - 3D and 4K take longer. Most external CRU bays can use either USB or SATA but one must add an external SATA port to the server to use that.
Using SATA or USB3 on an external bay, or the internal SATA CRU bay on the 4U ShowVault, is a bunch faster: a feature should ingest in about 1/4 to 1/3 of its playing time.
FTP over a 1000bT network is about the same speed. Over 100bT ... take a long lunch. And speed does depend on the source, a slow computer won't manage the maximum 1000bT data rate.
Also, as you undoubtedly know, a Doremi DCP or SV server doing playback will be very slow ingesting.

Posted by Leo Enticknap (Member # 534) on 04-30-2018, 07:43 AM:

quote: Carsten Kurz
Regarding FTP - you don't get features on FTP-capable ingest media, so, if you ingest something though FTP, it will most likely have been copied to a network drive, NAS, another server, etc. That copy process takes time as well, so, even if FTP ingest as such would be faster, depending on the situation, you would need to factor in the preceeding copy operation.

Usually, but not always.

At my last theater job I handled festival ingests using a custom-built CentOS PC with both a USB3 card and a built-in CRU bay. I would ingest into either of the two screen servers (one was a Barco Alchemy, the other a DSS200) by FTP-ing directly from the connected CRU drive or USB device (flash stick or USB bus-powered hard drive) into them, via gigabit ethernet. I used Filezilla for this, and it worked well. I would typically achieve 100-120mbps from the SATA2/CRU and 70-90 from USB3. Obviously, USB2 was a lot slower - usually around 25-35. I found that this was a good way to take advantage of the USB3 speed gain to ingest into a screen server that does not have a USB3 jack.

Posted by Biraj Thakuri (Member # 10203) on 04-30-2018, 08:13 AM:

quote: Carsten Kurz
You probably already experienced that ingest is extremely slow when the server is currently playing.

Regarding FTP - you don't get features on FTP-capable ingest media, so, if you ingest something though FTP, it will most likely have been copied to a network drive, NAS, another server, etc. That copy process takes time as well

Yes I have an older model showvault with usb 2.0
Yes the usb 2.0 is very slow. Takes 1 hr to ingest 80GB feature.
I had given the FTP a thought because we have a gigabit switch and it would make sense to copy feature to PC one time and then network ingest in two servers. I manage two screens.

quote: Dave Macaulay
FTP over a 1000bT network is about the same speed. Over 100bT ... take a long lunch. And speed does depend on the source, a slow computer won't manage the maximum 1000bT data rate.
Also, as you undoubtedly know, a Doremi DCP or SV server doing playback will be very slow ingesting.

Thanks for the info Dave. Yes the speed does take a hit if ingest is done during a playback. We usually get features on CRU drive but in some cases they send portable drive which are unbearably slow. That's why I wanted to setup a PC server to copy at both screens at same time. What would be the speed if I ingest from server to server by ftp? also what PC specs are good for such type of transfers.

Posted by Carsten Kurz (Member # 5396) on 04-30-2018, 09:33 AM:

You should be able to achieve half realtime/playtime ingests durations through FTP.
If you use two servers, it certainly is not a bad idea to create a simple file server where you can copy ingest media to, and ingest from there. It doesn't need to be a high-end PC or server. Get a simple tower with some space for at least 3-4 3.5" drive slots.
network port/card should be Gigabit capable, but all current mainboards should offer that. You could use a cheap second hand PC, provided that they offer enough SATA ports/drive bays. And yes, even if your greatest ingest handicap are USB2.0 drives, that PC should offer a USB3.0 port. As many portable drives nowadays are in fact USB3.0, but will still only offer USB2.0 speed on the Doremis.

A simple USB2.0 copy to a local drive will be much faster than a USB ingest on a Doremi. You could also try to mount a CRU-frame into that PC, however, the benefit is small, as, once you ingested content from CRU onto one of your Doremis at decent speed, you can ingest it to the second Doremi at similar speeds. I guess USB2.0 is your main issue.

Every copy carries the risc of file corruption, though, so, if possible, you should use a dedicated file copy/sync application that will also explicitly verify the local copy against the source media.

- Carsten

Posted by Mark Gulbrandsen (Member # 72) on 04-30-2018, 12:29 PM:

If you use the USB 3.0 on it you also have to have a CRU bay that works on that spec to get the high transfer speed it offers. If you have an older CRU USB compatible bay it's going to ingest at the speed it was capable of doing when it was made.

Mark

Posted by Frank Cox (Member # 6258) on 04-30-2018, 12:41 PM:

On my GDC server the USB ingest works but it is oh-my-ghawd slow.

I've had a few occasions where I've had a CRU drive come in where the case is slightly bent such that it won't fit into the slot on the server so I've ingested it through USB and as I recall the last time I did that it was about a five hour wait. Ingesting features through the CRU slot usually takes around a half-hour.

So I'm glad I've got USB as an option for those sorts of circumstances but I sure wouldn't recommend it as the preferred practice.

Posted by Dave Macaulay (Member # 813) on 04-30-2018, 02:26 PM:

For server-server transfers, the two servers have to be networked: generally you want a separate network (ie on eth1) for media transfers so you don't lose automation or security data from a saturated network.
Then just set each server's "content feed manager" in control panel to have the other server listed. You use ingest manager for transfers between them. Setting up eth1 and just running a cable between them usually works fine for two servers but a gigbit switch is recommended as they handle handshaking and such better than port-port does.

Posted by Mark Gulbrandsen (Member # 72) on 04-30-2018, 04:02 PM:

With the GDC SX-3000 it does all management and transfers through one port and you only need one gigabit switch to make it all work. This also eliminates running dual network cables back to the TMS and it cuts down on that cost.

Mark

Posted by Dave Macaulay (Member # 813) on 04-30-2018, 07:42 PM:

Ahh! An unsolicited useless tidbit of information for equipment he doesn't have!

Posted by Biraj Thakuri (Member # 10203) on 05-02-2018, 02:20 AM:

Thanks Carsten and Dave.

Yes I had previously linked those two servers with each other and added them to the content feed manager as suggested. As Carsten suggested the real and main reason for me going through the FTP way was to ingest from usb 2.0 much faster. I have them linked with Gigabit switch .First copying the content to PC with usb 3.0 and then distributing them through FTP to both servers. Should I transfer one server at a time or both at once?

About the copying and file corruption. I am planning to use linux for the PC so to reduce such cases.

Yes Frank. Sometimes the CRU cases come a little bent and refuses to go all the way creating a major headache.

Right now I am having issue with setting the IP on the eth 1 interface in doremi. After configuring ETH0 I reboot and
After rebooting I get 0.0.0.0 on Eth0(projector connection) and the projector fails to connect. What am I doing wrong here?