Film-Tech Forum ARCHIVE: Ingest issues

This is topic Ingest issues in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=003684

Posted by Ohad Ephrath (Member # 10492) on 05-14-2019, 05:43 AM:

Hello all.
I work in the film festival industry and do a lot of DCP ingestion and testing.
I have recently bought a CRU DX115 and have decided to put a Crucial MX500 2TB in it for faster ingestion. (My first CRU is with the WD Black 2TB)
I have formatted the drive to be EXT3, 128 inode.
I work on a mac system and use Paragon EXTfs to transfer DCPs to the CRU.
I have noticed that some DCPs do ingest but some give a file hash error.
Is there an issue with SSDs being used for ingestion? Do all SSDs work well with EXT3? Should I format it to NTFS?

Posted by Ed Gordon (Member # 10866) on 05-14-2019, 05:36 PM:

There is a mention of similar issues at:
https://groups.google.com/forum/#!topic/opendcp/fEqNav8BJGM

You might also search "wear leveling and file hash errors" in regard to SSD usage. The use and validation of images of SSD's (not necessarily individual files) is a concern for digital forensic professionals. Example article at:

https://www.forensicmag.com/article/2014/06/solid-state-drives-part-6

Ed

Posted by Marcel Birgelen (Member # 6801) on 05-14-2019, 07:01 PM:

The problems with complex NAND-based disks (also known as SSDs) and forensics are a bit out of scope here. Nobody is trying to recover data that, according to the filesystem doesn't exist anymore. A forensic copy of an SSD is still something problematic, compared to magnetic discs.

In this particular case, there seems to be data-corruption happening somewhere, which is most likely an indicator for a driver problem or a defective SSD.

You can simply check if files are different by creating a hash of the original and the one on the SSDs, e.g. a simple MD5 hash would be a good indicator (yes, MD5 isn't usable anymore for real security related stuff, but for comparing two files without any possible malicious intent it's still fast, ubiquitous and reliable).

Posted by Carsten Kurz (Member # 5396) on 05-15-2019, 09:00 AM:

From my experience, Paragon ExtFS is not very reliably. If you do a lot of copying, you should try to use a native Linux (you should be able to boot your Mac into e.g. GPARTED or another Linux on a stick.) Also, try to perform a hash check immediately after the copy, or while copying.

How is that SSD connected to the Mac? Note, SSDs usually do not perform faster during ingests than hard discs. They may be a bit fast during copying from your Mac, but the ingest on the DCI server is usually slowed down by housekeeping tasks.

- Carsten