Film-Tech Forum ARCHIVE: DCP Hard drive formatting and EXCEPTION ERRORs

This is topic DCP Hard drive formatting and EXCEPTION ERRORs in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=002449

Posted by Frank A Cianciolo (Member # 7941) on 10-24-2015, 04:51 PM:

I love the DCP-O-MATIC program. I've gotten quite good with it and enjoy making promotional ads, music and policy DCPs. Here's an example of my animation and music in a "welcome" policy piece:

https://www.youtube.com/watch?v=-kg6GLPq26A

I have only one problem. DCPs that I delete from my IMB and then later try to ingest again often won't re-ingest. I get an EXEMPTION error. What does that mean? How might I avoid it?

I have an NEC900c - and am ingesting material through the usb port from a 1TB hard drive.

Posted by Carsten Kurz (Member # 5396) on 10-24-2015, 05:12 PM:

Which Server/IMB do you use with/in the NC900C?

- Carsten

Posted by Frank A Cianciolo (Member # 7941) on 10-24-2015, 05:18 PM:

SX-3000S2K-N2 GDC - Standalone 2K IMB w/ HDMI & 2TB Panel Mount Storage for NC900C

Posted by Frank Cox (Member # 6258) on 10-24-2015, 05:29 PM:

Are you sure that isn't a EXCEPTION error? If that's what it is, I think that indicates a bad spot on the distribution hard drive (or whatever you're ingesting from).

Posted by Frank A Cianciolo (Member # 7941) on 10-24-2015, 05:50 PM:

Oh yes - EXCEPTION - my mistake -

Interesting, I just checked and noticed the hard drive is formatted MS-DOS (FAT), but the partition is Mac OS Extended. It's strange that it *usually* works.

Maybe I should reformat another drive and partition both MS-DOS (FAT) and try re-ingesting after tonight's shows?

Posted by Carsten Kurz (Member # 5396) on 10-24-2015, 06:30 PM:

You are probably mixing partition scheme with format. A partition can never be MS-DOS/FAT AND Mac OS Extended at the same time. You may have MBR partition table + Mac OS Extended format though.

ISDCF drive format recommendations are here:

http://isdcf.com/papers/ISDCF-Doc3-Disk-Drive-Recs-ChangesShown.pdf

Best option for commercial mass distribution is MBR partition table and ext2 (or ext3) format with inode size=128.

ext2 or ext3 is a bit of a pain to create on both windows and Mac OS systems, not possible without extra tools of varying robustness.

Most servers nowadays cope well with reading NTFS, and if you are coming from windows, NTFS is usually easier to create/copy to if you know it works for your system.

Most systems create GUID partition maps nowadays per default when partitioning an empty disc. Creating MBR needs special attention/procedures and works only on drives up to 2TB.

Now I still wouldn't understand why you only get this error when reingesting a DCP that previously ingested without any issue...

Maybe there is a bug in the IMB software that creates incomplete database entries when deleting DCPs.

It may still happen due to a wrong ingest media format, if it really only happens with DCPs you create yourself and ingest from a disc with a strange formatting or partition scheme.

- Carsten

Posted by Frank A Cianciolo (Member # 7941) on 10-24-2015, 06:49 PM:

Thanks for the info and link - unfortunately your reply and the link are mostly over my head - but maybe this info will help -

Here is the info on the hard drive:

Name : Hitachi HDS5C3020ALA632 Media
Type : Disk

Partition Map Scheme : Master Boot Record
Disk Identifier : disk1
Media Name : Hitachi HDS5C3020ALA632 Media
Media Type : Generic
Connection Bus : USB
USB Serial Number : 123456789012
Device Tree : IODeviceTree:/PCI0@0/EHC1@1D,7
Writable : Yes
Ejectable : Yes
Location : External
Total Capacity : 2 TB (2,000,398,934,016 Bytes)
S.M.A.R.T. Status : Not Supported
Disk Number : 1
Partition Number : 0

Here is the info on the partition:

Name : DCP_DRIVE
Type : Volume

Disk Identifier : disk1s1
Mount Point : /Volumes/DCP_DRIVE
File System : Mac OS Extended
Connection Bus : USB
Device Tree : IODeviceTree:/PCI0@0/EHC1@1D,7
Writable : Yes
Universal Unique Identifier : EA2E5056-4E1F-3547-BD97-92F4E6084150
Capacity : 2 TB (2,000,398,901,760 Bytes)
Free Space : 653.29 GB (653,291,036,672 Bytes)
Used : 1.35 TB (1,347,107,864,576 Bytes)
Number of Files : 763,162
Number of Folders : 896
Owners Enabled : No
Can Turn Owners Off : Yes
Can Be Formatted : Yes
Bootable : Yes
Supports Journaling : Yes
Journaled : No
Disk Number : 1
Partition Number : 1

UPDATE:

I just successfully re-ingested a DCP that was getting the EXCEPTION error by putting it on a new HDD with a partition formatted MS-DOS (FAT)!

Posted by Steve Guttag (Member # 268) on 10-24-2015, 11:52 PM:

James Gardiner has a decent video on how to create EXT2/EXT3 drives on either Mac or Windows computers by basically using Ubuntu (Linux). Not by installing the other OS (unless you want to on a separate partition) but by running the OS right off of an image of it.

F-T isn't allowing me to post the link (something about it having problems in the past). But if you go to his web site, and search for recent posts or even DCP Windows Mac you should find it.

Posted by Frank A Cianciolo (Member # 7941) on 10-25-2015, 01:44 AM:

Thanks Steve - I thought I had it all working - but my MS-DOS (FAT) partition still doesn't seem to work on the bigger DCP files - I started doing some research from Carsten's post and was beginning to think I need to go Linux(which maybe I still should!) - but for now, I'll check out the video!!!

Posted by Frank Cox (Member # 6258) on 10-25-2015, 01:57 AM:

There is a maximum file size of 4gb (4,294,967,296 bytes minus 2 bytes) with a DOS FAT32 formatted drive.

So if your DCP exceeds 4gb less two bytes, it won't fit onto a FAT-formatted disk, even if there is otherwise unlimited space available.

Posted by Carsten Kurz (Member # 5396) on 10-25-2015, 07:43 AM:

It may be that the GDC is able to read HFS volumes, but not in the most robust way.

I assume your are using a Mac to copy the DCP to your drive?

FAT(32) is okay as long as your individual FILES are not larger than 2/4GBytes. That will usually work for trailers only. Even a decent short will be larger than 4GB.

Note this does not mean the full DCP folder size, but only the individual MXF files in it. If you pack a DCP into a ZIP or other archive format, the single ZIP file may run against that limit, while unpackaged, no single MXF may come close to it.

You can install Fuse for OS X/fuse-ext2/NTFS-3G and use ext2/3 or NTFS as well for ext2/ext3 and NTFS read/write support. Both are a bit wonky for writing, but if you are careful enough, they should work. I regularly write DCPs to NTFS and ext2 drives from my Mac. If you restrict drive use for DCP copying and not general use, it should be okay.

Some external drives for Mac come with the commercial version of NTFS-3G (Tuxera NTFS for Mac), which is faster. I found it on a Toshiba USB drive.

Bootcamp Windows or Windows/Linux in virtual machines are other options to write to NTFS or ext2/3 drives (fast and safe).

Steve - James missed the point in his video to create an MBR partition table. His video shows todays default GUID partition table being used. I hope he'll be able to fix it in the video. The video is here:

https://www.youtube.com/watch?v=n6rGDYoNXi0

- Carsten

Posted by Leo Enticknap (Member # 534) on 10-25-2015, 12:41 PM:

Maybe the OP would like to rename this thread? It's becoming clear that this is not a DCP-o-Matic related problem at all, but rather one with his combined server/IMB not liking HFS+ and maybe some other non-ISDCF approved journaling filesystems.

Agreed with Steve that Ubuntu is a very useful way of handling the full range of filesystems. All the computers in our booths are partitioned to dual-boot into Windows and Ubuntu for precisely that reason. You can add the capability to read and write HFS and HFS+ volumes to it as well (from a terminal window, enter "sudo apt-get install hfsprogs"). Once done, GPartEd* will create pretty much any partition table and filesystem combo you could ever want, and Nautilus will copy files to and from it.

* GPartEd is preinstalled on the live CD image, but not on a permanent hard disc installation of Ubuntu - you have to add it ("sudo apt-get install gparted").

Posted by Carsten Kurz (Member # 5396) on 10-25-2015, 12:52 PM:

Indeed. The reason for his issue is not DCP-o-matic, but the drive he is using exclusively to copy the DCPs he created with DCP-o-matic, in contrast to commercial distribution drives that come with the proper format and do NOT cause these issues.

- Carsten

Posted by Frank A Cianciolo (Member # 7941) on 10-25-2015, 02:20 PM:

THANK YOU EVERYONE - this is a problem I have been dealing with for a couple years - In the end I would just remake DCPs from their original .mov files - looking forward to fixing this problem!

Posted by Carsten Kurz (Member # 5396) on 10-25-2015, 02:44 PM:

What Mac OS X version do you run? Do you have a bootcamp or a virtual windows installation running on that Mac as well?

As long as your DCP MXF files do not grow larger than 4GB, you can continue to use a FAT formatted disc. Sooner or later you will run into trouble with larger files, and you will need to use either ext2/3 or NTFS format.

- Carsten

Posted by Frank A Cianciolo (Member # 7941) on 10-25-2015, 04:19 PM:

You were right a few posts back, I have an MBR partition table with Mac OS Extended format partition - with this, I can ingest as large a DCP as I like, but often get these EXCEPTION errors on re-ingests.

I formatted a second drive last night to MS-DOS and was able to upload previously "EXCEPTION" error-ed DCPs, but COULDN'T do DCPs larger than around 16Gb (which apparently contain individual files over the 4GB ceiling).

Posted by Leo Enticknap (Member # 534) on 10-26-2015, 11:51 AM:

This appears to be a bad interaction between your server/IMB and HFS+. I'd suggest trying NTFS or ext2/3.

The problem with that will be if you're using a Mac to make your DCPs. My understanding is that Mac OS will read NTFS, but cannot write to NTFS volumes.

Putting Ubuntu on an old PC or laptop would solve that problem. You could create your DCPs on your Mac and save them to an external, HFS+ formatted hard drive. Then, connect that drive to the Ubuntu computer, plus another, ideally partitioned and formatted as MBR/ext3, copy the DCP onto it, then ingest.

Alternatively, can you ingest DCPs via FTP? For ad, trailer, snipe and short DCPs I find that this is the easiest way to do it, though our network transfer speed is too slow to make this viable for features (around 20-25 minutes per gigabyte).