Film-Tech Forum ARCHIVE: Servers won't recognize DCP drive

This is topic Servers won't recognize DCP drive in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=002749

Posted by John Wilson (Member # 269) on 07-12-2016, 06:06 PM:

Hi guys

Have a client-made DCP been supplied to us. I can get it to play on my non-DCI compliant PC, but no server in the booth will recognize it. Can anyone point me to telling them where they've gone wrong?

<?xml version="1.0" encoding="UTF-8"?>

-<PackingList xmlns="http://www.digicine.com/PROTO-ASDCP-PKL-20040311#">

<Id>urn:uuid:12c7ca9d-a451-48a3-bc7a-85df1c012c13</Id>

<AnnotationText>GRFinalist07_SHR_F_EN_AU_20_2K_20160705_MOT_SMPTE_OV</AnnotationText>

<IssueDate>2016-07-05T01:35:08-00:00</IssueDate>

<Issuer>The Official Top Ten Night</Issuer>

<Creator>ClipsterDCI 3.7.0.44</Creator>

-<AssetList>

-<Asset>

<Id>urn:uuid:34448c03-56c6-446d-9151-ff30083dfbd3</Id>

<Hash>qwOGiymho/qi1BJKgFvhWA62CR8=</Hash>

<Size>98376354</Size>

<Type>application/x-smpte-mxf;asdcpKind=Sound</Type>

<OriginalFileName>GRFinalist07_SHR_F_EN_AU_20_2K_20160705_MOT_SMPTE_OV_audio_01.mxf</OriginalFileName>

</Asset>

-<Asset>

<Id>urn:uuid:86568270-6b35-43e2-94ef-49857b906ad1</Id>

<AnnotationText>2016</AnnotationText>

<Hash>RRVW+Yb5WKtp+HPuBC0oJmLLwrM=</Hash>

<Size>7875</Size>

<Type>text/xml;asdcpKind=CPL</Type>

<OriginalFileName>CPL_GRFinalist07_SHR_F_EN_AU_20_2K_20160705_MOT_SMPTE_OV.xml</OriginalFileName>

</Asset>

-<Asset>

<Id>urn:uuid:9a341e50-aeef-4fc4-812a-a8e438627c4d</Id>

<Hash>wwE8LSU/L08cTayCQYjijrNUHW4=</Hash>

<Size>4783001217</Size>

<Type>application/x-smpte-mxf;asdcpKind=Picture</Type>

<OriginalFileName>GRFinalist07_SHR_F_EN_AU_20_2K_20160705_MOT_SMPTE_OV_01.mxf</OriginalFileName>

Posted by Monte L Fullmer (Member # 2797) on 07-12-2016, 06:12 PM:

Try this assetmap to compare. This is a trailer assetmap:

<?xml version="1.0" encoding="UTF-8" standalone="no" ?><AssetMap xmlns="http://www.digicine.com/PROTO-ASDCP-AM-20040311#">
<Id>urn:uuid:93c57148-7184-41a3-a264-b6ab0ac850fc</Id>
<AnnotationText>CapCivWar_US</AnnotationText>
<VolumeCount>1</VolumeCount>
<IssueDate>2016-04-22T03:22:26-07:00</IssueDate>
<Issuer>DeluxeTechnicolorDigitalCinema</Issuer>
<Creator>dlx-combomapper 1.0.3</Creator>
<AssetList>
<Asset>
<Id>urn:uuid:1a20864b-c961-4858-afd3-d583594d3d1c</Id>
<PackingList/>
<ChunkList>
<Chunk>
<Path>PKL_1a20864b-c961-4858-afd3-d583594d3d1c.xml</Path>
<VolumeIndex>1</VolumeIndex>
</Chunk>
</ChunkList>
</Asset>
<Asset>
<Id>urn:uuid:a457053c-e774-44f9-ba3e-7dc50c1e215f</Id>
<ChunkList>
<Chunk>
<Path>DoctorStrange_TLR-1-2D_S_EN-XX_US-GB_51-HI_2K_DI_20160412_DTB_IOP_OV_audio_01.mxf</Path>
<VolumeIndex>1</VolumeIndex>
</Chunk>
</ChunkList>
</Asset>
<Asset>
<Id>urn:uuid:c054a1dc-0e09-4c86-a8bc-281870575b94</Id>
<ChunkList>
<Chunk>
<Path>DoctorStrange_TLR-1-2D_S_EN-XX_US-GB_51-HI_2K_DI_20160412_DTB_IOP_OV_01.mxf</Path>
<VolumeIndex>1</VolumeIndex>
</Chunk>
</ChunkList>
</Asset>
<Asset>
<Id>urn:uuid:ee064029-8fa3-4e20-95fc-a952db850c8a</Id>
<ChunkList>
<Chunk>
<Path>CPL_ee064029-8fa3-4e20-95fc-a952db850c8a.xml</Path>
<VolumeIndex>1</VolumeIndex>
</Chunk>
</ChunkList>
</Asset>
</AssetList>
</AssetMap>

-Monte

Posted by Scott Norwood (Member # 30) on 07-12-2016, 06:22 PM:

Are you sure that it is the DCP files themselves that are failing and not the way in which the drive is formatted or the partitions are laid out?

Posted by John Wilson (Member # 269) on 07-12-2016, 06:31 PM:

I don't have the answers to those questions Steve. I don't believe the owner will have them either. I'll have to go find who actually made it.

Posted by Carsten Kurz (Member # 5396) on 07-12-2016, 06:31 PM:

Yup. Check the drive format first. What servers do you operate, was is your computers OS? What drive type came this on? Do your servers show ANYTHING on that drive (e.g. folder names), or simply 'nothing'

If it's a short, you may be able to copy it to an NTFS formatted USB stick on your PC and retry. You could also try to install Filezilla Server on your PC and ingest from there over the network.

- Carsten

Posted by John Wilson (Member # 269) on 07-12-2016, 07:56 PM:

Thanks Carsten. I've managed to get the video and audio files into DCP-o-matic and re-make them.

This guy owes me a beer...

In answer to your questions, nothing shows up on our Doremi servers 'there is no content to ingest' and the TMS does nothing at all.

Thanks everyone for your input.

JW

Posted by Mark Gulbrandsen (Member # 72) on 07-12-2016, 09:03 PM:

Just show the thing a hunk of 70mm film. That ought to make it feel threatened so it loads!

Mark

Posted by John Wilson (Member # 269) on 07-12-2016, 10:38 PM:

Drive was formatted exFAT.

Posted by Ian Freer (Member # 1940) on 07-12-2016, 10:52 PM:

quote: John Wilson
Drive was formatted exFAT.

That's highly likely the issue. exFAT is great for Windows (well, Win7 onwards, not sure about Vista) and Mac, but not so great for Linux.

Obviously you have it sorted now, but in theory you didn't need to go to the effort of re-wrapping the DCP, simply copying the DCP from the exFAT formatted drive to an NTFS drive (if you're in Windows) would have solved the issue.

I'm not sure what the Mac solution would have been, if anybody wants to put up a solution?
I don't think using HFS+ would be universally reliable for linux based cinema servers, and Mac cannot write to NTFS can it?

I suppose you can use an app to read/write ext on a Mac...? If so, that's maybe the best way.

If it's a short film with files no bigger than 2 or 4GB, then FAT can save you, but anything longer than a few minutes, the video mxf is gonna kill that...

Posted by John Wilson (Member # 269) on 07-12-2016, 10:57 PM:

Hi Ian

I did try transferring it to an NTFS drive. It didn't work.

JW

Posted by Ian Freer (Member # 1940) on 07-12-2016, 11:21 PM:

Hmmm, the plot thickens...

Posted by Darryl Spicer (Member # 711) on 07-13-2016, 10:40 AM:

quote:
<OriginalFileName>GRFinalist07_SHR_F_EN_AU_20_2K_20160705_MOT_SMPTE_OV_audio_01.mxf</OriginalFileName

It could be possible that the servers are not recognizing the SMPTE format. See if they can make the content in the IOP format.

We have had issues with our TMS not seeing any content that is SMPTE. The individual servers will see it though, I think because of an update that was applied to them at some point.

Posted by Scott Norwood (Member # 30) on 07-13-2016, 12:26 PM:

Is the drive an "advanced format" type? If so, try mounting partition 2, not partition 1.

Most servers do not support exfat, but most should support NTFS.

The DCP files generally need to be in the root directory or one level below that.

Posted by Leo Enticknap (Member # 534) on 07-14-2016, 10:11 PM:

If the DCP server's software hasn't been updated for a few years, it might not like SMPTE DCPs.

If the frame rate is 24, might it be worth trying to remake it in DCP-o-Matic as Interop?

Posted by Stephen Furley (Member # 1277) on 07-15-2016, 07:48 PM:

I've several times had problems with small USB drives, and I don't know what the problem is. It's not the format, and it's not inadequate power, because I've tried connecting via a powered hub, but some of these disks will not mount. If there is a light it comes on, and the motor can be heard to spin up. Toshiba and Western digital 'pocket' disks both seem to be affected. The LaCie 'Ruggeged' disks with the orange rubber covering are interesting. The older ones with USB 2 and FireWire 400 do not work, and neither do the 'mini' ones which are USB only. The later ones, with USB 3 and FireWire 800 work, as do.the 'Rugged' USB sticks. I haven't yet tried the latest version of these disks, with USB 3 and Thunderbolt.

I have never found any solution to this problem other than mounting them on a Linux laptop, and copying the files to different type of disk.

Of course, these small disks are very attractive to small producers who send out their own films directly and do not have a distributor and who can just post these small disks in a padded envelope or two.

We're getting to the point now where shorter films could be delivered on a large USB stick, and these always seem to work, but are quite expensive.

Posted by Carsten Kurz (Member # 5396) on 07-16-2016, 08:00 AM:

An increasing number of film-makers ignore the ISDCF recommendation for ext2/3 single-partition MBR drives. I have seen many GUID dual partitioned NTFS&HFS+ drives that do not work on many servers, especially when created on a Mac with the system default hidden rescue or fake MBR partitions.

Part of the problem is that now more or less all current general computer systems default to GUID partition schemes.

People seem to think that a drive working on their personal machine should work just as well on a DCI server.

- Carsten

Posted by Leo Enticknap (Member # 534) on 07-16-2016, 01:28 PM:

Stephen: large capacity USB sticks aren't a more expensive option than the 2.5" SATA hard drive in a USB case/adaptor packages sold in Office Depot-type places anymore, at least not in the US. 128GB sticks go on Amazon for $30-50, depending on the speed. The only gotcha is that the cheaper ones take a very long time to write (typically 2-3 hours to write a 100GB DCP), but once done they are so much cheaper and less fragile to move around than hard drives, and you are guaranteed to have no problem with there not being enough USB bus power on a server.

If I'm making a DCP for someone else I would always advise them either to use a high capacity flash stick or a hard drive in a CRU cartridge (the latter if ingestion speed will be an issue for them and they are prepared to spend more), never a bus-powered USB drive, for all the reasons you give.

Admittedly, if you want a capacity greater than 128GB, then 2.5" consumer USB hard drives are still a cheaper option than a flash stick, but I doubt if this'll be the case for long.

The distributors who insist on using consumer USB 2.5" bus-powered drives really annoy me. Kino Lorber, Rialto and Alamo Drafthouse are the worst offenders: roughly half their drives either refuse to spin up when connected to our servers, or the server can't see the DCP on them, and I have to take them home and copy the DCP to one of my CRU drives in order to ingest. And more often than not I find, as Carsten mentions, that these distribution drives have GPT partition tables, multiple partitions, HFS+ or NTFS content partitions ... in short, all the things that are specifically not allowed under the ISDCF recommendations and thus server manufacturers aren't obligated to support. It's pretty clear that these outfits, who call themselves professional movie distributors, are buying consumer drives from Office Depot (or some such place), and aren't even bothering to re-write the partition table and content partition per ISDCF recommendations before writing the DCP on them. They're just taking them out of the box and putting the DCP onto them, leaving EFI partitions, "rescue" partitions, Windows crapware and what have you still on them as well.

Why do these idiots believe that they know better than the industry standard-setting body for DCP distribution media? Saving $150 on a CRU kit and buying a cheap consumer drive instead is well worth a few lost shows (or shows that have to use the backup BD, if one is supplied), because the drive won't work at a given venue and the projectionist there doesn't have the IT skills to copy the DCP from the non-functional (when connected to a DCP server) drive, isn't it?!

Posted by Leo Enticknap (Member # 534) on 07-19-2016, 10:13 PM:

Further to the above post, I popped in to Office Depot this morning to buy a replacement for a failing network switch (panic phone call on Sunday: DSS200 can't connect to cat745), and, while browsing, saw that they were selling 256GB USB3 flash sticks for $50. That's big enough to hold a typical 4K, two-hour feature. If USB flash memory has now become this cheap, I can't understand why anyone would want to distribute their DCP on a spinning rust drive that needs USB bus power, given how problematic we all know that they are in the field.

Posted by Randy Stankey (Member # 64) on 07-19-2016, 10:40 PM:

quote: Ian Freer
I'm not sure what the Mac solution would have been, if anybody wants to put up a solution?
I don't think using HFS+ would be universally reliable for linux based cinema servers, and Mac cannot write to NTFS can it?

Paragon Software's ExtFS for Mac: https://www.paragon-software.com/home/extfs-mac/

Does, pretty much what it says on the box but performance can be slower than normal and it can also take a while for volumes to mount and dismount. However, it does what I need it to do, when I need it done so I don't complain much.

It does come with a free ten-day trial period. [Wink]

Do remember... When you format a disk for a DCP system, it has to be Ext formatted AND it has to be set as Master Boot Record or else the drive won't mount.

Posted by Steve Guttag (Member # 268) on 07-20-2016, 03:48 AM:

Leo, did you fill out a case number on Dolby Customer for that CAT745 disconnection? More than likely, you needed to reboot the DSS200 to get it to connect again.

Posted by Carsten Kurz (Member # 5396) on 07-20-2016, 06:01 AM:

quote: Leo Enticknap
If USB flash memory has now become this cheap, I can't understand why anyone would want to distribute their DCP on a spinning rust drive that needs USB bus power, given how problematic we all know that they are in the field.

It's not practical yet. 256 is not big enough for an inventory, because many features come in many versions and need 500-1000GB disc sizes. Plus, these sticks are incredibly slow to write, which is a nogo in mass duplication. Slow read for ingest is another issue, of course. For the indy filmmaker, it may become an option.

- Carsten

Posted by Stephen Furley (Member # 1277) on 07-20-2016, 07:48 AM:

quote: Leo Enticknap
saw that they were selling 256GB USB3 flash sticks for $50

I wonder what sort of price you could get if you were to buy, say, a thousand of them? The CRU inside a rather flimsy plastic box inside either an even more flimsy cardboard box or a 35 mm fibre transit case is starting to look rather silly, and the courier service isn't cheap.

Posted by Leo Enticknap (Member # 534) on 07-20-2016, 10:17 AM:

quote: Steve Guttag
Leo, did you fill out a case number on Dolby Customer for that CAT745 disconnection? More than likely, you needed to reboot the DSS200 to get it to connect again.

I didn't open a support case, because it was very obvious what the problem was: other devices that communicated through the same switch started to play up, too. With the switch replaced, the problem disappeared. No server reboot was needed, but a projector reboot was.

quote: Carsten Kurz
256 is not big enough for an inventory, because many features come in many versions and need 500-1000GB disc sizes. Plus, these sticks are incredibly slow to write, which is a nogo in mass duplication. Slow read for ingest is another issue, of course. For the indy filmmaker, it may become an option.

I have a SATA CRU bay in my main home PC: writing to that and to a USB3 flash stick happens at about the same speed. Older USB2 sticks are significantly slower to write than a CRU drive (in particular, the cheap Sandisk 128GB sticks only write at around 5mbps, and therefore take 4-5 hours to write the entire volume), but the newer ones are just as fast.

In any case, I wasn't proposing them as a replacement for CRU drives for Deluxe/Technicolor-scale distribution, but as a replacement for the consumer, USB-powered hard drives that a lot of smaller distributors use (and which suffer from slow read for ingest anyways). At least 90% of these that I receive contain a DCP that is smaller than 256GB, and probably around 70% are smaller than 128.

Posted by Scott Norwood (Member # 30) on 07-20-2016, 10:34 AM:

Agreed with Leo--those tiny USB-only hard disks and their easy-to-lose standard/mini/micro USB cables need to go.

That said, I don't really like the USB flash drives much more. Odd as it sounds, they tend to be too small and too easy to lose, and also offer little or no space on the device itself to identify its contents (most have a hook or loop which can be used to attach a cardboard "price tag" with a piece of string, but few people/companies do that).

I really don't see what is so wrong with CRU drives that causes independent producers and distributors to feel the need to constantly re-invent the wheel with respect to DCP distribution. No one has any right to complain about cost for feature-length material, since the alternative would be to spend $1k+ on a 35mm print.

Posted by Stephen Furley (Member # 1277) on 07-20-2016, 12:01 PM:

A few months ago I received a CRU which was very much lighter than usual. On examination it contained a 2.5" drive, but I don't know if it was a laptop or server type.

Posted by Carsten Kurz (Member # 5396) on 07-20-2016, 04:23 PM:

Yes, these appear more often now. They actually need to custom drill special mounting holes into the CRU carrier to fit the smaller drive's mounting holes. Looks a bit D.I.Y.

Two years ago I thought that we would see SSDs for mass distribution sooner or later for their speed and robustness. Now I think we're stuck with CRUs and spinning discs until most cinemas will receive their content diskless. The prices for SSDs won't come down quickly enough.
Duplicators would love them for their write-speed.

We never had problems with those cheap external 2.5" drives (our Sony has USB 3.0 and this offers 1A USB current). However, we had lot's of problems with the CRU DCmini gear. Wether these were related to the drives or the enclosure/electronics, I don't know - I'm glad they are only used by a single distribution company here in germany. Unfortunately, not the smallest, though. Just received two of them, and one caused trouble again, the other I haven't tested yet.

- Carsten

Posted by Leo Enticknap (Member # 534) on 07-20-2016, 04:51 PM:

quote: Carsten Kurz
Two years ago I thought that we would see SSDs for mass distribution sooner or later for their speed and robustness. Now I think we're stuck with CRUs and spinning discs until most cinemas will receive their content diskless. The prices for SSDs won't come down quickly enough.

Interesting that the price of USB flash sticks seems to be in freefall (or, more accurately, the capacity that a given price will buy you is increasing rapidly), but not SATA SSDs. Maybe there are some serious patent royalties on the SATA interfaces and/or firmwares that won't go away anytime soon.

Posted by John Wilson (Member # 269) on 07-25-2016, 11:41 PM:

Just updating...

Managed to import the content over to DCP-O-Matic and remade the files in there. All good and thanks all for your suggestions / help.

JW

Posted by Jim Cassedy (Member # 4115) on 08-04-2016, 07:36 PM:

quote: Ian Freer
Mac cannot write to NTFS can it?

No, natively MAC's cannot write to NTFS.

However, I've run into 'issues' mostly with festival stuff where I really needed
the capability to format, write or modify an NTFS drive or usb stick on a MAC
that I found a solution called "Tuxera NTFS for MAC"

It doesn't take up a lot of disk space, and works well. I did a lot of testing
with the "free" version and had no interoperability or compatability issues
on any of the server systems that I've tried using the 'MAC fomatted"
NTFS drives on.

There's a fully functional "free trial' version that you can download and
try/test for 15 days. I liked it so much I bought the licensed version.
It wasn't expensive- around $30US if I recall, and worth every penny.

Best place to checkitout is on the manufacturer's website linked below:
TUXERA NTFS FOR MAC

Posted by Carsten Kurz (Member # 5396) on 08-05-2016, 05:48 AM:

There are a few options to support NTFS writing in OS X, but the commercial/pro version of Tuxera is the most solid and also fastest in my opinion. It actually comes for free when you buy certain Toshiba external 2.5" USB drives. After installation, it works for other manufacturers drives/USB sticks as well.

- Carsten

Posted by Leo Enticknap (Member # 534) on 08-06-2016, 03:13 PM:

A related question: can Macs read a drive formatted with an MBR partition table and an ext2 or ext3 content partition without add-on software? Thanks in advance.

Posted by Stephen Furley (Member # 1277) on 08-06-2016, 03:40 PM:

Leo, rather surprisingly since Mac OSX is Linux-based, no they cannot. I have a Linux virtual machine on one of my laptops and can read them on that, and copy files to or from a shared folder. Unfortunately, the version of Linux which I have cannot format a disk as Ext 2 or 3, only as Ext 4, so I have to format on another machine, but can then read and write them on the laptop.

Posted by Carsten Kurz (Member # 5396) on 08-06-2016, 04:39 PM:

Only with the free OSXFuse/ext2 driver or Paragon extfs, but not natively.

Actually OSX is not Linux based - Linux didn't even exist when NeXTStep was forked from BSD and was later transformed into Openstep, Darwin, Rhapsody and OSX.

Still I consider it very strange that Apple never built-in native ext support.

OSXFuse/fuse-ext2 allows to format ext-volumes (MBR or GUID) through the standard OS X disk utility dialog. However, as most modern implementations, it will create partitions with inode sizes of 256, not 128 as mandated by ISDCF. They will still work with most servers, but may cause ingest problems on windows based TMS systems (because some of them may use outdated ext-drivers that only support inode size=128).

- Carsten

Posted by Leo Enticknap (Member # 534) on 08-06-2016, 07:39 PM:

Thanks folks. The reason I ask is that I'm making some DCPs for a forthcoming show, and providing copies to the programming partner we're working with at the end of it. As she's a Mac user and I've never owned one, I didn't know if she'd be able to read the content on a DCP drive formatted per ISDCF. As I now know that the answer is no, I'll format it NTFS (which the DSS200 can read), which I know that MacOS can read natively (though not write).

The way I create inode 128 partitions is by "mke2fs -t extX -I 128 -L DiskLabel /dev/sdXY" within a Linux terminal window - it's always worked for me.

Posted by Carsten Kurz (Member # 5396) on 08-07-2016, 09:12 AM:

While I would never advise to ignore the ISDCF hard disk recommendations for mass duplication/distribution, coming from a windows based DCP creation tool, NTFS is the safest way into a DCI ingest process as a matter of fact, across all servers and TMS's.

It's useless to setup a VM, install ext drivers, format ext drives etc. if you simply want to transport 'your own' DCPs to a DCI system. As long as ext2/3 is not supported natively in windows, it creates too many possible problems in real life.

If you create/store your DCPs in a Linux environment, using ext2/3 is a no-brainer.

- Carsten

Posted by Jim Cassedy (Member # 4115) on 08-07-2016, 12:43 PM:

quote: Carsten Kurz
Tuxera is the most solid and also fastest in my opinion.
It actually comes for free when you buy certain Toshiba
external 2.5" USB drives.

Lol! Unfortunately, I found this out the hard way. I had two or 3
brand new, spare, Toshiba drives sitting here that I had bought
when they were on sale for future use in case I needed them.

I didn't realize they came with a free, fully authorized version of
the "Tuxera For Mac" program on them until a couple of weeks after
I had bought the 'paid' version online when I unboxed one of the
drives to use it for a project. D'oh! [Eek!]

Oh well; "ya win some & you lose some..."

Posted by Leo Enticknap (Member # 534) on 08-07-2016, 04:09 PM:

quote: Carsten Kurz
coming from a windows based DCP creation tool... [...] If you create/store your DCPs in a Linux environment, using ext2/3 is a no-brainer.

In this case, I create my DCPs in a Linux environment, but need to provide copies to someone who uses a Mac for long-term storage. Therefore, I'm thinking that NTFS is the best best, because the Linux-based OS I use (Ubuntu) is able to write to it, and MacOS is able to read it, and most DCP servers out there can read it as well.

My other option would be to go the HFS+ route (using the hfsprogs add-on for Ubuntu), but my impression is that fewer DCP servers can read HFS+ than can read NTFS. Of the two models of server I have significant experience using, one (Dolby DSS200) can read GPT/HFS+, but the other (Doremi DCP2000) cannot.