Film-Tech Forum ARCHIVE: Opening zip files created on a Mac of 4GB or bigger on a Windows or Linux PC

This is topic Opening zip files created on a Mac of 4GB or bigger on a Windows or Linux PC in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=002938

Posted by Leo Enticknap (Member # 534) on 02-01-2017, 07:11 PM:

This appears to be a well-known bug with the zip creator built in to Mac OS: zip files bigger than 4294967296 bytes cannot be opened on anything other than another Mac OS computer. If you Google this, you'll find lots of stuff online about it, but this page summarizes the problem pretty well.

We're being given more and more download links for trailer DCP zips that I can't open because of this, and I was wondering if anyone had stumbled across any fix or workaround (e.g. a third party zip program) that enables the contents of these big Mac zips to be extracted in either Windows or Linux (ideally in Ubuntu, but I'd be open to using another flavor of Linux live boot if that would do the job).

Posted by Stephan Shelley (Member # 8587) on 02-01-2017, 07:31 PM:

Have you tried 7-Zip. I have had luck opening files that winzip will not. Not sure about large Mac ones.

Posted by Leo Enticknap (Member # 534) on 02-01-2017, 07:42 PM:

No luck with 7-Zip.

Posted by Monte L Fullmer (Member # 2797) on 02-01-2017, 07:50 PM:

Love 7-zip. Agree, better than WinZip.

Posted by Frank Cox (Member # 6258) on 02-01-2017, 08:52 PM:

If they are sending you defective archives then I suggest the proper solution would be to tell them to stop doing that.

There are numerous, arguably better, options for creating compressed archives other than zip. I think modern versions of the Apple operating system are based on BSD Unix so tar and bzip2 should be readily available.

Posted by Magnus Eriksson (Member # 6085) on 02-02-2017, 02:07 AM:

I'm not sure it's the same issue but this is what I have experienced lately:
About 5% of downloaded trailers won't extract properly. I'm normally using WinRAR on Win 7, 8, 10 btw. WinRAR throws an error (I can't remember exactly what right now, but it's something about something that is too big).
My workaround then is to double click on the zip archive and open it (WinRAR does that automatically). When I get to the DCP folder I left click and pull the folder out on the desktop. Another form of extraction takes place then with no errors, the now unzipped DCP can be transferred with FTP or USB and ingested without issues.
/M.E.

Posted by Carsten Kurz (Member # 5396) on 02-02-2017, 06:07 AM:

It's a known problem indeed, and all I can suggest for now is to buy a cheap second hand Mac or run OS X in a VM to get a grip onto those files.

There may be other issues around ZIP files larger than 4GB as well, not strictly related to an OS X archive function.

Fortunately, I think I have never seen a trailer of that size. Even the larger ones we download usually end around 3GB or so. I think ATMOS trailers now start to break that barrier.

- Carsten

Posted by Leo Enticknap (Member # 534) on 02-02-2017, 10:41 AM:

It's usually when you get the flat and scope versions in the same zip that they bust the 4GB barrier.

Frank - agreed, and I'm surprised that this is happening as often as it does (about once every 4-6 weeks for me, at a guess). My booths can't be the only ones on the planet that don't have a Mac in them, and so I'm surprised that the distributors of these big zipped trailers aren't experiencing more "I can't open this" complaints and investigating why.

This latest one that's giving me a headache is for an Oscar-nominated movie (A Man Called Ove), so it's not exactly some obscure student documentary about eggplant farming in Backofbeyondistan, destined only to appear at one or two arthouse fests.

Carsten - thanks; I like the idea of Mac OS in a VM. I do have a Mac OS CD, with which I experimented making a Hackintosh partition for one of my PCs a few years ago (it sorta worked, but kept freezing up on me). When I get a couple of hours, I'll experiment with that.

Posted by Carsten Kurz (Member # 5396) on 02-03-2017, 06:29 AM:

I think it's easy to get a hand on a Snow Leopard DVD and install it in a VM. Might be handy for other uses as well. A suitable second hand Mac Mini or Mac Book would come out around 200-300US$ I guess.

I have never seen two trailers packed into one ZIP - makes no sense, especially with FLAT and SCOPE versions, since most cinemas usually show only one of these formats, and shouldn't be forced to download both. Yes, some may have different screen setups in multiple auditoriums and need both formats, but I guess the majority only needs either version.

- Carsten

Posted by Leslie Hartmier (Member # 7053) on 02-03-2017, 05:48 PM:

ZIP files using WinZIP, 7-Zip, xzip, etc have a current maximum size of 16 exabytes, or if you are more comfortable with a better known file size, 16,000,000,000GB/ 16x10^9 GB, and has since June, 2015.

My point in stating this is that if you are having a failure with extraction with the standard ZIP dearchiver in MacOS (which is fair, since the original standard maximum is 4GB (original open source specification: <removed the link, as pkware removed it> )), you can use http://wakaba.c3.cx/s/apps/unarchiver.html to bypass the 4GB limitation.

Just drag it into your Applications, and you can do what you need to.

All that said, the issue is unlikely to be file size, but rather the integrity of the archive.

Hope that helps.

Leslie

Posted by Carsten Kurz (Member # 5396) on 02-03-2017, 08:09 PM:

Leslie - it's the other way round. There are many applications available on the Mac that are able to uncompress >4GB archives. The trouble is, only OS X' own archive utility is able to extract ZIP files >4GB CREATED on a Mac. If you receive such a ZIP on a windows or linux machine, as Leo, you're out of luck.

I was actually hoping that Stuffit Expander could help, as it is/used to be quite common on the Mac, and should be able to deal with 'OS X native' file formats. However, it seems that not even their Mac version is able to uncompress large ZIP files created with the OS X archive tool. So I have no reason to assume the windows version of Stuffit Expander is doing a better job. It's weird, but is does seem that currently there is no other solution than to uncompress these files within OS X, bet it on a Mac or within a VM.

- Carsten

Posted by Randy Stankey (Member # 64) on 02-04-2017, 10:17 AM:

Why would anybody want to compress a DCP?

A DCP contains mostly JPEG2000 files. JPEGs don't compress very much because they are already compressed. Even when they are aggregated in an MXF, they still can't be compressed.

The only reason to ZIP something like that would be for the convenience of being able to download just one complete file instead of several. But, if that's the reason for using ZIP, there are better ways to do it such as packaging as a DMG or an ISO image file.

Yes, I've known about the 4 GB thing but I rarely use ZIP for anything over a few hundred MB. For larger files, I use ISO/DMG.

Posted by Frank Cox (Member # 6258) on 02-04-2017, 12:02 PM:

quote:
Those who do not understand UNIX are condemned to reinvent it, poorly.

That's a relatively famouns quote from a mailing list post by Henry Spencer many years ago.

Tar was created for just exactly the purpose that you're discussing here. And it's available on almost all platforms, as far as I know.

Posted by Carsten Kurz (Member # 5396) on 02-04-2017, 02:46 PM:

All good and well, but not an option when you simply receive such a ZIP and have no way to get back to the originator to tell them to do it 'properly'.

Randy - all trailers distributed over broadband nowadays are ZIPed. But they usually will not grow bigger than 2-3 Gbytes.

- Carsten

Posted by Randy Stankey (Member # 64) on 02-05-2017, 03:18 PM:

Just because everybody is doing it, that still doesn't make it right.

I don't know of any up-to-date computer system, available today, that doesn't have a built-in way to make image files, either ISO, DMG or similar. That is the correct way to package a large DCP for download.

This is a classic case of people doing something the wrong way then complaining when it doesn't work.

Posted by Carsten Kurz (Member # 5396) on 02-05-2017, 05:01 PM:

DMG is an Apple proprietary format. ISO, unlike ZIP, carries no built-in integrity information and would need to be verified against a separate hash file in a separate process with separate tools (typically commandline driven). ZIP is perfectly up to the task. It is just a bad idea to create large ZIP files with Apples OS X context menu archive function, instead of using one of the many available free tools that creates universal compatible ZIP64 files.

Besides that, real world DCPs do actually benefit from compression to a certain extent. It is quite common for the audio MXF files to contain silent PCM tracks, and those do compress very well with standard methods. Even static J2K sequences do occur regularly, e.g. in leaders and titles, and if the block size is optimized, will compress nicely. Even a few percent overall will payout with respect to storage space and transmission effort. That's probably why the major trailer distribution portals use it. One Dolby trailer I just looked up on my notebook is 2.85Gbytes uncompressed, and 1.44GBytes compressed.

I'm a Mac user myself, but the issue is very simple - Apple screwed the system archive function, leaving large ZIP files incompatible with the ZIP64 format.
Would be no problem if they wouldn't give these files the ZIP extension, and treat them as Mac proprietary. Hell, they even managed to pack different versions of ZIP and UNZIP into some of their OS X releases so that you couldn't unarchive >4GB files created on the same machine.

Besides that issue, I have nether ever encountered any issue with ZIP compressed trailers ever. And we do have around 600 or so on our file storage, they all came ZIP compressed.

- Carsten

Posted by Leo Enticknap (Member # 534) on 03-28-2017, 10:22 AM:

The solution I eventually went for was to buy a used Mac Pro tower of 2009 vintage for $70 on Ebay. Listed as won't boot - for spares or repair. Turned out that a "click of death" hard drive was the only thing wrong with it. Another $15 got me a Mac OS 10.7 flash stick (the latest version that this model can run, according to various places online), and it now works fine: it unzipped the last trailer zip I had trouble with (A Man Called Ove) without complaining.

I do occasional bits of work on the side doing digital transfers of old audio recordings (mainly 78rpm phonograph discs and 1/4" tapes) and home movie camcorder tapes (S/VHS, Hi8, Mini DV, etc.), and so another advantage of this is that if a customer asks for the transfers on a drive that (s)he can read on a Mac, I can now connect it to an actual Mac and be 100% certain of that. Previously, I'd created HFS+ volumes in Ubuntu with hfsprogs, and had never received any "I can't read this" complaints: but it's nice to have that extra assurance.

Posted by Mattias Mattsson (Member # 4308) on 03-28-2017, 05:21 PM:

It seems that maybe support for these files are being added to 7-zip:
https://sourceforge.net/p/sevenzip/bugs/2038/#9f4f

There are also links to some blog posts there for those who want to read up on the details of this issue.

Posted by Carsten Kurz (Member # 5396) on 03-29-2017, 05:18 AM:

I find it interesting that the 7-ZIP guy never heard about this issue before. That OS X ZIP anomaly exists for years. Recently I gave a workshop on DCP-o-matic and that issue came up as well.

Good to know they may be working on a solution in 7-ZIP.

Better if Apple would finally acknowledge their flawed implementation.

- Carsten

Posted by Jeremie Dunand (Member # 9938) on 04-07-2017, 09:36 AM:

Hello, I have also seen this problem many times.
The solution I found was using an old MacBook that I was no longer using (a 2006 model). It is plugged into my LAN and when I encounter a problem with a compressed file I transfer it to it, I uncompress it from the mac and I retrieve it via LAN on Windows or Linux.
It's not the fastest, but it works!
I will follow the work of 7zip regarding this problem.
Jérémie

Posted by Leo Enticknap (Member # 534) on 04-10-2017, 11:38 AM:

Looks like we're all agreeing on the same solution: Hackintoshing and VMs involve too much time and effort, especially if you don't need or want a Mac for anything else, and given the street price of used, older ones.

Posted by George Germisashvili (Member # 6597) on 04-10-2017, 03:00 PM:

Hi! Try win-rar http://www.rarlab.com/download.htm
Will test it tomorrow, make zip on Mac 5gb size and use winrar on win 7.
(now trying to do this on Mac with VMvare virtual machine with installed win7 on it)

Posted by Ioannis Syrogiannis (Member # 3235) on 04-17-2017, 12:53 PM:

Just my two cents here:
If someone doesn't have a DVD (or USB, nowadays) for installing a Mac OS X version handy and time is pressing (no ebay solutions), I would suggest that a vagrant box* could provide a (fairly) working-out-of-the-box virtual machine.
A living and kicking mac would be failure-proof. No doubt. But there aren't always many choices to pick from.

*vagrant is a cross-platform tool, made for managing virtual machine configurations and environments in a more controlled and global way that can import/export such environments in a "vagrant box", which are easier to be shared.
A number of such vagrant boxes are freely available in community supported repositories and easy to be found, if looked for.

(P.S. I won't start with the reasons a non-Windows PC wouldn't solely fit in a projection booth.)

Posted by Mattias Mattsson (Member # 4308) on 05-03-2017, 02:18 PM:

7-Zip version 17.00 beta was released just a few days ago. I have done a few tests and can confirm that it can successfully extract large zip files (>4GiB) created by macOS Finder.

Download it here: http://www.7-zip.org/download.html

Posted by Carsten Kurz (Member # 5396) on 05-03-2017, 04:09 PM:

Wow! Great times...

Thanx for sharing!

- Carsten

Posted by Mattias Mattsson (Member # 4308) on 08-29-2017, 06:15 AM:

7-Zip v17.01 beta was released yesterday:
https://sourceforge.net/p/sevenzip/discussion/45797/thread/de8085ea/