Film-Tech Forum ARCHIVE: DCP-o-Matic: Problem creating 5.1 VF

This is topic DCP-o-Matic: Problem creating 5.1 VF in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=002702

Posted by Leo Enticknap (Member # 534) on 05-27-2016, 11:59 AM:

I am creating a DCP that will play in two theaters: we can do ProLogic matrixing from a 2.0 TV mix in one (using a DMA8+), but not the other. The source file is 2.0. I've made an OV DCP from the source file, and am now trying to make a VF containing audio that uses the "upmix to 5.1" feature, to play in the non-ProLogic house.

After rendering the OV, I start a new project, import the OV DCP as a folder, and check "refer to existing DCP" in the image tab on the content side, but not the audio. On the DCP side I change the processor option in the audio tab to 5.1 upmix A, and then, without changing anything else, tell it to render the DCP.

It then says "Encoding image data" (which is odd, because it shouldn't have to touch the image data - I'm just trying to add new audio to play with an existing MXF video file), creates a PCM MXF file in the DCP project folder, which grows to a certain size, and then DCP-o-Matic hangs forever (still saying "encoding image data"). I've just tried it with an 87-minute DCP, and it's now been hung two days, and has not tried to write anything to the MXF file for well over a day (going by the "last modified" stamp in the file properties). It only took 13 hours to render the OV. I've also tried this with two other (much shorter) DCPs and got the same result, so I'm not dealing with a specific issue with this source file or DCP.

I'm using version 2.7.4 under Ubuntu 14.04.4 LTS, 64-bit. Am I doing something stupid, or have I hit a bug with DCP-o-Matic? Thanks in advance.

Posted by Carsten Kurz (Member # 5396) on 05-27-2016, 01:06 PM:

MXF handling has been improved in recent versions, and VF is still fairly new. Try again with the current 2.8.4 release.

You are right that it should have worked the way you describe it.

You can use the trim feature to test small segments of the footage so you don't have to wait so long for it to complete.

- Carsten

Posted by Manny Knowles (Member # 1171) on 05-28-2016, 12:58 PM:

Definitely test out that up-mix in its entirety before releasing that DCP. I got unacceptable results using the up-mix feature. Rampant steering of sibilants into the surrounds.

This problem was not observed when the 2.0 was decoded by our DMA8plus, so I'm not inclined to blame the content.

Not sure what (Windows) version of DCP-o-Matic I was using, but it was within the last 6-8 months.

Posted by Leo Enticknap (Member # 534) on 05-28-2016, 01:33 PM:

Agreed completely - it's not as good as what a DMA8+ or the matrix processor in the AP20 can do, the main issue being, as you say, higher dialogue frequencies bleeding into the surrounds. However, in the house I want to play this DCP in, it's better than the echo you get from playing a 2.0 TV mix as is, especially for someone sitting significantly off-center. And the movie I'm dealing with has very little dialogue, so for this particular title it should be OK.

Haven't gotten around to updating DCP-o-Matic on the machine I use for DCP rendering (it's not normally connected to the Internet), but am hoping to in the next day or two - thanks to Carsten for the suggestion.

Posted by Carsten Kurz (Member # 5396) on 05-29-2016, 06:29 AM:

I love the DTS Neo:6 Upmixer that is available in the AP20 for stereo sources. Unfortunately, it doesn't seem to be available as a software solution for common computer platforms.

You may need to test DCP-o-matics different upmixing functions, they are still experimental and currently Carl does not really endorse them as the best available, but is collecting user feedback. The main function is still to create a center dialog mix from stereo sources.

- Carsten

Posted by Leslie Hartmier (Member # 7053) on 05-31-2016, 01:53 PM:

As Carsten mentioned, if possible, you should upgrade to 2.8.0 or 2.8.4 - there are a myriad of improved functions.

Posted by Leo Enticknap (Member # 534) on 05-31-2016, 06:02 PM:

I've done that, am now on 2.8.0, and then tried creating a 5.1 VF from a 90-second trailer as a test, before starting the 87-minute feature again.

It created the trailer VF successfully in under a minute, which I've tested and played OK on a DCP server (ingested OV and VF, then played the VF).

This is where things get curious.

I then started the 87-minute feature rendering again. It began rendering at 0003hrs on Monday morning, and is still saying "Encoding image data" as I write, just under 40 hours later. However, it has been writing to the audio MXF file in the project folder continuously, throughout that time. The audio file in the 2.0 OV is 1.5GB. As the VF will have six channels rather than two, then common sense suggests that the 5.1 audio file should be around 4.5 GB. It's been growing slowly but steadily since the render started, and is now at 2.0GB. At this rate, the process should finish some time on Friday.

What I can't understand is why rendering the OV took well under a day for both image and sound, yet creating another audio track looks like taking around 5 days. Furthermore, given the time it took to create the upmix for the trailer, it should have finished with this feature in under an hour.

Posted by Carsten Kurz (Member # 5396) on 05-31-2016, 07:12 PM:

Hi Leo:

---
http://dcpomatic.com/release-notes.php?v=2.8.6

This release has the following changes:

Updated fr_FR translation from Thierry Journet.
Updated uk_UA and ru_RU translations from Igor Voytovich.
Updated nl_NL translation from Rob van Nieuwkerk.
Support for a wider range of audio file types including MP3.
Hopefully fix strange colour fringing on subtitles when burning them into existing DCP content (#752).
Reduce CPU usage for the preview.
Fix incorrect date when using copy-as-name (#869).
Add free-text notes field to cinemas and screens.
Request confirmation before resetting preferences (#867).
Use CPL title for KDM AnnotationTexts.
Keep audio plot and video waveform dialogues always on top (#756, #820).
Add Cancel button to custom colour conversion dialogue (#880).
Provide option to abort more operations that require films to be saved (#847).
Various fixes to text subtitles muxed into video files.
Add hint on high audio levels (#822).
Use up-to-date version of FFmpeg.
Fix progress updates when making a VF using new audio.
Change some uses of the word ‘frame’ to ‘sample’ when talking about audio (#814).
Add speculative Rec 1886/2020 presets.
Disallow referencing of Interop DCPs in SMPTE films, and vice versa (#804).
Fix missing words in properties windows (#874).
---

- Carsten

Posted by Leo Enticknap (Member # 534) on 05-31-2016, 07:45 PM:

quote: Carsten Kurz
Fix progress updates when making a VF using new audio.

Hmm ... I'm on 3.8.0 (the latest stable version available for download). 41 hours after starting the DCP creation process, it's still saying "Encoding image data: 0%," but the size of the audio PCM MXF file it's created so far suggests that it's in fact around 40% done. I'm reluctant to install 4.8.6 right now, because it would mean cancelling this process and losing almost two days of rendering time. Assuming that it does manage to conclude the creation of this VF successfully, I'll update to 3.8.6, and see if anything changes trying another DCP of a similar length.

Posted by Carsten Kurz (Member # 5396) on 06-01-2016, 06:34 AM:

Normally, creating a VF from a previous encoded OV should only take a couple of minutes. I have no idea what is going on on your system. If you have enough time to wait, then wait. If not, cancel it and restart with 2.8.6

You should also send your logs to Carl, but it sounds as if you are suffering from the bug mentioned in the 2.8.6 release notes.

The headline above the DCP-o-matic test-release notes suggest that these are not stable versions. As a matter of fact, they do contain bug fixes for the release version just as many as new experimental features. Basically, the risc for a release version to contain an issue seems to be on par with most test versions. At least you should always check the changelog to see wether there is something that concerns you.

Releaseversion does not mean it doesn't contain severe bugs. They may just not have been reported before.

I do admit it may be easier to switch between versions on a connected machine. On a windows machine, it's matter of a minute or so to install a previous or next version.

- Carsten

Posted by Carl Hetherington (Member # 7107) on 06-01-2016, 08:47 AM:

quote: Leo Enticknap
It then says "Encoding image data" (which is odd, because it shouldn't have to touch the image data - I'm just trying to add new audio to play with an existing MXF video file)

Fixed in the current test version, thanks.

The extreme slowness is very odd. As Carsten says, this operation should be very quick. I will create a big DCP now and see if I can reproduce the problem.

The last few hundred lines of the "log" file in your project might be informative (carl@dcpomatic.com)

Posted by Leo Enticknap (Member # 534) on 06-01-2016, 03:36 PM:

Many thanks, Carl.

The MXF file only grew by 200MB overnight (from 2.0 gigs to 2.2). I'm going to give it until the end of the week and then give up on this version, and will email you the log files whatever the outcome, then update it to 2.8.6.

Posted by Carsten Kurz (Member # 5396) on 06-01-2016, 04:05 PM:

Leo, you could capture the current log file just as well and send it to Carl.

- Carsten

Posted by Carl Hetherington (Member # 7107) on 06-01-2016, 06:42 PM:

Having looked at it I think it's just a bad inefficiency in reading of existing DCP audio. I wouldn't bother with 2.8.6 as I think it will be just as bad. 2.8.7 should fix it in the next day or so.

Posted by Carl Hetherington (Member # 7107) on 06-02-2016, 10:30 AM:

I believe things should go a lot faster with 2.8.7, if you want to try that:

Test download page

Posted by Leo Enticknap (Member # 534) on 06-02-2016, 12:59 PM:

Many thanks - downloaded and installed 2.8.7, and it's now 14% "encoding picture and sound" through the VF render after about an hour and a half. If it keeps going at the same rate, it should take 9-10 hours total and be done when I get back from work tonight. Will update then!

I also emailed you the log from the effectively stalled 2.8.0 render.

Posted by Carsten Kurz (Member # 5396) on 06-02-2016, 02:42 PM:

That still sounds too slow for me. But let it go...

- Carsten

Posted by Carl Hetherington (Member # 7107) on 06-02-2016, 03:10 PM:

It is rather slow, but at least now it doesn't get any slower as it goes on.

The slow bits now, I believe, are the filters for the upmixer. I need to look at those.

Posted by Carsten Kurz (Member # 5396) on 06-02-2016, 03:44 PM:

Ah, yes the upmixer. Haven't used it on a full length feature yet.

- Carsten

Posted by Leo Enticknap (Member # 534) on 06-03-2016, 10:29 AM:

Although the progress reporting is now a lot more informative, I'm afraid that the basic problem still exists. When I got home last night it was on 32%, and it's only on 33 now (seven hours later).

Something is happening whereby the rate of audio processing decreases as it gets deeper into a longer DCP. I'm pretty convinced that the same thing is actually happening as when I tried doing this with 2.8.0: it completes the first 15-20% of the process relatively quickly, and then slows to a crawl.

Posted by Carl Hetherington (Member # 7107) on 06-03-2016, 10:46 AM:

Hmm. Could you re-send the log from this new run?

Posted by Carsten Kurz (Member # 5396) on 06-03-2016, 11:48 AM:

Maybe try without the up-mixer?

Not that it really matters here - but what type of machine (CPU/Speed) is this? Could you check memory/swap usage during the conversion?

Carl - would there be special log options to activate in order to facilitate analysis?

- Carsten

Posted by Leo Enticknap (Member # 534) on 06-03-2016, 12:51 PM:

New log file emailed to Carl.

I don't think it's a hardware issue, because I've just tried the same thing on another computer running the same operating system, and once again I'm at 14% after around an hour and a half. For the record, the computer I've been using so far is an Acer Aspire V laptop with a Core i5 processor, RAM upgraded to 8GB, with the hard drive nuked, UEFI disabled and Ubuntu 14.04 LTS (64-bit) installed. The keyboard and touch pad are so sh!tty that it's useless for any interactive work, so I use it almost exclusively for long copying jobs and DCP rendering.

It's definitely an audio processing issue that occurs when:

1 - You're creating a VF that "refers to existing DCP" for the pix but not the sound. There is no problem doing this the other way round, because I've created countless scope VFs from flat OVs (e.g. hour long slide loops for walkins) without this slowing down issue.

2 - You're trying to modify the audio in some way, but I don't think it relates to the upmixer specifically and exclusively.

While I was recreating the issue on my main PC, I also started another render going on the laptop, but changing audio settings only in the VF. This time I deselected the left channel altogether and moved right to center (to simulate the correction of a 2.0 mono file to a 1.0 on center channel mono DCP, something it would be very useful to be able to do). This was on a 22-minute file. This time it got to 45% (ish) pretty quickly, and then slowed right down again. So unless I'm missing something, the problem has to be something to do with the way it's creating new audio to sync to existing video.

Posted by Carsten Kurz (Member # 5396) on 06-03-2016, 02:59 PM:

I didn't suspect a hardware problem, but I wanted to get an idea about the general performance of your machine and wether the slowdown could be caused by a memory leak leading to excessive swapping.

- Carsten

Posted by Leo Enticknap (Member # 534) on 07-04-2016, 12:33 PM:

Very happy to report that this bug is fixed in version 2.8.16 - it created the audio VF for the 87-minute feature in 13 minutes. Very many thanks for Carl for fixing this.

Posted by Buck Wilson (Member # 5885) on 07-04-2016, 05:22 PM:

That's fantastic. What great service.

Carl, what drove you to create this great program? And why free?

Bless you!

Posted by Carl Hetherington (Member # 7107) on 07-04-2016, 05:44 PM:

Hey Buck,

Initially I made it because I was a projectionist and we were getting a lot of stuff on DVD/MP4/whatever that "just" needing running in the middle of a set of DCP trailers, or something. Work were never going to spring for CineAsset or whatever so I looked at OpenDCP and tried to make a one-click version of it. Initially the idea was to integrate DVD ripping so you could put your DVD in, click go and get a DCP out! The first versions were basically a set of command-line tools (opendcp and FFmpeg mainly) gaffer-taped together with a quite crappy front end.

It started being useful at work so I released it, and it's amazing what a few emails from interested folks will do for one's motivation.

I left my projectionist job when available hours started fading away (with full automation of digital shows and decreasing 35mm) but by then people were showing more interest in DCP-o-matic so I kept on with it. And here we are... It's a shame; the projectionist job is still my favourite [Smile]

DCP-o-matic is free because I think free-as-in-speech is the only sensible way to make and release software, so that's ideological I suppose. The fact that few have figured out a way to make a living out of it is an implementation detail [Wink]

Cheers, Carl