Film-Tech Forum ARCHIVE: Subtitle Engine not processing 'C', 'K', 'Q', 'J', or 'F'

This is topic Subtitle Engine not processing 'C', 'K', 'Q', 'J', or 'F' in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=003595

Posted by Tom Lewin (Member # 6367) on 02-15-2019, 08:02 PM:

I've got an odd issue that I'm trying to pin down.

House projector is an NEC NC3200S with an IMS 1000.
NP-90MS01
Software ver 2.6.7-0
Firmware ver 4.6.6-0

We were sent a DCP for a special documentary screening that contained closed captions. For the actual screening we'll be bringing in a CaptiView, but client asked to test closed captions ahead of time, so I enabled Doremi's Subtitle Engine and have it set to process closed caption data if no subtitle content is detected. Lo and behold, it's not processing 'C', 'K', 'Q', 'J', or 'F' characters.

So for instance a caption should say, "IT HAD BECOME MORE THAN JUST A QUICK FIX" but is displayed as "IT HA EOME MORE THAN UST A UI IX."

Previously the server was a few revisions out of date, so Dolby just supplied me with the up-to-date versions above, but the issue is still there.

The house that made the DCP said they ran it with a CaptiView system in another venue and had no issues. They've also supplied an open caption version as a backup and informed me that the XML files used in the creation of both is identical, so the issue is likely with the server.

Does anyone have any ideas as to what else I can try to generate these captions correctly?

Posted by Steve Guttag (Member # 268) on 02-16-2019, 02:05 AM:

Make sure that your Captiview software is up to date. It has more effect than the server software.

Posted by Marcel Birgelen (Member # 6801) on 02-16-2019, 05:13 AM:

Maybe it's a font issue in the DCP. The CaptiView system doesn't use the font, so it could work fine with that system, yet not display correctly when rendered on screen.

Posted by Harold Hallikainen (Member # 5405) on 02-16-2019, 06:21 AM:

I think Marcel is on to something! If indeed it is a bad font file in the DCP, it would behave similarly on other systems. Can the Doremi be set to use an internal font?

Harold

Posted by Carsten Kurz (Member # 5396) on 02-16-2019, 06:36 AM:

Is this an interop DCP?

If yes, XML and font can easily be inspected. If it can be traced back to the font, using an actual captiview should be safe, as they use their own internal font. But one should make sure the text encoding is correct.
I don't know what happens when you leave out the font file. Some servers may then use a default internal font, but another outcome could be that the DCP is not ingested properly. One could change the font reference in the DCP by hand, but then some hashes need to be recalculated. The internal caption renderer of the Doremis have no (documented) way to be configured other than caption positioning.

Is this DCP encrypted?

- Carsten

Posted by Ioannis Syrogiannis (Member # 3235) on 02-16-2019, 09:54 AM:

Behind those disappearances, and after those remarks, it seems that the culprit is the child of a font compressor that didn't take into account the closed caption text in all capital.

Posted by Leo Enticknap (Member # 534) on 02-16-2019, 12:43 PM:

Make sure that the projector software/firmware bundle (including the ICP) is up to date, too, as that could affect subtitle rendering. The current version is, IIRC, 4.312, which includes ICP version 4.5.

Posted by Carsten Kurz (Member # 5396) on 02-16-2019, 02:12 PM:

quote: Ioannis Syrogiannis
it seems that the culprit is the child of a font compressor that didn't take into account the closed caption text in all capital.

I thought of that as well, but, why would a FontCompressor run throw out just:

C', 'K', 'Q', 'J', and 'F' ?

Of course, somewhat may have gone wrong during testing a file with Font Compressor and that ill-fated font file found it's way into a new DCP. I usually advise to strictly rename compressed (reduced) fonts to a new unique name expressing it's reduced nature.

- Carsten

Posted by Harold Hallikainen (Member # 5405) on 02-16-2019, 02:42 PM:

On font file naming, I recently ran into a movie where the font file names (URI in LoadFont) were something like 70 characters long on reels 1-4 and 10 or 20 characters long on reels 5 and 6. This caused my timed text file parser to crash since it only allowed XML attributes up to 50 bytes long. I am working on an update that increases the size of the attribute buffer AND truncates the string copied to it should the size be excessive. So, it's always something! I wrote that code in 2010 and in 2019 it breaks. In closed captions, we don't use the font file, but all elements, including LoadFont, are parsed, thus causing the issue.

Harold

Posted by Ioannis Syrogiannis (Member # 3235) on 02-17-2019, 03:24 PM:

quote: Carsten Kurz
...why would a FontCompressor run throw out just: C', 'K', 'Q', 'J', and 'F' ?

If I had to guess, it would be because of the lack of them within the subtitles of a certain reel.
In following the line of such an hypothesis, it could be more than only those capitals that were thrown away, yet those surfaced on the test.
If closed caption used just italic characters, the loss would probably be more significant.

Posted by Randy Stankey (Member # 64) on 02-17-2019, 04:09 PM:

It sounds like a subsetting problem to me.

In some types of files, like graphics, the program only includes the characters of a font that are actually used. It's supposed to save file space.

It seems, to me, that font subsetting was used but the information about which characters are actually used got messed up and characters were thrown out that shouldn't have been.

Possibly, when the font was compressed/packaged, it was set up for only part of the program. One part (or parts) do not actually use those characters but, the rest of the program does. Then, somebody (accidentally) used that packaged font for the whole movie. Thus, when the movie plays and the left-out characters are needed, they don't display as they should.

Basically, when the movie was being packaged, I think somebody checked the "subset fonts" box in the software when they shouldn't have.

In this day and age, I don't know why anybody would use font subsetting, anymore. We have enough storage space. Bandwidth is good enough. The space saved by subsetting fonts would be a fraction of a percent, compared to the size of the entire movie file. (DCP package)

It's freakin' stupid! We're not talking about somebody working on a home or office computer. We're talking about a BUSINESS where things are supposed to work without problems, the first time, every time. In other words, "Mission Critical."

We're paying a LOT of money for movie projection equipment... Hundreds of thousands of dollars... even MILLIONS! It's all supposed to be mission critical operation.

What kind of a dain bramaged fool is going around clicking random options on a computer screen when they shouldn't?

Come to think... Why is the "font subset" option even still avaliable?

Posted by Ioannis Syrogiannis (Member # 3235) on 02-17-2019, 04:29 PM:

I *totally* agree!
Though it might be because of the ICPs and the servers not rendering themselves the subtitles that are still around.

Posted by Leo Enticknap (Member # 534) on 02-17-2019, 04:35 PM:

quote: Randy Stankey
We're paying a LOT of money for movie projection equipment... Hundreds of thousands of dollars... even MILLIONS! It's all supposed to be mission critical operation.

Which is why, in the USA at least, subtitles encoded for rendering in the projector have gained such a poor reputation for reliability that the few foreign language releases that do play widely tend to have their subtitles burned in (e.g. LAS-en as distinct from LAS-EN) now. I think it was the Attack on Titan debacle which provoked that, with dozens, if not hundreds, of screens with Series 1 equipment that hadn't been updated since installation being unable to play the subtitles.

I get the impression that because subtitles are more widely used in Europe, theater operators and techs there are more conscious of the need to get everything right, both in mastering the DCP and in setting up the server and projector.

Posted by Marcel Birgelen (Member # 6801) on 02-17-2019, 04:40 PM:

quote: Carsten Kurz
Of course, somewhat may have gone wrong during testing a file with Font Compressor and that ill-fated font file found it's way into a new DCP. I usually advise to strictly rename compressed (reduced) fonts to a new unique name expressing it's reduced nature.

Maybe someone used a demo-font on the final run which on-purpose misses some glyphs, or it's just a stupid bug. [Smile]

quote: Harold Hallikainen
This caused my timed text file parser to crash since it only allowed XML attributes up to 50 bytes long.

Keep in mind that, depending on the original encoding of the XML, attributes are allowed to be in Unicode. Which often especially breaks stuff when the encoding is UTF-8 and suddenly people start to introduce characters from the Unicode namespace rather than the ASCII namespace only. [Wink]

Posted by Harold Hallikainen (Member # 5405) on 02-17-2019, 04:52 PM:

I don't know if it's still the case, but the TI CineCanvas system only supported font files up to 640 kB. See http://www.deluxecdn.com/dcinema/reference/ti_subtitling_spec_v1_1.pdf#page=13 . Thus the need for "font compression."

Harold

Posted by Randy Stankey (Member # 64) on 02-17-2019, 08:29 PM:

quote: Leo Enticknap
Which is why, in the USA at least, subtitles encoded for rendering in the projector have gained such a poor reputation for reliability that the few foreign language releases that do play widely tend to have their subtitles burned in...

Versus what? Getting somebody who knows what they are doing to do the job?

I'm not saying that everybody or everything has to be perfect every time. Everybody makes mistakes. That is what it is to be human.

The point I'm getting at is that people just don't seem to put forth the effort to do things the best way they know how and they don't seem to care when things go wrong.

This causes other people to have knee-jerk reactions like "burning in" subtitles. Now, we can't turn them off when we don't want them.

The whole point of the subtitling system was to be flexible and easy to use, provided a competent operator is at the controls. But, now that we have INcompentent people upstream of us, we are forced to make changes that hamper our ability to do the job.

These people who don't care about their jobs should be forced to work on an assembly line in a factory, some place, where one little mistake can upset the whole operation, costing thousands and thousands of dollars. It's boring, monotonous, sweaty, back-breaking work, on top of it all.

If those people worked in a place like that for one month, I bet their whole attitude would change.

Unfortunately, I'm not "Dictator-In-Chief of the World" or things would be run differently. [Wink]

Posted by Leo Enticknap (Member # 534) on 02-17-2019, 10:17 PM:

quote: Randy Stankey
This causes other people to have knee-jerk reactions like "burning in" subtitles. Now, we can't turn them off when we don't want them.

You can't turn them off when playing a CPL with projector-rendered subtitles anyways. The rationale behind having them rendered in the projector was to cut down on the volume of data on distribution drives. Instead of having to double the size of the drive to accommodate both a subtitled and an subtitled version, all you need is the unsubtitled OV and a subtitled VF, which will add no more than a couple of megabytes to the overall packages. You can also include multiple language versions, again, at very little space penalty.

I agree with you that if DCP authoring were all done correctly, and projectors and servers were all kept up to date, this problem wouldn't exist. But the reality of the situation is that the whole rendered subtitles workflow/ecosystem is highly intolerant of things not being done strictly by the book, and close attention being paid at every stage in the process (I can't think of a one-word antonym for "idiotproof," but that's what I mean). Agreed completely that people should do their job properly, but we have to face the fact that, as in most other industries, there are some in ours who don't. This is not a new problem: if it were, film prints would never have gotten scratched.

Add to that the fact that the Interop to SMPTE transition broke the ability of systems with older software/firmware to play subtitles, and the end result was that the distributors of the few foreign language movies that play to a wide audience in the US are increasingly deciding to play it safe and use burned-in subtitles.

Posted by Randy Stankey (Member # 64) on 02-17-2019, 11:01 PM:

I get it. I fight it every day at my current job. There are idiots everywhere.

One of my responsibilities is to replenish plating bath chemistries. (Electroplating shop.) The additions that I have to make is based on the amp-hours that a bath is used. Most of the tanks (of chemicals) have amp-hour meters on their rectifiers but, for various reasons, some don't. Because of that, I am dependent on the platers to report the number of amp hours they put on a bath so that I can make the needed additions.

It's like pulling teeth!

It's easy to calculate amp-hours... Number of loads x (minutes ÷ 60) x amps. We're talking eighth grade arithmetic. But, I RARELY get correct reports when I get reports at all.

Then they (the platers) come back to me and complain because their work gets rejected by Q.C. How can I do my job when the guys upstream of me don't do theirs?

Some days, I feel like going postal! [Wink]

On the other hand, the shop where I work does a lot of gold plating. There aren't many shops in this area that plate gold. It's finicky work and, due to the cost of materials, small mistakes can cost the company thousands of dollars.

They don't let just anybody work on the gold plating line. Just because you know how to plate tin and copper doesn't mean you can plate gold and silver. A new plater starts by working with tin and copper. When he gets enough experience, he learns to plate nickel and, when he shows that he knows what he's doing, he moves up to silver and, eventually, gold.

What I'm saying is like a metaphor... The movie industry has a lot of people who are barely qualified to plate tin who are trying to plate gold.

As much as I am an opponent of digital projection, movie presentation in a commercial theater is like unto plating gold.

It has to be done by competent people who can do the job right, the first time, without making too many mistakes.

Back when I was running film, people took their jobs seriously. Yes, there were wankers but they got weeded out fairly quickly. Most of the people who work with me tried to do their jobs as well as they knew how.

Nowadays, just because everything is all digital, people seem to act like things don't matter. They just do "Whatever" and, if it works, that's good but if it doesn't... "Que sera, sera."

As much as I am upset to be out of a job because digital projection replaced real movies, the thing that truly upsets me is that "Que sera, sera" attitude which digital projection has brought about and which seems to be permeating the entire industry.

Posted by Frank Cox (Member # 6258) on 02-18-2019, 12:30 AM:

quote: Randy Stankey
Number of loads x (minutes ÷ 60) x amps. We're talking eighth grade arithmetic. But, I RARELY get correct reports when I get reports at all.

If I was in that position, I would make those folks a program or an app (depending on what computing hardware is available in the immediate vicinity) for logging that. Then they just enter the numbers off of whatever they're using to time this thing and let the machine do the math and the record keeping.

A long time ago in a galaxy far away I wrote something like that (but a bit more elaborate, I suppose) for logging projection figures from an oil drilling rig. The push could put a (dos) computer and a dot matrix printer in the doghouse and everybody was happy, including the government inspectors.

Posted by Harold Hallikainen (Member # 5405) on 02-18-2019, 08:24 AM:

I'd probably buy amp-hour meters for the rectifiers that do not have them.

Harold

Posted by Tom Lewin (Member # 6367) on 02-18-2019, 12:21 PM:

quote: Carsten Kurz
Is this DCP encrypted?

DCP is not encrypted, but I won't be back in the venue until tomorrow to double check whether it's Interop or SMPTE. If it's the former, I'll see if I can recreate the DCP and leave out the font file.

quote: Leo Enticknap
Make sure that the projector software/firmware bundle (including the ICP) is up to date, too, as that could affect subtitle rendering. The current version is, IIRC, 4.312, which includes ICP version 4.5.

Good point, this venue last serviced the projector when it was installed in 2015 (we are contracted by the venue to provide technical services, but we are not in-house projectionists). Does anyone know where I can find the latest projector and ICP bundle? If I can get the files, I'll update tomorrow so that at least both the Doremi and the NEC are up-to-date, and that can be ruled out as a cause.

quote: Randy Stankey
It sounds like a subsetting problem to me.

The media house that packaged the DCP informed me that they had no issues at another venue using a CaptiView. I'll certainly ask them if they chose the option to subset fonts, but would subsetting be consistent with the CaptiView generating its own subtitles correctly using its internal engine, and the Doremi engine not reading the subset font properly? Just checking as this will be the first time I, and the venue, are bringing in a CaptiView (which is coming with its own engineer), so I'm a bit unfamiliar with the particulars.

Posted by Harold Hallikainen (Member # 5405) on 02-18-2019, 12:46 PM:

CaptiView and other closed caption devices use an internal font instead of that delivered with the DCP while projector rendered and server rendered open subtitles or open captions use the font specified in the DCP (using the LoadFont element in the timed text file). So, it's quite possible that some characters are missing from the font file causing open subtitles or captions to not work properly while closed captions or subtitles do work properly.

Harold

Posted by Carsten Kurz (Member # 5396) on 02-18-2019, 02:11 PM:

It would be interesting to look at the font/subtitle file, no matter Interop or SMPTE (SMPTE needs to be unwrapped beforehand).

If you create a simple test subtitle file with ABC in capitals and lower case, and reuse the supplied font for it, one could easily find out wether glyphs are missing from the font file. Some operating systems also allow to doubleclick on a font file for a quick inspection of the charset.

- Carsten

Posted by Randy Stankey (Member # 64) on 02-18-2019, 10:54 PM:

quote:
I'd probably buy amp-hour meters for the rectifiers that do not have the.

That's what I want but, right now, the shop is undergoing a renovation. Four of the tanks share rectifiers. Two tanks share one rectifier, respectively.

If I know how many A/h are on one tank I can subtract the time to get the A/h on the other.

The system works fine as long as the platers report their hours. We even hung clipboards on the wall. All they have to do is go over and write down their hours.

But, all it takes is one or two guys failing to report their hours and the whole thing gets messed up. [Frown]

Posted by Tom Lewin (Member # 6367) on 02-19-2019, 04:17 PM:

I just updated the ICP software/firmware bundle for the NC3200S (4.02/4.05 respectively, Release Prod 4.5) after receiving files from our NEC tech, but the issues are persisting.

I haven't had a chance yet to recreate a DCP, but I've uploaded a new report from the IMS-1000 and also from the NEC to this Dropbox link. I've also included the non-audio/video files of the DCP in question, including the subtitle MXF: https://bit.ly/2GPrKo5

That folder contains every file in the DCP with the exception of the jp2k and wav MXF files.

Should I regenerate the subtitle MXF without the reference to Arial?

Edit: Not that I know that much about generating subtitles, but I just tried to open the subtitle MXF in dcp-o-matic, and received the error

quote:
Could not read subtitles (Document not well-formed. / missing XML tag ID in SubtitleReel)

Posted by Carsten Kurz (Member # 5396) on 02-19-2019, 05:36 PM:

So, it is SMPTE.
DCP-o-matic can read MXF/SMPTE wrapped subs. Maybe there is something wrong with that file, or it has some specifics that make DCP-o-matic fail. Let's see...

- Carsten

Posted by Carsten Kurz (Member # 5396) on 02-19-2019, 07:38 PM:

Alright, the subtitle file uses capitals nearly exclusively, and, the font is indeed crippled.

Why they created that specific font file, no idea. It is not nearly large enough to cause issues (series 1 projectors have a 640KByte limit).

I don't believe that font reduction was based on a real job, too many important capitals missing. Maybe someone played around with font compressor and lost track of the files. Maybe they created the file for a specific short reel with only a few lines not containing these chars, and then reused the file for all reels. Whatever...

But - as we wrote above, this is for a CCAP device, so, the crippled font is not used and the CCAPs will play ok. Interesting to know though that Doremis do use the supplied font when rendering CCAPs on screen as OCAPs.

Thanks to Carl for unscrambling the file. There was a small flaw in it it seems.

- Carsten

Posted by Tom Lewin (Member # 6367) on 02-20-2019, 10:37 AM:

Huuuuge thank you to everyone, especially Carsten and Carl for digging through the files, and to Harold for suspecting the font file. Glad to hear it's not the venue's equipment, and it was good to get everything up to date anyway and learn something along the way.

I'll reach out to the DCP manufacturer to inform them of the error.

Posted by Antti Nayha (Member # 4980) on 03-04-2019, 02:13 AM:

I recently ran into a similar case, where the ICP wasn't showing capital Q's and W's when playing a certain Interop DCP. I looked at the font file in macOS's Font Book and sure enough, the broken characters were missing, just like in Carsten's screenshot above. And a trial version of Glyphs gave some rather cryptic error messages and refused to open the .ttf at all.

However, the author of the DCP sent me a screenshot of the font on his system, and all characters were visible there. So whichever software he was using, the font bug didn't seem to affect it at all. And all hash checks were successful, so it wasn't a case of file corruption during download/ingest.

The quick & dirty solution was to remove the font file and all references to it, and rewrap the DCP, so our ICP simply used its default font.

Posted by Carsten Kurz (Member # 5396) on 03-04-2019, 09:52 AM:

While the reason for that font being crippled may not be TI's font compressor, it is nevertheless missing many glyphs. An online font/glyp display tool shows the same. It indicates only 54 glyphs present in the file. File corruption of course is not impossible, but unlikely, I guess. In modern operating systems, font management is complex and often opaque to the average user. I wouldn't trust a user screenshot as an evidence without knowing anything else. A screenshot may e.g. come from an app using automatic font substitution.

- Carsten

Posted by Marcel Birgelen (Member # 6801) on 03-04-2019, 02:23 PM:

Nice to see that it's solved. The initial hunch somehow prevailed, although it seemingly took quite some digging. [Wink]

Sometimes there are demo fonts available for testing purposes that miss some glyphs on purpose, but in this case, it seems to be Arial, which is one of the most ubiquitously available fonts around.

Maybe somehow, in the DCP creation process, they inadvertently used a previously compressed font for the final output.