Film-Tech Forum ARCHIVE: IMAX DTAC proj. time sound reset problem

This is topic IMAX DTAC proj. time sound reset problem in forum Large Format Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=14;t=000080

Posted by Brian Michael Weidemann (Member # 2243) on 07-21-2004, 07:20 PM:

I've been wanting to pitch this problem for a while, but we kept thinking we were fixing it, and it keeps coming back ... so now I'd welcome some input. My tech has been constantly on the phone with IMAX and they are being most unhelpful.

THE SYMPTOM:
At a random point in the show, the DTAC "projector time" will reset back to 0:00:00 and run from there. The "show time" immediately finds itself too far ahead, drops out, and catches up with the new projector time, which is the very beginning of the trailer. The PLC keeps frame count perfectly and does not reset. But once the DTAC has effectively "started over", there is no way to synch it back to the console or the film.

THE FREQUENCY:
When the problem started occuring, it would happen roughly once or twice a week; random shows, random parts of the movie. It has happened twice in one day (different shows, though) and it has also gone as much as 18 days between instances.

THE CHRONOLOGY OF (ATTEMPTED) REMEDIES:
(months of periodic sound resetting)
The DTAC had its TIC card replaced with a new one.
(it happens again 6-09, and twice on 6-10)
The cable connecting the TIC card to the Sync Timing card was discovered to be bad, and replaced.
(it happens again 6-18)
A new Sync Timing card is installed, with another new cable.
(it happens again 6-28, and again 7-02)
A new "J" Module is replaced in the PLC.
(it happens again 7-20)

THE REALLY UNPROFESSIONAL QUICK-FIX:
Because the show is effectively ruined, passes and refunds are handed out, and management really hates us. So to "save" the show when this happens, this is what I've been doing. It doesn't really work, but at least we can finish the show.

Stop projector, stop DTAC, close DTAC show list, open SCREENING mode, open show list, advance to "feature" region, convert PLC frame count to hr:min:sec:frames, use the DTAC "Jump To..." and enter hr:min:sec:frames. Start projector, when projector is at full speed (about eight seconds) hit the "OK" and theoretically, the sound will start up at the same part of the movie that the projector is running at. (I did the math, and the eight seconds you lose from the restart was gained at the original start, since it goes from 59:52:00) It's not a perfect method, but because the sound is now in no way connected to the projector, you can use the "Jump To..." to fine tune the synchronization. I've got it to within half a second, which was good enough that NOT everyone wanted refunds.

THE GRIPE:
Why has Sonics NEVER included the ability to adjust the synch, outside of reseting frame count at the beginning. Even with the old DDP system, there were times when the ability to retard or advance the sound manually would have saved many, many shows.

STATUS AND FEEDBACK:
Our next step, apparently, is to sever the connection from the PLC to the DTAC completely, to further isolate where the damn problem is coming from. Because of the nature of the symptoms, it's a "Well, we'll just have to WAIT to see IF it happens again; if it does, call me" kind of problem. Which sucks.

So, any information, speculation, general discussion, etc. you'd like to throw out, go for it!

"I'll give you a topic: the PROJECTOR TIME is neither related to the PROJECTOR, nor in TIME with anything ... discuss."

Posted by Robert Stawiarski (Member # 1842) on 07-21-2004, 07:52 PM:

Brian, I wish I could help, but I have not experienced that myself.

Also, only a small handful of posters on here have DTAC experience. Might want to try the 1570.com Technical Issues forum where you will have a much better pool to draw from.

Posted by Gordon McLeod (Member # 33) on 07-21-2004, 09:54 PM:

That is when it is good to have the MagnaTec Dubber running in tandem

Posted by Adam Martin (Member # 641) on 07-21-2004, 10:19 PM:

We battled this exact problem in Sacramento for almost A YEAR. Our problems there were also complicated by random audio looping and skipping at no particular time and for no particular reason.

We replaced the Ginseng with the TIC card, replaced (one-by-one) EVERY card in DTAC, replaced the power supply, replaced the hard drives (twice) and finally stopped deleting soundtracks from the drives and stopped defragging.

Replacing the drives (we had the 80gig drives) the second time and no longer deleting soundtracks or defragging seemed to solve the problem. It never came back.

To overcome the obvious issues of not deleting soundtracks and not defragging, every service required deleting every soundtrack file off of the backup drive, unregistering unneeded soundtracks, rebooting, using windows explorer to copy all of the soundtrack files to the blank backup drive, swapping the drives, and repeating the deletion, reboot, and copying process back to the master drive.

Tim Silver is the Imax tech who had to deal with this and he's based in Los Angeles.

quote: Brian Michael Weidemann
Why has Sonics NEVER included the ability to adjust the synch, outside of reseting frame count at the beginning.

Because they fired everyone who knew anything about how DTAC works when no one wanted to move from Alabama to Toronto when Imax purchased the rest of the company.

Posted by Brian Michael Weidemann (Member # 2243) on 07-21-2004, 10:38 PM:

quote: Gordon McLeod
That is when it is good to have the MagnaTec Dubber running in tandem

I'm all for that! We just don't have one. And I haven't seen a DTAC/dubber combination in any theatre. It seems once the "reliability" of the DTAC showed up, the dubbers seemed unnecessarily superfluous (an intended redundancy).

Adam, I've worked with Tim before. He's been out for at least one service that I can recall. And I can get his number from my tech, too.

We had the audio looping, skipping, and freezing issues a long time ago, and haven't happened in a little over a year. The first time was when the sound got caught in Lion King, a two-second loop from the chorus of Hakuna Matata ... now that would drive ANYONE nuts!

quote: Adam Martin
Replacing the drives (we had the 80gig drives) the second time and no longer deleting soundtracks or defragging seemed to solve the problem. It never came back.

Interesting. The problem doesn't seem to be related to the hard drives at all, since it's never at the same point in the movie. But if new drives the first time didn't work ... hmmm. I get the impression that you guys never figured it out, it just seemed to stop at some point. It's been a while since we've deleted anything or defragged, but it had happened before the last time we did, too. Hmmm ...

Thanks so much for the info, though!

Posted by Adam Martin (Member # 641) on 07-21-2004, 11:05 PM:

quote: Brian Michael Weidemann
it's never at the same point in the movie

Ours wasn't at the same place in the movie every time, either.

I actually replaced the master drive with another drive of my own (later getting yelled at for trying to fix the problem [Razz]

) and still had the skipping, etc. problems. We could go for days without problems and some days we'd have a show go down 8 or 9 times. Sometimes hitting "Stop" on DTAC and re-starting (re-chasing) would correct the problem and other times, DTAC would entirely lock up.

It's my opinion that us not deleting and not defragging solved the problem, and that DTAC has issues with the 80gig drives (because there are several theaters out there having the same type of problems) but Imax seems to think it was the drives. Despite the fact that I used the third-party drive with the same results. [Shrug]

We documented the problems we were having pretty well and I probably have 50 pages in the log book devoted to DTAC problems. Unfortunately, I'm out of town for the next month and can't get to my copy.

Posted by Bob Brown (Member # 1205) on 07-22-2004, 11:23 PM:

So Adam, are you suggesting that I do not delete and defrag the HD in my DTAC unit?. IMAX installed new 80 gig HD's last December. I have, on the instructions of Norm at IMAX, been defraging the drives on a regular basis. I also have been deleting all old registered files. We have NOT experienced the problems you and Brian have been dealing with, at least not yet! Is this more of a GT problem than SR?, or just one more DTAC issue to be on the look out for.

Posted by Daryl C. W. O'Shea (Member # 1303) on 07-22-2004, 11:35 PM:

Unless you're using some lousy defragmenter there's no logical explanation why you'd have problems with a presumedly less fragmented volume.

However, not deleting things (solving the problem) makes sense assuming there is some sort of indexing system -- that is possibly (probably) getting corrupt, or inaccurate, when things are purged.

Posted by Richard Hamilton (Member # 321) on 07-23-2004, 02:17 AM:

You need to replace all of the equipment with Megasystems 8/70 projectors and sound systems;) We never have any sync problems!!!

Rick

Posted by Adam Martin (Member # 641) on 07-25-2004, 12:42 AM:

The hardware in the GT and SR DTAC systems differ a bit since the SR self-generates timecode and the GT requires an outboard converter.

Now that I think about it, the other theaters that I've talked to that are having the problems described above are all GT theaters.

I would say that if you're not having problems, continue whatever you're currently doing.

Posted by Brian Michael Weidemann (Member # 2243) on 07-25-2004, 01:02 AM:

Oh, now really, what good will that advice do?! [Wink]

I say if you're not having problems, do EVERYTHING different that you possibly can, until you DO have problems. That way you know exactly what NOT to do in the future. Simple. [Razz]

As a kind of update to our problem, my tech suggested I go back through the log to pinpoint when our specific reset problem started happening with when they connected the theatre's fire alarm to the E-Shut. (Parenthetical "off the record" comment edited out.) The fire alarm was connected a month before the first reset. The suggestion is that there is crosstalk, or some ungrounded pulse that's sneaking over to the frame count area from the theatre's fire alarm network, since it's been connected ever since the fire inspection.

It sounds fairly unlikely, but we'll see if anything happens. At this point, I'm willing to accept that the lost Mars probe really hit our building and is sending out interference which is causing this. In fact, that could explain a whole slew of random crap ... our auditorium wall lighting going out all the time, for instance. Or the ushers tending not to clean for the last show of the night. Makes sense.

[ 07-26-2004, 08:09 AM: Message edited by: Brian Michael Weidemann ]

Posted by Adam Martin (Member # 641) on 07-25-2004, 10:16 AM:

Our fire alarm was not connected while we were having the problems. Hmmm ... and neither was the network connection, but the phone line might have been.

quote: Gordon McLeod
That is when it is good to have the MagnaTec Dubber running in tandem

Unfortunately, you can't put a 150-minute reel on the dubber. (Unless, maybe, you concoct some sort of reel transport out of a tower or something.)

quote: Daryl C. W. O'Shea
Unless you're using some lousy defragmenter

Diskeeper running under WinNT 4 is what's installed. And there's apparently no way to upgrade the OS and still have the hard/software work. [Roll Eyes]

Posted by Gordon McLeod (Member # 33) on 07-25-2004, 12:21 PM:

"quote: Gordon McLeod

That is when it is good to have the MagnaTec Dubber running in tandem

Unfortunately, you can't put a 150-minute reel on the dubber. (Unless, maybe, you concoct some sort of reel transport out of a tower or something.)"
When Fanasia came out I visited one theatre that had an Eprad Wall Mut beside their dubber for just that purepose

Posted by Daryl C. W. O'Shea (Member # 1303) on 07-25-2004, 09:21 PM:

So long as it's NT4 SP6a it should be a very solid system. Knowing that it's Diskeeper, I would be confident in ruling out defraging as a part of the problem. It's far more likely an indexing problem.

Is the timecode generator well grounded? Is it on a power conditioner, or at least a good, not cheap, UPS?

Is there a way to monitor the timecode coming out of the TC generator before it hits the DTAC?

Posted by Gordon McLeod (Member # 33) on 07-25-2004, 09:47 PM:

Use a DTS and JSK generator off the tach and all the problesm will go away [Smile]

Posted by Stewart Anderson (Member # 1809) on 07-25-2004, 11:41 PM:

I have an extra DTS E-175 that you may buy (cheep) now that we had a new DTS XD10 installed with a new E-480. You probably have Sonics and I don't know anything about how Sonics operates or even if the E-175 is compatible... but if you can use it I'm sure my bosses would love to sell it.

PS. We never had any problems with it.

Posted by Adam Martin (Member # 641) on 07-25-2004, 11:56 PM:

I want to say that it's NT 4 SP 5. I almost guarantee that there will be no OS upgrade from Imax.

Posted by Daryl C. W. O'Shea (Member # 1303) on 07-26-2004, 12:12 AM:

SP5 was released in May of 1999. There's been a lot of fixes between then and SP6a (even a small number of hotfixes after 6a).

As long as the software isn't actually crashing it's probably not an OS issue anyway. Either an indexing issue (if not deleting things totally solves the problem) or a loss of TC from the TC generator (SR doesn't have the problem).

Is the DTAC unit the same for both the SR and GT (is it the SR projector generating the TC or a different DTAC model)?

Posted by Adam Martin (Member # 641) on 07-26-2004, 08:55 AM:

DTAC receives timecode from something ... the SR generates its own timecode; the GT sends pulses to either an outboard timecode generator which then sends tc to DTAC or to a TIC card in the DTAC itself.

In our case, we started the problem with the outboard generator that ran just fine. In fact, DTAC (and the outboard generator) continued showing proper timecode when the errors occurred, but the audio was looping or fell out of sync such that the audio timecode matched the projector timecode but the audio timecode wasn't the piece of audio that was actually playing. Once we changed out the outboard tc generator to the TIC card generator, the problem continued, so it was not a problem with the timecode being received. (Which sucks, because the outboard generator was more flexible in the case of emergencies and with someone who knew how it all worked.)

I should probably clarify that the audio comes from a 6-channel .wav file. Also, it didn't matter whether it was a feature-length film or a regular short Imax film.

I have also confirmed that the OS is Windows NT 4 SP 5 for both GT and SR systems. In fact, the software is exactly the same ... you just configure it to whichever system you're using. There's GT with/without TIC card, dubber, DDP, etc.; SR with/without dubber, etc.; and M-SR, which is a cut-back hardware version for really cheap people.

There were many instances where the software actually appeared to freeze up with no choice but to reboot the system. I say "appeared" because occasionally if we cancelled the show and left it to run through the projector, the audio would start up again 15 or 20 minutes later and complete the show. [Mad]

Once it locked up, I couldn't even get to the task manager to see what was locked up ... CPU, memory usage, etc.

(This is where the outboard generator flexibility comes in ... if you had to reboot DTAC with a TIC card, the card would reset to a zero timecode. If you had an outboard generator, you could reset JUST DTAC and you'd still have your timecode and wouldn't lose the goddamn show.)

DTAC ships with 16 gig drives. Every theater I have talked to that has had this audio looping/timecode problem has received the upgrade to 80 gig drives. Some are on TIC cards, some are outboard generators; some have backup audio sources that come in through DTAC, some are just DTAC.

Posted by Daryl C. W. O'Shea (Member # 1303) on 07-26-2004, 04:23 PM:

Well, if timecode isn't glitching at any point and the software is locking up, somebody should accidentally install Windows NT 4.0 Service Pack 6a, after making a backup clone of the system drive (or just choose to save old files for a possible uninstall), and see if their problems go away.

I assume that NTFS is used?

Could someone leave the task manager open and wait for the DTAC to lock up? Having it set to the processes tab and sorted by CPU usage might show something usefull when the system craps out.

Posted by Adam Martin (Member # 641) on 07-26-2004, 05:46 PM:

I suggested that the OS be upgraded to see if the problem was corrected, but immediately received death threats from the technical people.

I know that each drive was partitioned into two so that the OS and software resided on one side and the data resided on the other side. I think that one of the partitions was NTFS and the other was FAT.

I vote that Daryl should call Imax and offer to fix DTAC. [Big Grin]

Posted by Daryl C. W. O'Shea (Member # 1303) on 07-26-2004, 05:54 PM:

People install software accidentally all the time! There's a good chance there could be issues running under 2000, but there's a better chance that there wouldn't be any problems moving to SP6a (and the SRP for SP6a, not that it is really needed).

Who wants to check to see what size, and what format, each of the partitions are?

Somebody at Imax needs a good slap in the head for letting a problem like this go unresolved. [thumbsdown]

Maybe I'll drop in and do that myself! [thumbsup]

Posted by Dick Vaughan (Member # 540) on 07-27-2004, 01:55 AM:

Daryl

NT System FAT 995MB
DTAC Audio NTFS 75,320MB

Posted by Daryl C. W. O'Shea (Member # 1303) on 07-27-2004, 02:53 AM:

Thanks Dick. I forgot to ask, is it the server or workstation version?

Does anything else get installed when the drive is upgraded to 80GB?

None of these DTAC units are on a network, right?

Posted by Dick Vaughan (Member # 540) on 07-27-2004, 04:23 AM:

quote: Daryl C. W. O'Shea
Thanks Dick. I forgot to ask, is it the server or workstation version?

Not sure but I think workstation

quote: Daryl C. W. O'Shea
Does anything else get installed when the drive is upgraded to 80GB?

Not that I am aware of

quote: Daryl C. W. O'Shea
None of these DTAC units are on a network, right?

Correct

Posted by Daryl C. W. O'Shea (Member # 1303) on 07-27-2004, 08:38 PM:

Thanks again Dick. I've got a few ideas what could be causing the problem... just waiting for someone (anyone) to send me a copy of the NT system event log when the problem happens and hopefully for someone to have the process list open, sorted by CPU utilization, when the problem happens.

In the meantime, just in case, somebody hasn't set Diskeeper to defragment whenever it wants to (like during a show) via a schedule, "set it and forget it" or smart scheduling, right?

Posted by Brian Michael Weidemann (Member # 2243) on 07-28-2004, 04:31 AM:

As far as I know about the system, nothing else can be opened when the "DTAC" program is running. And I'd be wary of running anything else anyway when it's trying to track during a show. I haven't done too much experimenting like that.

I do know that if, while running a show, the Help button is pushed, it opens up the PDF like it's supposed to, but the system locks up for a few minutes, then it MAY come back.

Posted by Daryl C. W. O'Shea (Member # 1303) on 07-28-2004, 05:30 PM:

Opening Adobe Acrobat on any system is going to bog it down.

Running the task manager won't hurt anything (to be safe make sure the DTAC window is the active window) and it might even float over the DTAC software window. Not only will it not cause any problems, it's the only way you're going to find out what is loading down the system when the problem happens.

Posted by Brian Michael Weidemann (Member # 2243) on 08-16-2004, 04:23 PM:

Current update:

Last week, the problem crept up and attacked three shows in one day. The personnel concerned/involved at this point goes WAY up there, since we are something like the highest grossing IMAX theatre under Regal, and Sony is a little put-off at the numbers (as they appear AFTER refunds!), so ...

Tim Silver was out with our tech to assess. We dropped about a day-and-a-half worth of shows while they worked. A new TIC card (the third one, so far) was finally made to work (since when it was first installed last week, it worked perfectly for an hour before losing it's ability to remember what frame count IS, let alone track it accurately!), and the power supply voltage was upped. Apparently, the voltage might not have been strong enough and the logic of the circuit does some weird things otherwise. The problem, as far as they've concluded, has been narrowed down to this: it's definitely occuring in the TIC card. It sounds reasonable, but (like always) we'll see.

As per Regal corporate, we are running a 35mm print in tandem as a backup, and making announcements to the audience every show as to the "possibility of a sound-related problem, at which time we will conclude the show with our 35mm print, and you are all then welcome to visit Customer Service and (bitch all you want before you) receive compensation."

Well, even though we run it, we made it through the weekend without actually having to USE the 35mm. Here's to many, many more!

By the way, Daryl, I've gone through the Windows NT event list and it contains no events at all that match up to the times of the problem happening. I went back through March in its log. Nothing anomalous.

Posted by Dan Suomi (Member # 2585) on 08-16-2004, 08:41 PM:

How about something simple, although I'm sure you have checked? Are all your connections tight? Maybe a loose wire/connection.

Posted by Daryl C. W. O'Shea (Member # 1303) on 08-19-2004, 03:23 AM:

Random thought: Have you tried moving the timecode interface card to another slot in the computer (you're using the internal card, right)? Assuming it is a PCI card it could be a bus master issue. Moving the card to the immediately neighboring slot (down one) may make a big difference.

Even if it is an ISA card, changing slots could make a difference since some boards share resources between one of the PCI slots and the ISA slot(s).

Posted by Brian Michael Weidemann (Member # 2243) on 08-19-2004, 03:44 AM:

Yep. Moving the second TIC over to another slot was a couple steps back, but didn't solve anything for too long. Right now our newest card (number three) is in the newer slot, and with the power supply adjustment (which we've all got really high hopes for!), we're still running without incident.

As for possible loose connections, Dan; I wish! Connections and cables have been checked, changed, and rechecked countless times.

There's a part of me that somehow "knows" it's even simpler (possibly weirder) than that ... like the CAPS lock has been on this whole time! Oh, wouldn't THAT be a hoot!? [Razz]

Posted by Daryl C. W. O'Shea (Member # 1303) on 08-19-2004, 04:01 AM:

Since two is always better (and you can always get two for twice the cost), maybe it's time to get another DTAC running in parallel that you can switch over to when the first one locks up.

Now if I could find a DTAC unit (and maybe an Imax projector) on eBay, I'd redesign the thing to run under Unix, with a port to OS/2 for Gordon, and move as much of the processing as possible to software since today's computer hardware could handle the extra work that it couldn't when DTAC was first designed.

Posted by Brian Michael Weidemann (Member # 2243) on 08-19-2004, 04:05 AM:

We could get another Palm Pilot upgrade, this time for the DTAC. Now THERE's an OS! The whole IMAX system, all run from a Palm Pilot network! Beauty.

Posted by Gordon McLeod (Member # 33) on 08-19-2004, 09:57 AM:

Daryl said "Now if I could find a DTAC unit (and maybe an Imax projector) on eBay, I'd redesign the thing to run under Unix, with a port to OS/2 for Gordon" all well and good but I think a good old 6track mag would work just fine for me [Big Grin]

Posted by Daryl C. W. O'Shea (Member # 1303) on 08-19-2004, 11:43 AM:

So I should just "remove" your DTAC unit from the booth next time I'm in Toronto? Sounds good.

Posted by Gordon McLeod (Member # 33) on 08-19-2004, 12:59 PM:

We don't have no friggin [bs]

DTAC here just a early DDP and Mag

Posted by Daryl C. W. O'Shea (Member # 1303) on 08-19-2004, 10:27 PM:

That's right too... my bad.

Posted by Dick Vaughan (Member # 540) on 08-20-2004, 03:07 AM:

Ahh the DDP! Why did they design it to look like an Egyptian obelisk? [Confused]

Posted by Gordon McLeod (Member # 33) on 08-20-2004, 09:39 AM:

dick said"Ahh the DDP! Why did they design it to look like an Egyptian obelisk?" because it somedays needs to be aligned on the axis of the sun to work right [Wink]

Posted by Dick Vaughan (Member # 540) on 08-20-2004, 11:00 AM:

And we all know where the sun shines from ,right! [Roll Eyes]

Posted by Mark Gulbrandsen (Member # 72) on 08-22-2004, 05:02 PM:

Actually you don't want a Mega [bs]

....Yuck! At least the SR projector is a reliable piece, the CDC projector would be the better choice as you retain the 15/70 format and reliability... no royalties to pay there either.
__________________________________________________
"Use a DTS and JSK generator off the tach and all the problesm will go away"

What Gord said is the safest way to go and would certainly be a seamless way to back up the DTAC system. I'd bet that Imax would have a cow though if everyone started installing XD-10's in their racks.....but then the show must go on anbd the XD-10 is easily as capable as the DTAC source is. The new DTS E480 Time Code generator that goes in with the XD-10 is also very flexible and allows you to advance/retard on the fly.
______________________________________________________________

Oh oh....Here goes Stewart again........ [Confused]

"I have an extra DTS E-175 that you may buy (cheep)"

Actually you are supposed to return that E-175 generator to DTS if you buy the new E-480. Better check with them first or you may end up paying for both TC gennys.

Mark @ CLACO