Film-Tech Forum ARCHIVE: Restoring content after rebuilding RAID on Hollywood Software (Cinedigm) LMS

This is topic Restoring content after rebuilding RAID on Hollywood Software (Cinedigm) LMS in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=003707

Posted by Justin Hamaker (Member # 2165) on 06-04-2019, 07:50 PM:

Our LMS is just about 8 years old and we just had the first RAID drive failure. Rather than simply replacing the bad drive, I am considering rebuilding the array using larger drives. Currently it's six 1TB drives for a total capacity of 3.6TB.

Unfortunately we frequently run out of space and need to delete content which we could still need in order to make room for whatever is coming in next. This isn't helped by the fact that we have a satellite server and they push all 16 versions of a movie. They also push any titles they are about to delete. There have been more than a few times I've left on Monday with 25% available for the new title coming out this week, then come in to find the new title didn't ingest because they pushed 3 old titles which took up all the space.

Anyway, I was considering replacing the 6 1TB drives with 6 3TB drives and then rebuilding the array. However, I do have trailers and policy clips I will need but can't get from the studio. I've thought about pushing those out to the GDC servers, but I'm not sure how to get them back on the LMS. Alternatively, can I backup just certain files to an external drive and then re-ingest those? I know that I can call Deluxe to have hard drives shipped for the features I need to keep.

Any feedback is appreciated.

Posted by John Thomas (Member # 6509) on 06-04-2019, 09:58 PM:

Good on you for deciding to replace all the drives. Even if you were not increasing capacity, it is well past the time to replace them. The other drives were manufactured at the same time, probably in the same factory, and have lived in the same environment for 8 years, so you can (should) expect multiple failures in a short time span.

That being said: Run a consistency check on your LMS RAID before you consider the data you're keeping to be good. Just because the controller hasn't kicked out any other drives doesn't mean their data is completely intact.

I assume your enclosure only has 6 slots. If it has more, I would suggest using more of those slots and configuring a hot spare.

Someone who knows more about GDC's may have some advice about getting content off of them. If you have any Doremi servers it's a breeze.

Posted by Steve Guttag (Member # 268) on 06-05-2019, 12:25 AM:

Since Justin has a TCC system (last I checked), Odds are, his server was put together by either InSight or Strong and the RAID is in a RAID-50 configuration with a Global Hot Spare with the OS running on its own pair of mirrored drives.

As for saving the content. Presuming you can log in as a tech_user and can get to the Windows desktop, you can manually move content off of the DATA drive to a suitable storage device. The TCC does compartmentalize content (puts captions in one folder, CPLs in another...etc. so you do have to pick up the pieces.

Alternately you can push the content to your SMS servers and depending on the servers you use, it can be more or less easy/difficult to back the content off from there. For instance, a Dolby DSS server makes it super easy...get a large HDD and put it in a CRU carrier and then merely move the content off. If you have recent Dolby software on it, it should move the whole package, not just the bits that Dolby needs. Alternately, you can go to the "generated packages" folder accessible from the normal FTP folder/login and move the entire package off that way (via FTP).

It is almost always possible to do it with most SMS servers that way, by the way. They have a means for FTP so using FTP credentials you likely can move content off via FTP.

Once you have your content on your large drive, then reingest to the LMS using that drive.

Posted by Justin Hamaker (Member # 2165) on 06-05-2019, 12:47 AM:

Steve, you are correct. We have the TCC. I believe the equipment was purchased through Ace, but I don't know who put it together.

Based on what you're saying, I wish we had Dolby Servers. All of our servers are GDC. Thinking this through, I'm wondering if I can just connect a desktop computer to the content network and the FTP into one of those GDCs using a client like Filezilla. Then I can easily dump the transfered files onto a CRU drive - or a regular hdd and drop that in a CRU chassis.

Posted by Steve Guttag (Member # 268) on 06-05-2019, 03:03 AM:

GDC does give one the ability to use an FTP transfer utility like Filezilla or WinSCP.

Use normal FTP and use their normal Content FTP credentials (it is published somewhere in their user manuals for setting up server-to-server content transfers). Logging in in this manner will place you in their directory with the content and, like on a Dolby "generated packages" folder, you'll see folders with all of the various pieces however it will all be UUID based, not using content names but everything should be tidy in those folders.

Posted by Carsten Kurz (Member # 5396) on 06-05-2019, 08:25 AM:

Would the TMS not offer SOME kind of backup function? Wondering wether you could setup a standard PC with FTP server as a 'screen' and simply push all content to it?

Doing a local file based backup on the TMS with the assets separated as Steve describes it appears very risky to me. You may end up with having everything backed up, but nothing being usable for reingest...

- Carsten

Posted by Steve Guttag (Member # 268) on 06-05-2019, 10:41 AM:

You would have to backup the database before the the backup so it would know what files it has. It keeps track of its database rather than constantly reloading/scanning what it has (makes it MUCH faster).

There is nothing to stop you from copying the "D" drive to a suitably large drive and then copying it back. Just know, it is not going to be a fast procedure for doing a lot of content since your connectivity is a bit limited. Probably eSATA on your vintage of LMS is going to be your fastest datapath.

Posted by Justin Hamaker (Member # 2165) on 06-05-2019, 06:02 PM:

A tech I talked to told me it could take 36 hours to restore a full backup of your content.

Realistically I'm just looking to back up a few policy trailers and similar content which I can't get restored by DCDC. Trailers and features are sitting on the satellite server in my projection room and everything I currently need is going to be on there for at least a couple more weeks.

Posted by John Thomas (Member # 6509) on 06-06-2019, 09:59 PM:

If you're going the FTP route, cut out the middle man: Put a couple CRU drives into the bays on your LMS, then use FileZilla under tech_user to download the content from your GDC servers to those CRU drives. Then once your LMS RAID is up and ready, the content is right there and ready to ingest.

Posted by Steve Guttag (Member # 268) on 06-06-2019, 10:40 PM:

He doesn't even need Filzilla for that. Presuming he has access to the Windows interface, the drives will show up like any other drive. It is then a matter of dray-n-drop the files over.

Posted by John Thomas (Member # 6509) on 06-07-2019, 12:34 AM:

While true and possible, the ftp transfer would benefit from FileZilla's queuing -- and it's probably already installed.

One would be wise not to trust multiple large concurrent ftp transfers to Windows explorer, especially when using Windows Server 2003 or 2008.

Posted by Justin Hamaker (Member # 2165) on 06-07-2019, 10:10 PM:

When I log in to a GDC with an ftp client, I understand that the folders I see are the uuid for the particular clip. Do I want to down load the uuid folder, or the content withing the folder.

I was able to get logged in to the servers without any trouble once I was connected to the content network.

Posted by Steve Guttag (Member # 268) on 06-08-2019, 08:40 AM:

The whole folder

Posted by Mark Gulbrandsen (Member # 72) on 06-09-2019, 01:42 PM:

Whie I don't remember if the RAID in the TCM rebuilds manually or automatically, and it will also depend on if it's IBM or Dell system and since you will obviously be doing it I'd just replace one drive at a time. Then, either do the manual rebuild or let it auto rebuild. That way you will maintain all your existing content. I do know that Dell and IBM raid controllers work very differently, so check out the exact procedure before hand.

Mark

Mark

Posted by Justin Hamaker (Member # 2165) on 06-13-2019, 01:23 AM:

Turns out the folder names on the GDC servers are not the uuid of the content. Fortunately Filezilla allows you to look at the uuid xml file, so I can figure out which folder is what. But it's a tedious process.

Posted by Steve Guttag (Member # 268) on 06-13-2019, 07:14 AM:

Yes, yes it is! Looking in the folder, the CPL file will have the name but yes, you had to open the folder of many wrong folders to find the one you want unless you know the date you ingested the content then the date of the folder could be your clue. Honestly, I didn't know that GDC didn't use the CPL UUID (I thought they did).

Posted by Ian Freer (Member # 1940) on 06-13-2019, 05:06 PM:

It's been a while since I ftp'd off a GDC, but yeah, I'm pretty sure it’s NOT the CPL UUID. It might be the PKL ID (or asset map, but unlikely), or it might be completely random, I can't remember if I ever fully worked it out and have forgotten, or if I just gave up trying...

I think remember being able to look at the folder size somewhere, or maybe it was just individual asset size in FileZilla. Features were obvious if they were authored with multiple reels so you can identify and skip those pretty quickly; features authored as a single reel needed a double check of asset size, but it was pretty obvious from that.

But when looking for smaller packages I think I ended up just grabbing every small package on the server and copying them off. In my cases, there were not hundreds of ad's and policy's etc., so it didn't take too long (but it did take toooo long compared to how easy it *could/should* be...)

Posted by Justin Hamaker (Member # 2165) on 06-20-2019, 04:01 PM:

I went ahead and replaced all 6 drives of the RAID - upgrading from 1TB to 3TB. We're just getting back on line today because it took 100+ hours for the RAID to initialize after changing the disks. But I will go from having 3.6TB of storage to 11.2TB. I haven't gotten to restoring the content yet, but I don't think it will be an issue since it was all visible on the USB drive.