Film-Tech Forum ARCHIVE: SSD drives in Doremi DCP2000

This is topic SSD drives in Doremi DCP2000 in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=002585

Posted by Marilu Magri (Member # 8953) on 02-13-2016, 05:42 PM:

Hi, are there any SSD's known to work reliably with a Doremi DCP 2000 server? This is for occasional touring use so long-term continuous uptime is not an issue but shock-resistance is! Server is currently being used for a 2 hour show in a different location each day and we are concerned about the drives surviving being bashed into and out a van etc.
Thank you! [Smile]

Posted by Brad Miller (Member # 2) on 02-13-2016, 07:12 PM:

Nothing approved officially. I've found all software raid servers are VERY finicky about their drives.

If you MUST have SSD drives, pick up a used Dolby DSS200. That thing is bulletproof no matter what kind of drives you put in it.

Posted by Dave Macaulay (Member # 813) on 02-13-2016, 08:58 PM:

I would try it. The Dolby/Doremi original drive warranty will, of course, be invalid and if something goes really bad Dolby might cancel the entire server warranty. THey will know - the drive type will be in the logs, and you will probably get three "unsupported drive" warnings every time you boot up.
I believe it should work, but I have seen some spinning rust drives just rejected by a Doremi server - not even found.
It would be fun to see how fast the RAID setup goes with SSDs.
I have been sticking with Intel SSDs for projects that use them, they aren't the bleeding-edge fastest but are very reliable in my experience.

Posted by Leo Enticknap (Member # 534) on 02-13-2016, 08:59 PM:

One thing that would make me reluctant to try SSDs in a DCP server is that I'm guessing that the RAID controller (be that software or hardware) will be constantly shuffling data around to make optimum use of the space and minimize seek time, etc. The main determinant of a "spinning rust" HDD's lifetime is hours and spin up/spin down cycles. For an SSD, it's the number of times each sector is written. The more often you write to it, the more quickly you'll wear it out. The desktop operating systems (e.g. Windows and Mac OS) know this, and will not automatically defrag a SSD in the way that they will a traditional hard drive.

But RAID controllers are probably not designed with SSDs in mind: the point of a RAID is storage volume and data security, not speed of access and ruggedness, which are the two main reasons for using a SSD. So even if a DCP server's RAID controller would work with SSDs, I'd speculate that the risk is there that it would burn them out very quickly.

If I were operating a mobile DCP server in a portable rig, I think I'd be inclined to carry one or two spare hard drives around with me in the accessories case, and swap one in and rebuild the RAID in the event that one did not survive transit.

Posted by Marilu Magri (Member # 8953) on 02-14-2016, 12:50 PM:

Thanks all for the input and ideas. As an alternative to SSD's, would a good alternative be to remove the raid drives and transport them in a padded container and then re-fit them at the show?
The reason I'm concerned about the drives is that at the last show there was no content visible at all on the first boot - literally nothing except the option to insert 'black' clips. As you can imagine this was a bit of a heart-stopper but a reboot cleared the problem.
I thought this could be a problem with the raid but then again on further thought perhaps the OS flash drive (which stores the playlists etc) could be flaky. This server has always been a little erratic with boot-up so could even be the mainboard. I have a spare mainboard but am a little reluctant to fiddle with things until the current run is finished.

Posted by Pietro Clarici (Member # 4937) on 02-14-2016, 02:10 PM:

You can remove and reinstall the drivers safely, obviously just don't do that while the server is powered up and make sure to label them in order to put them back up in the right order.

In an emergency I've even moved an entire 3-drive array from one DCP-2000 to another located in a different theater. No problem at all.

Posted by Carsten Kurz (Member # 5396) on 02-14-2016, 02:47 PM:

Yes, since they are drive carriers, it is very easy to take them out for transport. Maybe you can buy a small camera bag or something to carry them safely.

That said - if your server get's bumped around to the extent that you're concerned about the hard discs, maybe you should do something about that first, as the drives are not the only items that could suffer ;-)

BTW - I recently ordered a bare 3.5" SATA drive from Amazon. It 'bumped' into my post box with only a cardboard envelope around it. Not even 1mm of padding.
It works, but I will never order a bare drive directly from Amazon again...

- Carsten

Posted by Randy Stankey (Member # 64) on 02-14-2016, 02:50 PM:

As long as the server is powered off before moving I don't see s problem with using regular hard drives. Just make sure the system is completely shut off and has had a minute or two for the platters to stop spinning before moving the unit.

However I recommend that you get a proper road case. The kind with a solid metal frame and shock mounts.mmif you got one with fans and a power distrio, you could have a completely portable unit on wheels that can be rolled into place and hooked up with a couple-three sets of cables.

The thing I recommend is a power conditioner and/or a UPS!
If you are going mobile with this system, you'll never know what kind of crappy power feeds you'll have to tie into! A good power condioner will be vital!

Posted by Mark Gulbrandsen (Member # 72) on 02-14-2016, 03:41 PM:

When you power down drives the head automatically parks in a stop position and generally locks itself. Dead hard drives can take quite a bit of shock, powered up drives can not take any real shock without damaging some aspect of the drive. USL is the only server manufacturer right now that uses SSD's in their server.

On a portable system my advice is to have an extra set of hard drives already configured and built up in the same server with you in case you run into any difficulties. With a GDC server you can power down and swap sets of drives then power back up and keep going. I have done this for Film Fests. I would assume Doremi can do the same thing. If not, then you just discovered another advantage that GDC has.

Mark

Posted by Marcel Birgelen (Member # 6801) on 02-14-2016, 03:59 PM:

quote: Leo Enticknap
One thing that would make me reluctant to try SSDs in a DCP server is that I'm guessing that the RAID controller (be that software or hardware) will be constantly shuffling data around to make optimum use of the space and minimize seek time, etc. The main determinant of a "spinning rust" HDD's lifetime is hours and spin up/spin down cycles. For an SSD, it's the number of times each sector is written. The more often you write to it, the more quickly you'll wear it out. The desktop operating systems (e.g. Windows and Mac OS) know this, and will not automatically defrag a SSD in the way that they will a traditional hard drive.

Well, most RAID systems do not constantly shuffle data around actually. MOST RAID systems and especially that of the average DCP server are quite dumb. If there is any shuffling, it usually takes place at the filesystem level. There are lots of (semi-)intelligent storage systems out there that do some preventive data shuffling, but they are not the ones you find in your DCI server.

Also, the kind of data you need for DCI is quite boring actually. Huge files, which usually get read and written rather sequentially. So unless you constantly ingest and play the data for like 5 screens simultaneously on the same machine, the level of fragmentation on your average DCI system is really a non-issue.

In case of SSDs, there actually is some black magic going on INSIDE the SSD itself. The controller on your SSD may remap your data as it sees fit. For example, if you keep on overwriting the same data, your SSD may start to write this data on some other physical location on the SSD itself, to keep the wear more balanced. Your RAID controller or your OS isn't even aware of this happening.

Shock resistance for hard disks actually should be a non-issue too, at least if you're not operating them while transporting them. The whole concept of DCI got started by shipping those disks around. Once your disks are in a parked state, forces that would damage the disk would also damage other parts of your equipment.

The biggest pitfall here is if you get those disks work reliably with the Doremi system. And although this is essentially a vanilla Linux software RAID solution, I'm not sure their layer around it will support it. So, you might end up with quite some hackery to get it rolling.

Another pitfall to account for is TRIM support. TRIM support is essential to avoid your SSD drives grinding down to a halt.
Although SSD drives are usually blazingly fast, one thing they're VERY slow at is erasing data. Unlike the rotating rust counterparts, SSDs cannot just overwrite data, they have to erase it first and that's a painstakingly slow process. Also, keep in mind that most delete actions aren't actual deletes, it's a mere "unlink" from the inode table. The problem with this process is that alhough your OS thinks it's "gone", the data is actually still there on your SSD.

So, once you've filled your disks to the brim at least once, you often hit this "nice" wall of total performance degradation.
That's where TRIM comes in. TRIM needs support from both your OS, your RAID and/or controller layer and your SSD. TRIM uses unspent I/O and system time to do some preventive housekeeping. Your OS tells your controller and therefore your disks what data is essentially gone. And in times of low system activity, the data gets prematurely erased by the SSDs, so once there is new data, it can be written without having to erase previously deleted data first.

Once you keep ingesting, you will hit this virtual performance rather soon, even if you keep deleting content. This "performance barrier" should not hurt playback speeds, but it will dramatically impair ingest performance and MIGHT even negatively impact system performance. So a correct working TRIM implementation is not just a nice to have for such a setup.

Posted by Brad Miller (Member # 2) on 02-14-2016, 05:11 PM:

Mark, there is no advantage to a GDC as you insinuated above. You can do the same thing with a Dolby or a Doremi as well. In fact with a Dolby server having a spare set of drives means you also get a spare OS. With the GDC and it's un-raided OS drive, you are at a higher chance of failure than any other server.

Posted by Leo Enticknap (Member # 534) on 02-14-2016, 08:42 PM:

quote: Marilu Magri
Thanks all for the input and ideas. As an alternative to SSD's, would a good alternative be to remove the raid drives and transport them in a padded container and then re-fit them at the show?

We have two festivals a year that are sponsored by Barco, and so we have to have a temporary loaner Barco projector shipped in for them. Boston Light and Sound takes care of the installation, operation and de-rig (hired by the fests), and they bring in a Doremi server to use with the Barco, to avoid having to reconfigure our DSS200 and provide a cat745 (and also because they can pre-ingest and test the content and KDMs, and arrive with everything ready to play).

Removing the hard drives for shipment is precisely what they do - in fact, they pack them in the same padded cardboard boxes that the drives ship from the Doremi factory in. I'm guessing that the hard drive failure rate they have from doing this is low enough for it to be a non-issue for them.

Even taking on board Marcel's belief that RAID controllers are capable of being kind to SSDs, my instinct on this is that the sheer cost of SSD storage versus spinning rust is not worth it for the ruggedness gain. And as he points out, probably around 99% of DCPs are shipped in traditional hard drives in padded boxes. In almost two years since I re-entered the cinema biz working in a DCP environment, I can only recall two DCP drives DOA (due to "click of death" drive failure) in all of that time, and both of those were 2.5" consumer drives still in their consumer retail packaging, not CRU drives in properly designed shipping containers.

Posted by Marilu Magri (Member # 8953) on 02-15-2016, 05:27 AM:

This is all really great info thanks!
Does anybody have any idea why no content would show-up on first boot? Has this happened to anybody else?

On balance it would seem that I was probably barking up the wrong tree thinking about SSD's considering that 'traditional' hard drives survive couriers all the time - well at least most of the time.
We have the server mounted in a good strong flight-case along with CP 750, line mixer, PDU etc. It's not a 'shock-mount' type though as these appeared to be very expensive. We treat the case with as much respect as possible avoiding any hard knocks - at least I do, but you never know if your assistants are equally careful!
Yes I anticipated that dirty mains power could be a problem but unfortunately the UPS I did order was very portable but just not man-enough to power the server consistently. So next step is to order a better UPS to power the whole rack/case.
You guys on here have certainly helped me to think-through all the options and considerations.

If anybody has any ideas about that missing content..

Posted by Carsten Kurz (Member # 5396) on 02-15-2016, 06:32 AM:

BTW - does anyone know the brand/manufacturer of the Doremi RAID disc carrier system (those with the blue latches)? I would want to buy a set of them.

- Carsten

Posted by Ken Lackner (Member # 1002) on 02-15-2016, 08:26 AM:

quote: Pietro Clarici
You can remove and reinstall the drivers safely, obviously just don't do that while the server is powered up and make sure to label them in order to put them back up in the right order.

Are you sure the order matters on a Doremi? I don't know because I haven't tried, but I know that on a GDC it does not, and frankly, I don't understand why. Can anyone explain that?

quote: Brad Miller
With the GDC and it's un-raided OS drive, you are at a higher chance of failure than any other server.

What about Doremi? The OS isn't on the RAID, right?

Posted by Mark Gulbrandsen (Member # 72) on 02-15-2016, 08:26 AM:

Actually Brad,

That was tried here before and the server did not recognize the four other drives even though the new raid was done on the same server. Perhaps that only occurs after firmware version 4.7.2.9.3.1.5.8.0.1.01, anyway... We gave up on that in regards to Dolby and have since gotten rid of them all. Life is good!

Mark

Posted by Carsten Kurz (Member # 5396) on 02-15-2016, 08:39 AM:

quote: Ken Lackner
Are you sure the order matters on a Doremi? I don't know because I haven't tried, but I know that on a GDC it does not, and frankly, I don't understand why. Can anyone explain that?

At least Doremi says in their manuals that it DOES matter. I know other servers do not care, they use a marker on the disc to identify the order by software. It certainly would be possible on a Doremi as well if they'd care enough, but after all, how often do you swap RAID drives, and even if, what's the big issue to reinsert them if you swapped them accidentally? Having proper visual labels on them is, of course, useful to get it right from the start. Factory supplied drive sets do have these markers on them.

- Carsten

Posted by Ken Lackner (Member # 1002) on 02-15-2016, 08:52 AM:

Yes, I've seen the labels on the factory-supplied drives. And you are certainly correct that it takes no effort to take note of the drive order when removing them. My question was more of a "how does it work?" from a technical standpoint. Like, why possible on GDC but not on Doremi?

Posted by Dave Macaulay (Member # 813) on 02-15-2016, 09:25 AM:

Doremi labels the drives but you can mix them up and it still works. You get some error messages about drives being different but the data is unaffected.
Sometimes you can swap in a set of drives from a different server and they are not recognized. Doremi's procedure https://youtu.be/GQrDxRJ6p3Q for server replacement has the "failed" server's drives used in the "new" server so it should work.

Posted by Randy Stankey (Member # 64) on 02-15-2016, 12:59 PM:

quote:
We have the server mounted in a good strong flight-case along with CP 750, line mixer, PDU etc. It's not a 'shock-mount' type though as these appeared to be very expensive. We treat the case with as much respect as possible avoiding any hard knocks - at least I do, but you never know if your assistants are equally careful!

You know your people better than I do. If you think they understand how to take care of the equipment properly then what you have is probably fine.

When it comes to deciding how much to spend on protecting equipment, consider the cost of replacement. Up to half the cost of replacement is a good starting point. Vary the figure based on your needs.

My old boss balked at putting protective grilles over a bunch of speakers because he thought $50 each was too expensive until he had to shell out $500 to replace a speaker cone. Somebody punched a hole in one with the end of a 2x4. The requisition for those grilles was approved the next day.

$50 is a good investment if it saves you from having to spend $500 in order to replace something.

Also, don't forget to include the cost of business interruption in your figures. How much money will you lose if you have to wait for repairs to be made and replacement parts/equipment to arrive?

Posted by Carsten Kurz (Member # 5396) on 02-15-2016, 07:01 PM:

quote: Ken Lackner
My question was more of a "how does it work?" from a technical standpoint. Like, why possible on GDC but not on Doremi?

Well, just like the visual labels that help you with inserting the drives into the proper slots, the RAID software/controller is able to label each drive with an ID where in the RAID it does belong. So before mounting the RAID, it simply looks at these IDs and resorts the drives before mounting them in the proper RAID order. Doremi COULD do it as well, but they decided it's not worth the effort.

- Carsten

Posted by Mark Gulbrandsen (Member # 72) on 02-15-2016, 08:36 PM:

Both GDC and Doremi use Intel Software raid as the basis of their systems. You'll find those chips on the motherboards. Its actually a VERY robust system that offers great versatility and high reliability for Software RAID. The data on the Media drives on a cinema server are not considered to be irreplaceable data because that data still exists elsewhere and you would just re-ingest the data again. On Dolby where you also have the OS using software RAID is a bit more risky, especially for the OS to become corrupt. Doremi's and GDC's OS runs from separate drives and Dolby using the Hardware RAID makes good sense in that case because it offers the highest reliability. Cinedigm actually used Intel Software RAID on the micro TMS systems and those have also been very reliable over the last five years.

Mark

Posted by Marcel Birgelen (Member # 6801) on 02-16-2016, 03:34 PM:

quote: Carsten Kurz
BTW - does anyone know the brand/manufacturer of the Doremi RAID disc carrier system (those with the blue latches)? I would want to buy a set of them.

I've not found them yet and I've been looking for them in the past. Although they procure their recent motherboard and some other components from Supermicro, the 4U chassis is not built by Supermicro or one of their OEM labels but one of the many other OEM manufacturers out there. You could replace the whole bay assembly with some other bay assembly, as this is an industry standard format though.

Posted by Carsten Kurz (Member # 5396) on 02-16-2016, 03:45 PM:

I found a reference to RAID drive carriers in a Doremi dealer price list, at a really low price. I'll have to ask wether these are really available, though.

I'll see if I can find a manufacturer reference on one of these carriers.

We actually use a drive in the fourth (usually empty) slot of a 2k4 for backups, but would like to have 1 or 2 more carriers.

- Carsten

Posted by Stephan Shelley (Member # 8587) on 02-17-2016, 05:26 PM:

Actually I had a raid card fail in a Dolby DSS100. I have a spare DSS 100 on hand and was able to swap drives into the spare. The DSS 100 that failed was running system 4.7.3. The one it went into was 4.2.X. I did have to run the upgrade disk for firmware update on the raid card as well as the IDE flash boot disk. Back on screen. No loss of content or SPLs. Getting a new raid card to fix the one that went down.

Posted by Carsten Kurz (Member # 5396) on 02-18-2016, 06:01 AM:

My integrator quoted these Doremi RAID drive carriers now at around 35US$.

- Carsten

Posted by Mark Gulbrandsen (Member # 72) on 02-21-2016, 12:05 PM:

I also lost a raid card in a DSS-100 back in 2011. Even then Dolby could not supply a replacement card for it. Turned out it's a common LSI (if memory serves me) megaraid card of the era. I did not bother to fix this one however I was tempted to plug in a used identical megaraid card from Ebay and run the upgrade so it would get flashed with Dolby's version of the firmware. But alas... all the Dolbys were replaced with GDC's in the entire chain because of VPF's.

Mark

Posted by Leo Enticknap (Member # 534) on 02-21-2016, 02:40 PM:

Is the DSS100's RAID card the same as the one in the 200? If so, it's a 3Ware 9650SE-4LPML as a straight swap for the one that came in the machine, or a 8LPML if you want to use the four drive bays on the right and upgrade your server to an 8-drive RAID. Both cards are still widely available online from various vendors, including Amazon.

Posted by Steve Guttag (Member # 268) on 02-22-2016, 05:38 AM:

The last version of the DSS200 used the 9750 series of RAID cards.

Posted by Mark Gulbrandsen (Member # 72) on 02-22-2016, 08:51 AM:

Nope, the two servers have different buss. The whole key is to get the card to flash with the correct DOLBY version of the 3-Ware firmware which is certainly part of some of the Dolby full firmware Installs (but which ones?). Note that LSI and 3-Ware are now owned by Avago Technologies and unless you are a large industrial user of Avago stuff they will not even reply to your questions.

I have successfully flashed HP LSI cards with Dell firmware as the cards are identical. In Dolby's case the raid firmware is written to interface with Dolby's own OS.

Mark

Posted by Stephan Shelley (Member # 8587) on 02-24-2016, 10:33 AM:

DSS 100 is 3ware 9550sxu-4lp. DSL 100 would be different yet.

DSS 100 and early 200s use a flash ide boot device. The later 200s boot directly off the raid.