Digital studios and post houses need to incorporate new storage technologies now more than ever. It’s not as simple as storing data anymore — they need a strategy for centralized storage, transcoding, rendering, backup, and archiving, not to mention a workflow that incorporates all these solutions.

One company where we helped incorporate this type of strategy is media arts firm thelab. Our latest case study on this deployment is now live on our website.

In addition to a high-performance media SAN, StorNext and StorageManager, Control Group installed a centralized rendering solution for Maya in the form of a powerful render farm. This helped thelab complete a 30-second spot for the world’s fastest production sedan, the 2010 Cadillac 2010 CTS-V, in just 25 days.

Check out the finished product, and read more about CG’s solution!

Share this:

Whew! As always, NAB was quick, productive, and overwhelming – only this time it was in 3D!

Everyone, everywhere, was talking about 3D: cameras, displays, production software, trucks, expertise, etc. The race for preparedness is on, and people will spend as pushy studios and producers want to be the first to do this or that with 3D. First sitcom, first sporting event, first documentary, first newscast, etc. The reality is that we are a ways off from wide-scale adoption, and showrooms and special screenings will be the place for 3D for the foreseeable future. We are in a similar catch-22 phase as a few years ago during the early days of HD production – 3D TVs are just coming onto the market and will drop down to reasonable consumer level prices within the next 2-3 yrs. Consumers are asking “do I buy a 3D TV when there is little to no programming?” while content creators are wondering “do I produce in 3D when there is little to no audience?”

Being an industry event though, it’s about more than bragging rights or audience – it’s about technical feasibility, practice, and logistics – all things which will change when 3D comes to our living room. DirecTV has announced that they will carry four 3D channels starting in June, including ESPN 3D and a dedicated 3D pay-per-view channel. And Cablevision dipped its toes into the 3D pool a few weeks ago with an MSG Network broadcast of a Rangers and Islanders hockey game live from Madison Square Garden.

After talking to a few industry experts who participated in the recent Masters in 3D, the challenges with 3D production are less technical than logistical: camera placement for example. While HD favors wide top-down shots for seeing all the action, 3D is most effective with close, ground-level cameras – imagine Phil Mickelson’s birdie putt as viewed from grass-level, just across the green, the ball rolling right toward you as he sinks it…

There were also lots of other things of interest: Falconstor’s HyperFS, CatDV asset management, Avid’s Java app for editing over the web, 3ality cameras, Adobe CS5, among many more. We were particularly excited by the potential of Active Storage’s Innerpool appliance for metadata. This PCI Express card contains on-board redundant solid-state drives, specifically engineered for storing metadata in an Xsan environment. This has the potential of being something of a game changer, allowing us to more efficiently configure the storage in our Xsan integrations, and giving our clients more bang for their buck when deploying new SAN solutions.

We had some great meetings with prospective clients, old and new friends, fellow consultants and vendors, and we’re excited about some emerging strategic opportunities. Our work in online video technology and web delivery combined with our broadcast infrastructure and workflow experience means we are ideally positioned to help organizations streamline and bring these workstreams closer together.

Share this:

As summer starts to wind down and the air conditioning is on less and less, your company’s computing equipment is likely resuming its role as the largest consumer of electricity in the office. And while there’s not much that can be done to change that, there are a variety of ways to reduce your energy use without compromising your company’s IT needs.

The Server Room

Annually, the single greatest consumer of electricity in your office is likely the server room. Between the power hungry server hardware, the UPS with its constant AC-to-DC conversion and the air conditioning running 24 hours a day, 7 days a week, 365.25 days a year, all of that electricity use quickly adds up.

Hands-down, the easiest ways to save energy in your server room is to make sure your server room’s HVAC is doing its job as efficiently as possible. That means keeping the door closed at all times (and of course making sure there’s a door in the first place!) (many offices forget this part), keeping the HVAC system well-maintained, and cooling as small of a space as possible.

When buying buying new server hardware, Control Group can help you make the most efficient choices possible… this means anything from buying the right server for the right job (that eight-core print server might be overkill), to outfitting your new server with green-friendly hard drives which consume less power than standard drives. Control Group is an EPEAT Partner Reseller— EPEAT provides a “green” rating system for IT equipment based on a comprehensive criteria list.

The most efficient server room though, is no server room at all! As more and more businesses move aspects of their business to “the cloud,” they require less hardware onsite. Less hardware means lower electricity bills, which means money saved.

The Workstations

First off, if you still have any big bulky CRT monitors in your office, their time has most certainly come. They are big, ugly power hogs, and have no right to exist in 2009. No, not even on the intern’s computer in the janitorial closet. An LCD monitor will pay for itself in power consumption costs alone in under two years. When purchasing an LCD, consider looking into those which have LED-powered backlit displays. Not only do they consume even less power than LCDs using older CCFL technology, but they’re brighter, contain fewer harmful elements in their manufacturing process, and last longer to boot!

Pro-Tip: After you’ve replaced your CRT monitors, what do you do with them? Control Group works with Per Scholas, which is a local organization that recycles and re-uses computer equipment and works with New York City schools to provide students with their own computers as well as training.

Another great example of a small change that can make a big difference would be to make it office policy to power down computers at the end of the day (or at least the end of the week). For even more energy savings, consider flipping the switch on that power strip your computer is plugged into on top of powering down.

And for those employees who say that shutting down hurts their productivity, tell them to use Windows’ “hibernate” feature rather than the “shut down.” That will allow them to pick up right where they left off, with all of their AutoCAD windows and dozens of email drafts right where they left them.

The Office

One way Control Group saves energy around the office is that we have a “no personal printers” policy. We have three printers in the office: a black and white laser printer, and a color laser printer and a large format plotter to help us better support our architecture clients.

Not only does this centralized printing solution help save energy by avoiding having dozens of printers plugged in and on standby 24 hours a day, but it saves money on supplies—no more $18 ink cartridges—and saves us time from having to troubleshoot problematic printers (there’s a reason desktop printers are so cheap… they’re cheaply made and break constantly).

Perhaps most importantly, this printing setup also saves a lot of paper. Duplex printing (two-sided) is enabled on our laser printers and we have a dedicated “scrap paper” tray that we keep loaded with scrap paper, for those times when you need to print something but it doesn’t matter how it looks.

We may not be paperless yet, but we’re well on our way.

Finally, with all of the energy that your office is saving, consider moving over to wind power. Here in New York, ConEd offers a very competitive Wind Power for Business package. It’s 100% renewable energy and it’s easy to switch over.

While wind power may be 10-15% more expensive than coal, depending on how much you’ve managed to cut your overall power consumption, it’s entirely possible your company is still spending less on electricity than before even while using wind power. And on top of that, you can tell your customers you’re ahead of your competitors in reducing your company’s impact on the environment.

Share this:

As I’ve mentioned in previous posts about testing storage performance with lmdd and bonnie++, different applications require different characteristics from storage to provide the best performance. I’ve highlighted some tests that are good for large streaming files like video, and small file transactions like databases or mail servers. Today I want to look at a tool that runs a series of tests in many different ways to provide you with a holistic view of what the storage can and can’t do.

This tool is called iozone. iozone is open source and runs on a ton of operating systems (including Windows). It runs several tests which can take some time to complete but provide the best overall view of the capabilities of a piece of storage. For instance, iozone runs a write test with files of different sizes and with different size records (the amount of data written at a time). It does this over and over again with writes, reads, random writes, random reads, and so forth. Since it’s running all these tests you can see what sorts of operations will have good performance and which ones will not perform so well. Check out the iozone documentation here.

One really great thing about iozone is that the output it generates can be easily placed in a spreadsheet program like Excel to generate a great 3d diagram describing your storage. Here’s a diagram I generated from some tests on a Linux server.

Results of a write test with iozone

This particular server performed quite well with large files and a record size around 1 MB (interesting to note, this is the same storage from the lmdd post. Notice that the parameters I tested with there are the same as the best write that this disk can do according to iozone!).

If you’ve been following my posts on storage performance testing I hope you’ve learned about some new tools that you can use to see what’s going on. I use these on every deployment to make sure we’re giving our clients solutions that they can depend for performance and reliability. As always, let me know if you have any questions about these tools. Happy testing!

Share this:

Last time I posted about checking disk performance with lmdd. lmdd is great for checking streaming throughput, but what if you have a different kind of application? Every application accesses storage in different ways: with video we need to be able to provide constant throughput when writing a lot of data to the disk, but other applications may have different storage needs. For example, a database can make lots of very small changes to the data on disk in a short period of time. The best performing disk for a database will probably need to have very low seek time and good transactional performance.

bonnie++ is a series of file system tests that focuses on small files. It was designed to behave like a mail server does, creating and dealing with lots of small files (emails). bonnie++ is easy to run and outputs a CSV file that you can view with something like Excel. With the bon_csv2html command you can quickly generate html pages from the CSVs.

Here’s the output from bonnie++ running on a server:

The HTML output of bonnie++ on a Linux Server

At first glance the output can seem quite cryptic, but if we look close we can see that this provides us a great amount of information about latency and speed on different filesystem operations. I generally run this several times as I make changes to verify that the storage is providing the right performance characteristics. Tweaking a file system to make file system operations happen a few milliseconds faster may seem ridiculous, but in some environments it can make a huge difference.

Next time I’ll post about a tool that’s new to me but can test a disk in so many different ways I’m planning to run it on every system we install from now on.

Share this:

Your Data Here

I was having a conversation recently with one of our consultants, David Rocamora, as our team broke down the contents of a start-up’s data center, when I came to the realization that we may have built our last data center.

Now we don’t really build data centers, but we have racked a lot of servers, storage and network gear around the world in tier 1 data centers for our clients. With the change in the economy and the maturity of several cloud services, the data center that we know and love looks like it’s going the way of the wood pulp newspaper.

Certainly more data centers than ever are being built: Google, Microsoft, Apple, and Amazon are soaking up gigawatts (petawatts?) of power like never before. But the days of dressing down in jeans and a sweatshirt and going out to New Jersey or Colorado to rack servers in an earsplitting, freezing cold warehouse of caged servers and blinking lights seems to be drawing to a close.

Some numbers to consider:

This ‘scaled down’ dot com we were consolidating, had spent about $500,000 on a few racks of some amazing equipment (Sun, Check Point, etc) only to find out 4 months later they didn’t need it. Pennies on the dollar. The contract for the floor space, power, and bandwidth goes on for another 8 months and I bet you could buy a modest BMW for what it’s costing them.

Now a similar sized start-up we just started working with on a really innovative interactive image platform, is using the Amazon cloud and RightScale and is spending about $50,000 a year on cloud services. No capital outlay.

Flexibility?

The cloud is infinitely more flexible, we can put servers in Europe in a matter of minutes, set up high availability zones in different regions around the country, and if they start to get swamped with business like we think they will, we’ll be able to turn up as many servers as they need in a few minutes time.

What if the dot com of 4 months ago took off? Order servers, spend capital. Put in a request for more bandwidth, more cage space. Days, maybe weeks go by. Then get out the jeans and sweatshirt and head over to the data center. Earplugs. Man, those servers are loud.

You get the picture. But this is happening so fast it’s amazing. Six months ago when the dot com was building its data center, the Amazon cloud was still in beta, with no SLA, and it wasn’t an option for a serious start up. Today, building a data center isn’t an option for a serious start-up.

Now we have availability zones, provisioning and monitoring tools, the ability to drop terabytes of data into the cloud — shipped through FedEx! But the real promise is the rich API and the spirit of community innovation. Companies like RightScale are finding a niche in the cloud, developing something really valuable, and then selling it as a simple service that makes our lives so much easier.

It’s exciting to see this happen so fast.  To avoid being crushed by this wave, as an IT team, you need to really stay on top of it. IT in our part of the ecosystem is becoming more the art of selecting, deploying, integrating, managing and supporting cloud based services, and much less the craft of building serious web infrastructure.

It’s a little sad for the hardware geek in all of us, saying goodbye to the roar of the servers, putting down the Velcro ties and picking up some slick provisioning and automation scripts. But I think we could get used to deploying 50 servers in a few keystrokes from a quiet, comfortable seat in the office.

Share this:

One of the unique things about how Control Group works is that our focus is much more involved than simply putting in a solution for a client and then moving on. We work with our clients to determine how they work, so we can design IT solutions that really fit their needs. Since we have partnerships with a variety of vendors, we work with our clients to arrive at the best solutions for their business. This means we do quite a bit of research and planning before we begin a project — and then a great deal of testing during and after we install new hardware or software.

I do some work on implementing storage systems for our clients, and we’ve found that different applications have different storage requirements. For example a video post production facility — like the facility at WWE — generally needs lots of disk space that is very good at reading and writing large files at high speeds. The storage here needs to provide good streaming throughput, because high quality video files generally have high bit rates, and are being stored or played back from the disk in real-time for ingesting, editing, or playout. If the storage system is not fast enough to read or write the file in real-time, frames will be dropped. This can cause unsatisfactory media files, programs to crash, or audio and video to become out of sync.

10,000 RPM SAS disks. That's some serious storage.

Suboptimal read/write performance can become a huge problem. When we put in a new system this is something we need to test. I usually do the test with a tool called lmdd.

lmdd comes from the lmbench tools which are provided by Bitmover for benchmarking systems. lmdd is great for testing streaming bandwidth. In most of our engagements with video, we install a Stornext or Xsan filesystem so we’ll run our tests against this. lmdd will probably work on any filesystem that you can mount on your Mac or Linux computer (Leave a comment if you need a version for Mac OS X, I have one compiled).  lmdd lets us verify exactly what the maximum number of megabytes per second we can push through the storage and point us to where we need to make changes to the hardware or software configuration. I use lmdd like this :

lmdd of=/path/to/test_file count=1g

lmdd if=/path/to/test_file

The first tests write performance and the second tests read performance. More information about the syntax is available in the manual page for lmdd. The results of the command from a server I was testing looked like this:

2147.4755 MB in 6.8003 secs, 315.7914 MB/sec

lmdd is great because it’s easy to read. This result shows I could write to the filesystem at 315 megabytes per second. That’s really fast! This is from a test with a server with a lot of RAM and a special filesystem that took advantage of that cache. When I run it on my Macbook, I get a result like this:

18342.6171 MB in 376.7685 secs, 48.6841 MB/sec

So the next time you’re interested in how your storage is performing give lmdd a shot and let me know how it goes. If you’re looking for more information about storage performance testing then stay tuned; I’ll be posting about testing storage with tools that benchmark small reads and writes next.

Share this:

Thanks to Ivan for sending this one our way. Note the detailed labeling on all hardware in the rack…

We’ve got a real treat for you today folks. Thanks to the tireless work of our expert trackers, we’ve managed to photograph the elusive North American Server Room Gator (Alligator Servus). We’re not the first to find her though as she’s been tagged for tracking purposes.

Alligator Servus spotted in the wild. Note the detailed labeling.

At 1’1″, she’s a healthy beauty. Apparently she’s pregnant since it looks like she’s nesting on her traditional nursery, the autoloader. Be careful though, they tend to be rather vicious when with young. Hopefully she’ll be able to hunt down enough snack food remnants to raise a healthy brood.

Share this: