Tux the Linux Penguin

I work with Linux quite a bit at CG and it comes up in some interesting places. Consistently we find that the largest and smallest systems we work with are running Linux. If it’s a large system, like a database or storage network, Linux can provide the stability, robustness, and uptime required to make the project a success. If it’s a very small system, like an embedded display or a custom purpose device, we choose Linux because of its flexibility.

Over the last year or so, we’ve had a need to spin up many more virtual computers than we used to. It’s necessary to create a handful of  computers for each project we work on and the number of projects that we are taking on at a time is growing. Tools like EC2 and VMWare have made it very easy for us to create new virtual computers, but what about managing them? For Linux there are some great solutions. In this blog post I would like to talk about how we’re using some of the tools to deal with the large number of machines we are managing in Amazon EC2 for ourselves and our clients.

Cloud Control

The first part of the system is Amazon’s Elastic Compute Cloud (or EC2 for short). I’ve written about EC2 before, but to be really brief about it: EC2 allows you to create computers on the Internet quickly and lets you pay by the hour for their use. The instances you create are virtual machines that run in a large Xen based infrastructure that Amazon provides. To customize the software and configuration of the instances, Amazon lets you create snapshots of your computers (called Amazon Machine Images or AMIs) that you can launch new instances from.

It’s hard to say that making an AMI is difficult, it’s just a few keystrokes and a coffee break while the image is prepared and stored for your future use. When you begin to manage dozens and dozens of different images this becomes a problem. It became clear to us that we needed to have one AMI to manage, and build every computer from image dynamically. To do this we needed a way to pass specific information to each instance when it started up and then have a tool customize the instance based on the specific information about it. Amazon provides a method to pass specific information to an instance (it’s called user-data) and the good folks at Alestic have made some great AMIs around it with some good documentation.

Enter the Puppet Master

We have been using a tool called Puppet for nearly two years now to manage a handful of computers. We selected that to manage the whole lot of Linux computers that we were dealing with. Puppet allows us to describe how a computer should work in a general way. We can make collections of configurations for standard things so it’s easy to reuse what we create over and over again. Our Puppet configuration is stored in a Git repository so many administrators and developers can collaborate on it and our server configurations are backed up and version controlled automatically.

Bringing it all together

Puppet decides what configuration to use for a computer based on its hostname. We use the EC2 user data to pass a new instance the address of a Puppet server and the hostname that the machine should assume. When it boots up it sets its hostname and checks in to receive the configuration that we’ve stored for it. All changes to the machine happen through Puppet so we don’t have to spend a lot of time SSHing in and customizing the machine. It’s also very easy for us to duplicate a machine for testing or whatever we need.

While we originally did this to make our lives easier as we manage more machines there turned out to be some really cool side effects:

• Excellent Security: We don’t want to store sensitive information in Puppet. This means no passwords or secret stuff. To resolve this we require all developers and administrators to use key-based authentication to get access to the computers via SSH. This is very handy and it eliminates the need to remember passwords or for an administrator to have to reset passwords for users. Someday I would love to take the next step and have all of our users and machines be part of the Monkeysphere.
• Accountability: All of our configuration is tracked in a Git repository so we can see the history about what has changed on certain hosts and who changed it. Change control occurs automatically with Puppet and it’s easy to examine and understand what has changed and why it changed.
• Repeatability: By storing all Linux computer configuration in a single place we can easily repeat what we did for one server on another. Puppet institutionalizes all of our Linux knowledge in one place and saves us time every time we have to create a new machine. If you want to see how someone has done something in the past there’s no more need to dig through emails or documentation, just look at the facts in the Puppet repository and even copy and paste it into your new configuration.
• Portability: We use Puppet to manage much more than just EC2 instances. Physical machines and virtual machines in our ESX installation are managed this way too. It gives us one tool to take care of any Linux machine we deal with. Puppet also supports other operating systems. We’re looking to expand our use of it to Mac OS X machines and maybe even Windows sometime soon.

There are certainly other ways to solve the problems we’re up against, but this is the way we chose. We did some extensive evaluation of Chef (which is a tool like Puppet) and we’ve used Rightscale before. This sort of thing is becoming very important as we manage more and more computers. I expect we’ll see a lot of exciting products, techniques, and services in this space as time goes on.

If you have any questions or comments about our setup, or would like to discuss implementing something like this for your business, leave a comment or get in touch with us. We’d love to help.

Share this:

We have been doing some work with with Amazon’s Elastic Computing Cloud (EC2) which allows us to create virtual machines in the cloud in a few seconds. These are great for hosting websites, and what’s cool about them is that if you get Slashdotted or experience a similar unexpected spike in traffic you can create new hosts immediately. Recently Amazon added a new service called Elastic Load Balancing (ELB) which can distribute load across hosts. We’ve been looking at this for some of our recent development and infrastructure projects.

I just read this description of how ELB works by Shlomo Swidler from his Cloud Developer Tips blog. It’s a great reference.

You pay for ELB by usage just like everything else at AWS. From Amazon: “You are charged at $0.025 per hour for each Elastic Load Balancer, plus $0.008 per GB of data transferred through an Elastic Load Balancer.” For reference, on a deployment project in 2008 our Engineering team used a Cisco load balancer which I imagine cost a few thousand bucks.

Cost isn’t the only advantage. These can be created and destroyed quickly and remotely, allowing us to work more efficiently and spend less time visiting data centers in the middle of nowhere. This leads to improved quality of service for our clients as we can spend more time consulting on future technology growth plans and less time troubleshooting servers in cold, loud data centers.

This blog post brought to you by the iced coffee I am enjoying in the comfort and quiet of my office while deploying virtual machines!

Share this:

This tool is called iozone. iozone is open source and runs on a ton of operating systems (including Windows). It runs several tests which can take some time to complete but provide the best overall view of the capabilities of a piece of storage. For instance, iozone runs a write test with files of different sizes and with different size records (the amount of data written at a time). It does this over and over again with writes, reads, random writes, random reads, and so forth. Since it’s running all these tests you can see what sorts of operations will have good performance and which ones will not perform so well. Check out the iozone documentation here.

One really great thing about iozone is that the output it generates can be easily placed in a spreadsheet program like Excel to generate a great 3d diagram describing your storage. Here’s a diagram I generated from some tests on a Linux server.

Results of a write test with iozone

This particular server performed quite well with large files and a record size around 1 MB (interesting to note, this is the same storage from the lmdd post. Notice that the parameters I tested with there are the same as the best write that this disk can do according to iozone!).

If you’ve been following my posts on storage performance testing I hope you’ve learned about some new tools that you can use to see what’s going on. I use these on every deployment to make sure we’re giving our clients solutions that they can depend for performance and reliability. As always, let me know if you have any questions about these tools. Happy testing!

Share this:

bonnie++ is a series of file system tests that focuses on small files. It was designed to behave like a mail server does, creating and dealing with lots of small files (emails). bonnie++ is easy to run and outputs a CSV file that you can view with something like Excel. With the bon_csv2html command you can quickly generate html pages from the CSVs.

Here’s the output from bonnie++ running on a server:

The HTML output of bonnie++ on a Linux Server

At first glance the output can seem quite cryptic, but if we look close we can see that this provides us a great amount of information about latency and speed on different filesystem operations. I generally run this several times as I make changes to verify that the storage is providing the right performance characteristics. Tweaking a file system to make file system operations happen a few milliseconds faster may seem ridiculous, but in some environments it can make a huge difference.

Next time I’ll post about a tool that’s new to me but can test a disk in so many different ways I’m planning to run it on every system we install from now on.

Share this:

Your Data Here

I was having a conversation recently with one of our consultants, David Rocamora, as our team broke down the contents of a start-up’s data center, when I came to the realization that we may have built our last data center.

Now we don’t really build data centers, but we have racked a lot of servers, storage and network gear around the world in tier 1 data centers for our clients. With the change in the economy and the maturity of several cloud services, the data center that we know and love looks like it’s going the way of the wood pulp newspaper.

Certainly more data centers than ever are being built: Google, Microsoft, Apple, and Amazon are soaking up gigawatts (petawatts?) of power like never before. But the days of dressing down in jeans and a sweatshirt and going out to New Jersey or Colorado to rack servers in an earsplitting, freezing cold warehouse of caged servers and blinking lights seems to be drawing to a close.

Some numbers to consider:

This ‘scaled down’ dot com we were consolidating, had spent about $500,000 on a few racks of some amazing equipment (Sun, Check Point, etc) only to find out 4 months later they didn’t need it. Pennies on the dollar. The contract for the floor space, power, and bandwidth goes on for another 8 months and I bet you could buy a modest BMW for what it’s costing them.

Now a similar sized start-up we just started working with on a really innovative interactive image platform, is using the Amazon cloud and RightScale and is spending about $50,000 a year on cloud services. No capital outlay.

Flexibility?

The cloud is infinitely more flexible, we can put servers in Europe in a matter of minutes, set up high availability zones in different regions around the country, and if they start to get swamped with business like we think they will, we’ll be able to turn up as many servers as they need in a few minutes time.

What if the dot com of 4 months ago took off? Order servers, spend capital. Put in a request for more bandwidth, more cage space. Days, maybe weeks go by. Then get out the jeans and sweatshirt and head over to the data center. Earplugs. Man, those servers are loud.

You get the picture. But this is happening so fast it’s amazing. Six months ago when the dot com was building its data center, the Amazon cloud was still in beta, with no SLA, and it wasn’t an option for a serious start up. Today, building a data center isn’t an option for a serious start-up.

Now we have availability zones, provisioning and monitoring tools, the ability to drop terabytes of data into the cloud — shipped through FedEx! But the real promise is the rich API and the spirit of community innovation. Companies like RightScale are finding a niche in the cloud, developing something really valuable, and then selling it as a simple service that makes our lives so much easier.

It’s exciting to see this happen so fast.  To avoid being crushed by this wave, as an IT team, you need to really stay on top of it. IT in our part of the ecosystem is becoming more the art of selecting, deploying, integrating, managing and supporting cloud based services, and much less the craft of building serious web infrastructure.

It’s a little sad for the hardware geek in all of us, saying goodbye to the roar of the servers, putting down the Velcro ties and picking up some slick provisioning and automation scripts. But I think we could get used to deploying 50 servers in a few keystrokes from a quiet, comfortable seat in the office.

Share this:

I do some work on implementing storage systems for our clients, and we’ve found that different applications have different storage requirements. For example a video post production facility — like the facility at WWE — generally needs lots of disk space that is very good at reading and writing large files at high speeds. The storage here needs to provide good streaming throughput, because high quality video files generally have high bit rates, and are being stored or played back from the disk in real-time for ingesting, editing, or playout. If the storage system is not fast enough to read or write the file in real-time, frames will be dropped. This can cause unsatisfactory media files, programs to crash, or audio and video to become out of sync.

10,000 RPM SAS disks. That's some serious storage.

Suboptimal read/write performance can become a huge problem. When we put in a new system this is something we need to test. I usually do the test with a tool called lmdd.

lmdd comes from the lmbench tools which are provided by Bitmover for benchmarking systems. lmdd is great for testing streaming bandwidth. In most of our engagements with video, we install a Stornext or Xsan filesystem so we’ll run our tests against this. lmdd will probably work on any filesystem that you can mount on your Mac or Linux computer (Leave a comment if you need a version for Mac OS X, I have one compiled).  lmdd lets us verify exactly what the maximum number of megabytes per second we can push through the storage and point us to where we need to make changes to the hardware or software configuration. I use lmdd like this :

lmdd of=/path/to/test_file count=1g

lmdd if=/path/to/test_file

The first tests write performance and the second tests read performance. More information about the syntax is available in the manual page for lmdd. The results of the command from a server I was testing looked like this:

2147.4755 MB in 6.8003 secs, 315.7914 MB/sec

lmdd is great because it’s easy to read. This result shows I could write to the filesystem at 315 megabytes per second. That’s really fast! This is from a test with a server with a lot of RAM and a special filesystem that took advantage of that cache. When I run it on my Macbook, I get a result like this:

18342.6171 MB in 376.7685 secs, 48.6841 MB/sec

So the next time you’re interested in how your storage is performing give lmdd a shot and let me know how it goes. If you’re looking for more information about storage performance testing then stay tuned; I’ll be posting about testing storage with tools that benchmark small reads and writes next.

Share this:

We’ve got a real treat for you today folks. Thanks to the tireless work of our expert trackers, we’ve managed to photograph the elusive North American Server Room Gator (Alligator Servus). We’re not the first to find her though as she’s been tagged for tracking purposes.

Alligator Servus spotted in the wild. Note the detailed labeling.

At 1’1″, she’s a healthy beauty. Apparently she’s pregnant since it looks like she’s nesting on her traditional nursery, the autoloader. Be careful though, they tend to be rather vicious when with young. Hopefully she’ll be able to hunt down enough snack food remnants to raise a healthy brood.

Share this: