Tux the Linux Penguin

I work with Linux quite a bit at CG and it comes up in some interesting places. Consistently we find that the largest and smallest systems we work with are running Linux. If it’s a large system, like a database or storage network, Linux can provide the stability, robustness, and uptime required to make the project a success. If it’s a very small system, like an embedded display or a custom purpose device, we choose Linux because of its flexibility.

Over the last year or so, we’ve had a need to spin up many more virtual computers than we used to. It’s necessary to create a handful of  computers for each project we work on and the number of projects that we are taking on at a time is growing. Tools like EC2 and VMWare have made it very easy for us to create new virtual computers, but what about managing them? For Linux there are some great solutions. In this blog post I would like to talk about how we’re using some of the tools to deal with the large number of machines we are managing in Amazon EC2 for ourselves and our clients.

Cloud Control

The first part of the system is Amazon’s Elastic Compute Cloud (or EC2 for short). I’ve written about EC2 before, but to be really brief about it: EC2 allows you to create computers on the Internet quickly and lets you pay by the hour for their use. The instances you create are virtual machines that run in a large Xen based infrastructure that Amazon provides. To customize the software and configuration of the instances, Amazon lets you create snapshots of your computers (called Amazon Machine Images or AMIs) that you can launch new instances from.

It’s hard to say that making an AMI is difficult, it’s just a few keystrokes and a coffee break while the image is prepared and stored for your future use. When you begin to manage dozens and dozens of different images this becomes a problem. It became clear to us that we needed to have one AMI to manage, and build every computer from image dynamically. To do this we needed a way to pass specific information to each instance when it started up and then have a tool customize the instance based on the specific information about it. Amazon provides a method to pass specific information to an instance (it’s called user-data) and the good folks at Alestic have made some great AMIs around it with some good documentation.

Enter the Puppet Master

We have been using a tool called Puppet for nearly two years now to manage a handful of computers. We selected that to manage the whole lot of Linux computers that we were dealing with. Puppet allows us to describe how a computer should work in a general way. We can make collections of configurations for standard things so it’s easy to reuse what we create over and over again. Our Puppet configuration is stored in a Git repository so many administrators and developers can collaborate on it and our server configurations are backed up and version controlled automatically.

Bringing it all together

Puppet decides what configuration to use for a computer based on its hostname. We use the EC2 user data to pass a new instance the address of a Puppet server and the hostname that the machine should assume. When it boots up it sets its hostname and checks in to receive the configuration that we’ve stored for it. All changes to the machine happen through Puppet so we don’t have to spend a lot of time SSHing in and customizing the machine. It’s also very easy for us to duplicate a machine for testing or whatever we need.

While we originally did this to make our lives easier as we manage more machines there turned out to be some really cool side effects:

• Excellent Security: We don’t want to store sensitive information in Puppet. This means no passwords or secret stuff. To resolve this we require all developers and administrators to use key-based authentication to get access to the computers via SSH. This is very handy and it eliminates the need to remember passwords or for an administrator to have to reset passwords for users. Someday I would love to take the next step and have all of our users and machines be part of the Monkeysphere.
• Accountability: All of our configuration is tracked in a Git repository so we can see the history about what has changed on certain hosts and who changed it. Change control occurs automatically with Puppet and it’s easy to examine and understand what has changed and why it changed.
• Repeatability: By storing all Linux computer configuration in a single place we can easily repeat what we did for one server on another. Puppet institutionalizes all of our Linux knowledge in one place and saves us time every time we have to create a new machine. If you want to see how someone has done something in the past there’s no more need to dig through emails or documentation, just look at the facts in the Puppet repository and even copy and paste it into your new configuration.
• Portability: We use Puppet to manage much more than just EC2 instances. Physical machines and virtual machines in our ESX installation are managed this way too. It gives us one tool to take care of any Linux machine we deal with. Puppet also supports other operating systems. We’re looking to expand our use of it to Mac OS X machines and maybe even Windows sometime soon.

There are certainly other ways to solve the problems we’re up against, but this is the way we chose. We did some extensive evaluation of Chef (which is a tool like Puppet) and we’ve used Rightscale before. This sort of thing is becoming very important as we manage more and more computers. I expect we’ll see a lot of exciting products, techniques, and services in this space as time goes on.

If you have any questions or comments about our setup, or would like to discuss implementing something like this for your business, leave a comment or get in touch with us. We’d love to help.

When I started searching for a way to connect an audio feed to a gateway router, I found that there were no great turnkey solutions. What I did find is that there is a way to make use of an available FXO port to bring a live feed into the system with some inexpensive parts. The solution I have deployed requires a surface mount RJ 45, a chassis mount RCA jack, and a stereo 1/8″ mini to RCA adaptor. All of these pieces can be found at a nearby electronics parts store and purchased for under $20. The FXO port and an iPod will set you back some but if you want easy control of your hold music it may be worth it.

Build the Adaptor

To make the adaptor, open up the surface mount box and figure out how you will attach the RCA jack. In one case I used a drill bit to make a hole for it, in another case I found a form factor where the notch for a cat 5 cable fit the connector perfectly. You may also find a general purpose surface mount with modular fittings where an RCA “snap in” would snap right in.

Once you’ve sorted that you will need a pair of wires from either a solid or stranded piece of cat 5. Punch these down to pins 4 and 5 of the RJ45 and solder the other side to the RCA jack.

When you are done it should look like this.

Adaptor wiring detail. Though the jack in this configuration is an RJ 45 you can use a standard RJ11 phone cord to connect between the adaptor and the FXO port.

Configuring the Voice Gateway

Configuring the voice gateway is pretty straightforward:

1. Create a DN for the MOH call out with an unused multicast address

ephone-dn 274
number 1 no-reg primary
moh ip 239.12.13.1 port 2000 out-call 899

2. Configure the available FXO port remain open to the iPod

voice-port 0/3/3
signal loopStart live-feed
input gain 2
description To MoH Live Feed

3. Create a dial peer to connect to the FXO port

dial-peer voice 55 pots
destination-pattern 899
port 0/3/3

4. Configure CME to use the multicast for MOH

telephony-service
multicast moh 239.12.13.1 port 2000

Then Shut/No shut the voice port to invoke the change.

I have been using one of the built in USB ports on our gateway here at Control Group to power the iPod which turns out to be very convenient.

Safety and Other Considerations

There are some very important caveats.

1. The adaptor I describe is potentially dangerous. If you were to connect an iPod to a POTS line or an FXS port feeding a battery signal you could damage the iPod or the device connected to it. You could also receive a serious electrical shock.
2. The use of copyrighted material for Music on Hold can be a violation of the rights of the copyright owner.
3. The adaptor will only work with one channel of audio from the iPod. If you use a mono 1/8″ to RCA cable it will short out the other channel.

These issues need to be minded if you would consider this solution. Also, keep in mind that people’s taste in music varies widely. Implement Music on Hold and you will find out just how much!

Miles Green is a Senior Network Engineer at Control Group and worked as a Musician and Recording Engineer in a previous life.

I do some work on implementing storage systems for our clients, and we’ve found that different applications have different storage requirements. For example a video post production facility — like the facility at WWE — generally needs lots of disk space that is very good at reading and writing large files at high speeds. The storage here needs to provide good streaming throughput, because high quality video files generally have high bit rates, and are being stored or played back from the disk in real-time for ingesting, editing, or playout. If the storage system is not fast enough to read or write the file in real-time, frames will be dropped. This can cause unsatisfactory media files, programs to crash, or audio and video to become out of sync.

10,000 RPM SAS disks. That's some serious storage.

Suboptimal read/write performance can become a huge problem. When we put in a new system this is something we need to test. I usually do the test with a tool called lmdd.

lmdd comes from the lmbench tools which are provided by Bitmover for benchmarking systems. lmdd is great for testing streaming bandwidth. In most of our engagements with video, we install a Stornext or Xsan filesystem so we’ll run our tests against this. lmdd will probably work on any filesystem that you can mount on your Mac or Linux computer (Leave a comment if you need a version for Mac OS X, I have one compiled).  lmdd lets us verify exactly what the maximum number of megabytes per second we can push through the storage and point us to where we need to make changes to the hardware or software configuration. I use lmdd like this :

lmdd of=/path/to/test_file count=1g

lmdd if=/path/to/test_file

The first tests write performance and the second tests read performance. More information about the syntax is available in the manual page for lmdd. The results of the command from a server I was testing looked like this:

2147.4755 MB in 6.8003 secs, 315.7914 MB/sec

lmdd is great because it’s easy to read. This result shows I could write to the filesystem at 315 megabytes per second. That’s really fast! This is from a test with a server with a lot of RAM and a special filesystem that took advantage of that cache. When I run it on my Macbook, I get a result like this:

18342.6171 MB in 376.7685 secs, 48.6841 MB/sec

So the next time you’re interested in how your storage is performing give lmdd a shot and let me know how it goes. If you’re looking for more information about storage performance testing then stay tuned; I’ll be posting about testing storage with tools that benchmark small reads and writes next.

An ASR Restore Image in Disk Utility

The disk images asr works with are the same format used by Mac OS X’s Disk Utility. This means you can do a rollout over the network and keep the master file on hand in your re-imaging kit, should one of your workstations run into trouble and need to be re-imaged over FireWire. To ensure the sanctity of the final result, disk images include an embedded checksum which is automatically verified during the deployment process. This can be a significant advantage in using asr over of Apple NetInstall, which requires its own folder-based setup of restore source files.

Disk images are also hardware agnostic for the most part. You can build your image on a Mac Mini and apply it to anything from a PowerMac G5, to an Macbook Pro so long as it can get on the same subnet as the asr host.

The usual caveats of disk imaging apply unfortunately. You’re going to have to sweat individual serial numbers if you don’t employ network or volume licensing. Settings like hostnames and non-ubiquitous local users will require individual workstation visits without centralized management. However, having asr around to do the heavy lifting means you may be able to turn a strenuous two day deployment into a breezy one day affair.