Our friends at Debian-NYC will be running a workshop at Control Group on October 27th at 6:30 pm about what’s in a Debian package. Debian’s packaging system is a huge part of why it’s such a great operating system (the packaging system is also why Ubuntu is so great). This workshop will give you a tour of how it works:

This workshop will provide advanced theory useful for people modifying or creating packages. For people modifying packages, you’ll learn many typical motifs and about various build systems. For creating packages, you’ll be much better prepared to read and understand guides a deep level. However, this is still not a step-by-step guide in “how to build packages”, but will get you very close to there.

The Debian-NYC series of workshops as a whole is designed to introduce Debian, Debian tools and techniques, and the Debian community as well as provide skills to attendees. This workshop is targeted towards technically-skilled people who would like knowledge of the Debian packaging system and how to contribute back to the Debian community. Most of the material of these workshops will be applicable to other Debian-derived distributions, such as Ubuntu.

If you’re interested, please RSVP. These workshops have always been really fun and informative. I hope to see you there!

Share this:

There have been some changes since my last post about CG’s coffee maker. We’ve gotten a new “machine” that’s much simpler. It’s easier to understand and less likely to break down. It’s a 10 cup Chemex that we use with an electric kettle.

The Chemex is really easy to use. Heat up water in the kettle, position a filter in the top of the vessel, add coffee grounds, and pour hot water over it (a little at first, to let it bloom, then as much as you want to make). Sweet Maria’s has an excellent guide on making coffee with a Chemex.

This is week two of our Chemex experience. So far the kitchen is cleaner, the brew is stronger, and the coffee-notify mailing list is more active. Sure, it takes a little longer to make a pot, but I feel like it makes us more mindful of what we’re doing. Apparently we’re not the only ones that have recently switched to the Chemex.

I suppose the next steps are to grind our own beans (by hand of course). We’ll see what we can do. Stay caffeinated my friends.

Share this:

This picture is an exclusive behind-the-scenes look at Mercedes-Benz Fashion Week. It wasn’t taken backstage, on the runway, or at an after party. This image comes from the monitoring console in the CG war room where I was working with a few other engineers on a rapid response to a load increase on mbfashionweek.com, the event’s new website.

Since Mercedes-Benz Fashion Week only runs for a week, we needed to boost the website infrastructure quickly so that it could continue to provide updates and information about the event for seven days despite heavy traffic.

If this was a few years ago and we were using a traditional web-hosting infrastructure this would be difficult — maybe even impossible. But fortunately this is 2010 and IMG Fashion, the company responsible for the event, was using Amazon Web Services.

The application that powered their website was good, but there were some unexpected issues preventing it from performing well in a very high traffic environment. There was no time to profile, troubleshoot, and retool parts of the application. The fastest solution to the problem was to create more web server instances and to distribute the traffic to them.

We were already using an Elastic Load Balancer to spread traffic between their two web serving instances, so adding new instances was simple. We created new EC2 instances in several different Availability Zones to ensure that the site would stay online no matter what happened. A sophisticated system built into the application’s content management system (CMS) kept content on the different web servers consistent. And we used Amazon’s Relational Database Service to handle the database tier.

We increased the amount of servers several times over the course of the event to handle the incredible amount of traffic on the website. You can see two of those increases in the graph at 15:00 and 20:00. After the event was over and website traffic slowed down, we were able to reduce the infrastructure and costs. This proves that systems for events like Mercedes-Benz Fashion Week are perfect candidates for cloud computing — organizations can pay for exactly the amount of compute resources they need and no more.

Mercedes-Benz Fashion Week is over now. The models and designers have moved on and their EC2 instances are spinning down. And, while trends in computing seem to change as often as trends in fashion, I think we’ll see cloud computing and scalable websites stick around for a few more seasons.

Share this:

Our coffeemaker is broken.

Coffee is a big deal at CG. Most of our geeks pride themselves on being caffeinated and with the coffee machine down panic is on the rise.

The thing about this is that it’s not the actual coffeemaker that is broken, it’s the grinder built into the coffee machine that is having issues. We have a fancy machine that grinds coffee right before brewing it. When it’s working it’s pretty magical — nothing tastes quite like freshly brewed coffee made from freshly ground beans. It happens automatically and other than the noise from the grinder we don’t even know that it’s there.

When the machine is down it’s obvious. Some of us need coffee to work. We are dependent on that machine.

Thinking about the caffeine situation in the office made me wonder about other pieces of technology that we’re dependent on. Our email software runs in the cloud on Google’s computer. Our data traverses networks and is converted to light, microwaves, electricity and back again before it arrives at its destination. Do you know how an email sent from your phone is routed to its destination? What other things does technology do for us automatically that we don’t notice? Heck, I can’t even remember my wife’s phone number — my phone does it for me.

Someone sent me an article the other day entitled, “The Cloud Fails Again.” In the article, John Dvorak complains that a power outage left him unable to function because all of his data and services existed in the cloud and not in his own machine. He goes on to describe a “priesthood” of systems administrators that has existed since the early days of computing whose sole purpose is to “beat back the individualism” that desktop computers brought to all of us.

I was unaware that this cabal existed (if you are a member, please send me an invite) and I feel like the advances that technology has brought us in life, business and communication are really amazing. We live in a magical world. But even though the advances are great, they have made us completely dependent on technology. I think Dvorak’s article is a pretty good example for people who rely on technology and refuse to invest in their own infrastructure. In other words, we need to understand what we’re using so that we can evaluate the risks and benefits of using it.

Control Group’s mission is to help people and their organizations better understand and utilize their technology so they can be more efficient. That’s why Control Group is a great place to work — even when the coffee machine is down.

Our engineers were able to create a temporary workaround for the coffee situation. We’re also not exactly stranded in a coffee-free wasteland: Kaffe 1668 and The Blue Spoon are within walking distance. So, no worries, we’ll stay jittery.

Image via coffeeaddict/Flickr

Share this:

I wear a few different hats at Control Group. Very often I will join the team on projects that have something to do with post production, storage networking, product development and web application development. At a glance these things seem quite different. What do these very different projects have in common? Linux!

Tux the Linux Penguin

I work with Linux quite a bit at CG and it comes up in some interesting places. Consistently we find that the largest and smallest systems we work with are running Linux. If it’s a large system, like a database or storage network, Linux can provide the stability, robustness, and uptime required to make the project a success. If it’s a very small system, like an embedded display or a custom purpose device, we choose Linux because of its flexibility.

Over the last year or so, we’ve had a need to spin up many more virtual computers than we used to. It’s necessary to create a handful of  computers for each project we work on and the number of projects that we are taking on at a time is growing. Tools like EC2 and VMWare have made it very easy for us to create new virtual computers, but what about managing them? For Linux there are some great solutions. In this blog post I would like to talk about how we’re using some of the tools to deal with the large number of machines we are managing in Amazon EC2 for ourselves and our clients.

Cloud Control

The first part of the system is Amazon’s Elastic Compute Cloud (or EC2 for short). I’ve written about EC2 before, but to be really brief about it: EC2 allows you to create computers on the Internet quickly and lets you pay by the hour for their use. The instances you create are virtual machines that run in a large Xen based infrastructure that Amazon provides. To customize the software and configuration of the instances, Amazon lets you create snapshots of your computers (called Amazon Machine Images or AMIs) that you can launch new instances from.

It’s hard to say that making an AMI is difficult, it’s just a few keystrokes and a coffee break while the image is prepared and stored for your future use. When you begin to manage dozens and dozens of different images this becomes a problem. It became clear to us that we needed to have one AMI to manage, and build every computer from image dynamically. To do this we needed a way to pass specific information to each instance when it started up and then have a tool customize the instance based on the specific information about it. Amazon provides a method to pass specific information to an instance (it’s called user-data) and the good folks at Alestic have made some great AMIs around it with some good documentation.

Enter the Puppet Master

We have been using a tool called Puppet for nearly two years now to manage a handful of computers. We selected that to manage the whole lot of Linux computers that we were dealing with. Puppet allows us to describe how a computer should work in a general way. We can make collections of configurations for standard things so it’s easy to reuse what we create over and over again. Our Puppet configuration is stored in a Git repository so many administrators and developers can collaborate on it and our server configurations are backed up and version controlled automatically.

Bringing it all together

Puppet decides what configuration to use for a computer based on its hostname. We use the EC2 user data to pass a new instance the address of a Puppet server and the hostname that the machine should assume. When it boots up it sets its hostname and checks in to receive the configuration that we’ve stored for it. All changes to the machine happen through Puppet so we don’t have to spend a lot of time SSHing in and customizing the machine. It’s also very easy for us to duplicate a machine for testing or whatever we need.

While we originally did this to make our lives easier as we manage more machines there turned out to be some really cool side effects:

• Excellent Security: We don’t want to store sensitive information in Puppet. This means no passwords or secret stuff. To resolve this we require all developers and administrators to use key-based authentication to get access to the computers via SSH. This is very handy and it eliminates the need to remember passwords or for an administrator to have to reset passwords for users. Someday I would love to take the next step and have all of our users and machines be part of the Monkeysphere.
• Accountability: All of our configuration is tracked in a Git repository so we can see the history about what has changed on certain hosts and who changed it. Change control occurs automatically with Puppet and it’s easy to examine and understand what has changed and why it changed.
• Repeatability: By storing all Linux computer configuration in a single place we can easily repeat what we did for one server on another. Puppet institutionalizes all of our Linux knowledge in one place and saves us time every time we have to create a new machine. If you want to see how someone has done something in the past there’s no more need to dig through emails or documentation, just look at the facts in the Puppet repository and even copy and paste it into your new configuration.
• Portability: We use Puppet to manage much more than just EC2 instances. Physical machines and virtual machines in our ESX installation are managed this way too. It gives us one tool to take care of any Linux machine we deal with. Puppet also supports other operating systems. We’re looking to expand our use of it to Mac OS X machines and maybe even Windows sometime soon.

There are certainly other ways to solve the problems we’re up against, but this is the way we chose. We did some extensive evaluation of Chef (which is a tool like Puppet) and we’ve used Rightscale before. This sort of thing is becoming very important as we manage more and more computers. I expect we’ll see a lot of exciting products, techniques, and services in this space as time goes on.

If you have any questions or comments about our setup, or would like to discuss implementing something like this for your business, leave a comment or get in touch with us. We’d love to help.

Share this:

I wanted to add some thoughts to Chris’s post about Flash and HTML5. However I should preface this post by saying that HTML5 supporting video is really cool, both technically and because HTML5 is an open standard that anyone can implement for free. As we all know, for the last several years, Flash has been the de facto choice for online video delivery. Flash support on different platforms has been pretty good, but end users still don’t have total flexibility depending on their OS. Until recently, Flash on Linux has been about a version behind the release for Windows or OS X. Even now, Adobe only releases a player for x86, and the x86_64 version is unsupported beta software.

Everyone seems to be touting HTML5 video as the “open” alternative to the proprietary Flash plugin required for .flv playback in the browser. But how open is H.264, the codec that powers HTML5 video, and the current pick for encoding video for online delivery? Using H.264 as the codec behind HTML5 video sours things a bit for me. H.264 is encumbered by software patents; to develop or distribute a player or encoder for H.264 you might have to pay a licensing fee to MPEG-LA. Even though MPEG LA announced last week (PDF) that H.264 will remain fee-less for free internet video through 2016, this is not the same as being free or open. MPEG-LA can still go after people that produce the software to encode or decode H.264. And MPEG-LA is not just one organization, it’s a collection of patent holders that have their own agendas.

All this is a bit of a slap in the face to the open standards that power the web. Imagine if you had to pay a half million dollars to create or display JPEGs, GIFs, or HTML… The only people that would be able to afford to make software for the web would be huge companies. But what are our alternatives? Beyond Ogg Theora and Matroska, the pickings are slim. These codecs are open and free, but not necessarily better than H.264. Plus it would be next to impossible to compete with the marketing machine of Apple behind H.264.

Open and free standards have been what has made the Internet successful since its inception. I think it’s important that users understand this so that the Internet of the future cannot be controlled by corporations with enough cash to cover licensing fees.

Share this:

We have been doing some work with with Amazon’s Elastic Computing Cloud (EC2) which allows us to create virtual machines in the cloud in a few seconds. These are great for hosting websites, and what’s cool about them is that if you get Slashdotted or experience a similar unexpected spike in traffic you can create new hosts immediately. Recently Amazon added a new service called Elastic Load Balancing (ELB) which can distribute load across hosts. We’ve been looking at this for some of our recent development and infrastructure projects.

I just read this description of how ELB works by Shlomo Swidler from his Cloud Developer Tips blog. It’s a great reference.

You pay for ELB by usage just like everything else at AWS. From Amazon: “You are charged at $0.025 per hour for each Elastic Load Balancer, plus $0.008 per GB of data transferred through an Elastic Load Balancer.” For reference, on a deployment project in 2008 our Engineering team used a Cisco load balancer which I imagine cost a few thousand bucks.

Cost isn’t the only advantage. These can be created and destroyed quickly and remotely, allowing us to work more efficiently and spend less time visiting data centers in the middle of nowhere. This leads to improved quality of service for our clients as we can spend more time consulting on future technology growth plans and less time troubleshooting servers in cold, loud data centers.

This blog post brought to you by the iced coffee I am enjoying in the comfort and quiet of my office while deploying virtual machines!

Share this:

As I’ve mentioned in previous posts about testing storage performance with lmdd and bonnie++, different applications require different characteristics from storage to provide the best performance. I’ve highlighted some tests that are good for large streaming files like video, and small file transactions like databases or mail servers. Today I want to look at a tool that runs a series of tests in many different ways to provide you with a holistic view of what the storage can and can’t do.

This tool is called iozone. iozone is open source and runs on a ton of operating systems (including Windows). It runs several tests which can take some time to complete but provide the best overall view of the capabilities of a piece of storage. For instance, iozone runs a write test with files of different sizes and with different size records (the amount of data written at a time). It does this over and over again with writes, reads, random writes, random reads, and so forth. Since it’s running all these tests you can see what sorts of operations will have good performance and which ones will not perform so well. Check out the iozone documentation here.

One really great thing about iozone is that the output it generates can be easily placed in a spreadsheet program like Excel to generate a great 3d diagram describing your storage. Here’s a diagram I generated from some tests on a Linux server.

Results of a write test with iozone

This particular server performed quite well with large files and a record size around 1 MB (interesting to note, this is the same storage from the lmdd post. Notice that the parameters I tested with there are the same as the best write that this disk can do according to iozone!).

If you’ve been following my posts on storage performance testing I hope you’ve learned about some new tools that you can use to see what’s going on. I use these on every deployment to make sure we’re giving our clients solutions that they can depend for performance and reliability. As always, let me know if you have any questions about these tools. Happy testing!

Share this:

Last time I posted about checking disk performance with lmdd. lmdd is great for checking streaming throughput, but what if you have a different kind of application? Every application accesses storage in different ways: with video we need to be able to provide constant throughput when writing a lot of data to the disk, but other applications may have different storage needs. For example, a database can make lots of very small changes to the data on disk in a short period of time. The best performing disk for a database will probably need to have very low seek time and good transactional performance.

bonnie++ is a series of file system tests that focuses on small files. It was designed to behave like a mail server does, creating and dealing with lots of small files (emails). bonnie++ is easy to run and outputs a CSV file that you can view with something like Excel. With the bon_csv2html command you can quickly generate html pages from the CSVs.

Here’s the output from bonnie++ running on a server:

The HTML output of bonnie++ on a Linux Server

At first glance the output can seem quite cryptic, but if we look close we can see that this provides us a great amount of information about latency and speed on different filesystem operations. I generally run this several times as I make changes to verify that the storage is providing the right performance characteristics. Tweaking a file system to make file system operations happen a few milliseconds faster may seem ridiculous, but in some environments it can make a huge difference.

Next time I’ll post about a tool that’s new to me but can test a disk in so many different ways I’m planning to run it on every system we install from now on.

Share this:

Outside My Hotel in Malaysia – Do Not Feed The Monkeys!

I know the last time I posted here I said I’d be following up with another technical post, but instead I thought I’d share an experience I just had as I took a last minute trip for a client.

Normally if I take a trip it’s no big deal. I can write a blog post from where ever I go. My email is online, this blog is online, if I need to access something in my office I can just use our VPN to get connected. To use any of these I’d just need to have an Internet connection. Unfortunately, this wasn’t the case when last week I went to a fairly remote part of Malaysia.

A few coworkers and I were trying to make last minute adjustments to a product that one of our clients is launching. At first I wondered why even send us out there when we can get remote access or talk someone through it on the phone. When I arrived onsite I realized why this wasn’t an option — getting connected is near impossible there. We could head to a coffee shop and get some free WiFi, but with over twenty hops to servers in the United States and a twelve hour time difference, getting anything done was difficult.

The lack of connectivity was challenging. One of my responsibilities was interfacing with the local IT department and writing some scripts to integrate the client’s system with existing systems and processes. I quickly realized how much I depend on online references and documentation. When you can barely get connected to look up the answer to a question about syntax you really have to use your head. Not to mention, each software build for the project is about 300 megabytes and getting this from our office in New York was difficult and time consuming.

The idea of ubiquitous Internet connectivity is something that we take for granted. As connection speeds get faster and more reliable we lose efficiencies that we once had. I learned that the Internet is really an extension of my knowledge and a valuable tool that I need to do my job. Being cut off from it was an interesting and overall positive experience. Solving every problem by thinking and working it through was difficult and took more time, but genuinely figuring things out for myself was very rewarding.

Towards the end of my time there we found a cell phone store that sold GSM modems and prepaid 3G SIM cards that allowed us to get connectivity. While this does make the job a lot easier, I’m glad I had the experience of being mostly cut off from the rest of the net — something that will surely happen less often as the world becomes better connected.

Share this: