Tux the Linux Penguin

I work with Linux quite a bit at CG and it comes up in some interesting places. Consistently we find that the largest and smallest systems we work with are running Linux. If it’s a large system, like a database or storage network, Linux can provide the stability, robustness, and uptime required to make the project a success. If it’s a very small system, like an embedded display or a custom purpose device, we choose Linux because of its flexibility.

Over the last year or so, we’ve had a need to spin up many more virtual computers than we used to. It’s necessary to create a handful of  computers for each project we work on and the number of projects that we are taking on at a time is growing. Tools like EC2 and VMWare have made it very easy for us to create new virtual computers, but what about managing them? For Linux there are some great solutions. In this blog post I would like to talk about how we’re using some of the tools to deal with the large number of machines we are managing in Amazon EC2 for ourselves and our clients.

Cloud Control

The first part of the system is Amazon’s Elastic Compute Cloud (or EC2 for short). I’ve written about EC2 before, but to be really brief about it: EC2 allows you to create computers on the Internet quickly and lets you pay by the hour for their use. The instances you create are virtual machines that run in a large Xen based infrastructure that Amazon provides. To customize the software and configuration of the instances, Amazon lets you create snapshots of your computers (called Amazon Machine Images or AMIs) that you can launch new instances from.

It’s hard to say that making an AMI is difficult, it’s just a few keystrokes and a coffee break while the image is prepared and stored for your future use. When you begin to manage dozens and dozens of different images this becomes a problem. It became clear to us that we needed to have one AMI to manage, and build every computer from image dynamically. To do this we needed a way to pass specific information to each instance when it started up and then have a tool customize the instance based on the specific information about it. Amazon provides a method to pass specific information to an instance (it’s called user-data) and the good folks at Alestic have made some great AMIs around it with some good documentation.

Enter the Puppet Master

We have been using a tool called Puppet for nearly two years now to manage a handful of computers. We selected that to manage the whole lot of Linux computers that we were dealing with. Puppet allows us to describe how a computer should work in a general way. We can make collections of configurations for standard things so it’s easy to reuse what we create over and over again. Our Puppet configuration is stored in a Git repository so many administrators and developers can collaborate on it and our server configurations are backed up and version controlled automatically.

Bringing it all together

Puppet decides what configuration to use for a computer based on its hostname. We use the EC2 user data to pass a new instance the address of a Puppet server and the hostname that the machine should assume. When it boots up it sets its hostname and checks in to receive the configuration that we’ve stored for it. All changes to the machine happen through Puppet so we don’t have to spend a lot of time SSHing in and customizing the machine. It’s also very easy for us to duplicate a machine for testing or whatever we need.

While we originally did this to make our lives easier as we manage more machines there turned out to be some really cool side effects:

• Excellent Security: We don’t want to store sensitive information in Puppet. This means no passwords or secret stuff. To resolve this we require all developers and administrators to use key-based authentication to get access to the computers via SSH. This is very handy and it eliminates the need to remember passwords or for an administrator to have to reset passwords for users. Someday I would love to take the next step and have all of our users and machines be part of the Monkeysphere.
• Accountability: All of our configuration is tracked in a Git repository so we can see the history about what has changed on certain hosts and who changed it. Change control occurs automatically with Puppet and it’s easy to examine and understand what has changed and why it changed.
• Repeatability: By storing all Linux computer configuration in a single place we can easily repeat what we did for one server on another. Puppet institutionalizes all of our Linux knowledge in one place and saves us time every time we have to create a new machine. If you want to see how someone has done something in the past there’s no more need to dig through emails or documentation, just look at the facts in the Puppet repository and even copy and paste it into your new configuration.
• Portability: We use Puppet to manage much more than just EC2 instances. Physical machines and virtual machines in our ESX installation are managed this way too. It gives us one tool to take care of any Linux machine we deal with. Puppet also supports other operating systems. We’re looking to expand our use of it to Mac OS X machines and maybe even Windows sometime soon.

There are certainly other ways to solve the problems we’re up against, but this is the way we chose. We did some extensive evaluation of Chef (which is a tool like Puppet) and we’ve used Rightscale before. This sort of thing is becoming very important as we manage more and more computers. I expect we’ll see a lot of exciting products, techniques, and services in this space as time goes on.

If you have any questions or comments about our setup, or would like to discuss implementing something like this for your business, leave a comment or get in touch with us. We’d love to help.

First off, Buzz has some serious potential. Google gained an incredible amount of ground on Facebook and Twitter with this launch, and I do sense a shark-jumping moment for Facebook. Once the Google App ecosystem takes off, and social games and e-commerce get integrated, there will be a huge erosion in Facebook market share.  People want one thing, one place to go, and Gmail is already mandatory.

1. Lets start off easy here: Mobile. Google, you own the platform, how hard would it be to launch with an Android app? In the time it took to do the marketing piece on the mobile site, Google could have developed a full-fledged app. Instead I have an “above ground only” slow-loading mobile web page. HTML5 isn’t quite here yet – and Android 1.6 is not supported.
2. Two way integration! Getting Tweets in Buzz is great, but I still have to go out to Twitter or TweetDeck to post. If I had the option to choose which networks my updates appeared on from within Buzz (Facebook, Twitter, LinkedIn, etc), I would never have to leave Gmail… except for…
3. Where is Facebook? They have a strong API and a straightforward authentication service. Connect to Facebook and never make me go there again. Aggregate and publish (see above).
4. Buzz for Biz. I know it’s coming for Google Apps, but get it going already! How about full integration with LinkedIn – a CRM app would be a really interesting mash-up, as well as bringing all my connections into my address book. No more stale email addresses or phone numbers.
5. OK, I saved crazy for last. Google Profile. I am the strongest believer in an open information society – if everyone knows everything about everyone, then we are all equal. I understand that’s a little overboard, but why should I hide my information when it will only make the web more relevant to me, and get us to our ultimate destination quicker.  But with Buzz, your Google Profile went from obscurity to super relevant. So, quick inventory: Google has information about my friends, my browsing/search history (not to mention DNS info), my purchase history, my communications, the news I’m reading, and my documents. Now they want to know my age, sex, where I grew up? Are you crazy? Google is holding ALL the cards now. They keep repeating “don’t be evil” but you know what they say about absolute power… (did I mention my location?)

If history is any indicator of the future, Google will evolve and add features and services. They have done so consistently since inception, and people will choose convenience above all else, including quality and privacy. And I guess I’m one of them. See you on Buzz.

And if you disagree, see you in the comments!

In a recent Cloudsourcing event we held at our offices in the Woolworth Building we had the opportunity to meet many people interested in migrating to the cloud. The most common questions that evening seemed to revolve around “is the cloud secure?” and “will this work for my industry?” Being Google Apps partners and now AWS partners we are able to address client concerns with confidence and provide solutions.

Finding cloud-based solutions and implementing those solutions on the enterprise level can be a challenging task as these are still relatively new technologies. Keeping this in mind, part of the process includes stepping down from the cloud and relying on our team of Project Managers, Account Managers and Engineers along with AWS Consultants. With a solid team we are able to guarantee that a client’s needs and expectations are met in a timely manner.

When you think about the potential for cloud computing, you start to realize the role AWS plays in this game. Cloud computing is just now starting to meet the needs of large corporations and the data center of the future may very well be cloud based. We look forward to forging ahead in this area with Amazon Web Services.

The Server Room

Annually, the single greatest consumer of electricity in your office is likely the server room. Between the power hungry server hardware, the UPS with its constant AC-to-DC conversion and the air conditioning running 24 hours a day, 7 days a week, 365.25 days a year, all of that electricity use quickly adds up.

Hands-down, the easiest ways to save energy in your server room is to make sure your server room’s HVAC is doing its job as efficiently as possible. That means keeping the door closed at all times (and of course making sure there’s a door in the first place!) (many offices forget this part), keeping the HVAC system well-maintained, and cooling as small of a space as possible.

When buying buying new server hardware, Control Group can help you make the most efficient choices possible… this means anything from buying the right server for the right job (that eight-core print server might be overkill), to outfitting your new server with green-friendly hard drives which consume less power than standard drives. Control Group is an EPEAT Partner Reseller— EPEAT provides a “green” rating system for IT equipment based on a comprehensive criteria list.

The most efficient server room though, is no server room at all! As more and more businesses move aspects of their business to “the cloud,” they require less hardware onsite. Less hardware means lower electricity bills, which means money saved.

The Workstations

First off, if you still have any big bulky CRT monitors in your office, their time has most certainly come. They are big, ugly power hogs, and have no right to exist in 2009. No, not even on the intern’s computer in the janitorial closet. An LCD monitor will pay for itself in power consumption costs alone in under two years. When purchasing an LCD, consider looking into those which have LED-powered backlit displays. Not only do they consume even less power than LCDs using older CCFL technology, but they’re brighter, contain fewer harmful elements in their manufacturing process, and last longer to boot!

Pro-Tip: After you’ve replaced your CRT monitors, what do you do with them? Control Group works with Per Scholas, which is a local organization that recycles and re-uses computer equipment and works with New York City schools to provide students with their own computers as well as training.

Another great example of a small change that can make a big difference would be to make it office policy to power down computers at the end of the day (or at least the end of the week). For even more energy savings, consider flipping the switch on that power strip your computer is plugged into on top of powering down.

And for those employees who say that shutting down hurts their productivity, tell them to use Windows’ “hibernate” feature rather than the “shut down.” That will allow them to pick up right where they left off, with all of their AutoCAD windows and dozens of email drafts right where they left them.

The Office

One way Control Group saves energy around the office is that we have a “no personal printers” policy. We have three printers in the office: a black and white laser printer, and a color laser printer and a large format plotter to help us better support our architecture clients.

Not only does this centralized printing solution help save energy by avoiding having dozens of printers plugged in and on standby 24 hours a day, but it saves money on supplies—no more $18 ink cartridges—and saves us time from having to troubleshoot problematic printers (there’s a reason desktop printers are so cheap… they’re cheaply made and break constantly).

Perhaps most importantly, this printing setup also saves a lot of paper. Duplex printing (two-sided) is enabled on our laser printers and we have a dedicated “scrap paper” tray that we keep loaded with scrap paper, for those times when you need to print something but it doesn’t matter how it looks.

We may not be paperless yet, but we’re well on our way.

Finally, with all of the energy that your office is saving, consider moving over to wind power. Here in New York, ConEd offers a very competitive Wind Power for Business package. It’s 100% renewable energy and it’s easy to switch over.

While wind power may be 10-15% more expensive than coal, depending on how much you’ve managed to cut your overall power consumption, it’s entirely possible your company is still spending less on electricity than before even while using wind power. And on top of that, you can tell your customers you’re ahead of your competitors in reducing your company’s impact on the environment.

We have been doing some work with with Amazon’s Elastic Computing Cloud (EC2) which allows us to create virtual machines in the cloud in a few seconds. These are great for hosting websites, and what’s cool about them is that if you get Slashdotted or experience a similar unexpected spike in traffic you can create new hosts immediately. Recently Amazon added a new service called Elastic Load Balancing (ELB) which can distribute load across hosts. We’ve been looking at this for some of our recent development and infrastructure projects.

I just read this description of how ELB works by Shlomo Swidler from his Cloud Developer Tips blog. It’s a great reference.

You pay for ELB by usage just like everything else at AWS. From Amazon: “You are charged at $0.025 per hour for each Elastic Load Balancer, plus $0.008 per GB of data transferred through an Elastic Load Balancer.” For reference, on a deployment project in 2008 our Engineering team used a Cisco load balancer which I imagine cost a few thousand bucks.

Cost isn’t the only advantage. These can be created and destroyed quickly and remotely, allowing us to work more efficiently and spend less time visiting data centers in the middle of nowhere. This leads to improved quality of service for our clients as we can spend more time consulting on future technology growth plans and less time troubleshooting servers in cold, loud data centers.

This blog post brought to you by the iced coffee I am enjoying in the comfort and quiet of my office while deploying virtual machines!

Tom Mills from Google and Sean Finnegan from Mozy will be giving an in-depth review of their offerings and how they fit into an agile, post-recession office technology strategy.

I’ll be giving a brief overview of how I think we arrived at this point in IT and what it means for creative, innovative firms that are trying to do more with less.

In an effort to get my thoughts together and get some feedback, I’m using this blog post as a draft for the event.

Let me start off by giving a brief overview of our services, and then a little history about the evolution of our offerings:

We provide a number of technical services for our clients in the areas of infrastructure, application development, and industry-focused workflow consulting. As this is New York, we work with a number of creative firms; media, architecture, publishing, and design companies, as well as some key clients in the financial sector. We strive for long-term relationships with our clients, many of whom we’ve worked with for close to a decade. We have installed and managed hundreds of servers, network devices and application suites, but more recently we’ve been focusing on helping our clients select, migrate to, integrate, and manage Cloud-based services.

Since the 1990s and the introduction of pervasive bandwidth, we’ve gone through a number of permutations of the remote server/client model, and much has been written about the benefits and the irony of the shift back to the mainframe/thin client structure of the 1960s. Now everyone is talking about the future of ‘The Cloud‘; a vast array of computing resources, abstracted and presented as a single source to the consumer.

At the turn of the century, we found most small to mid-sized businesses with a pure Local Area Network (LAN), typically comprised of in-house mail – most likely Exchange – and a few other local services: file, print, etc.  A lot of these firms had an internal IT staff or a dedicated consultant to manage their servers, tape backup, networks, and desktops. Only a few were pushing the envelope by leveraging Application Service Providers (ASPs) to deliver back office services.

The risks with this situation were obvious. These systems mostly depended on a single Internet connection, a single building, and a single individual, prone to career changes and untimely vacations.  Remote access to these in-house services was expensive to do right and applications rarely worked as well remotely as they did in the office.

Over the next five years, we saw a gradual shift towards ‘Hosted Applications’. This typically came in the form of a service provider taking a LAN-based solution like Exchange or SharePoint out of the office and putting it in a data center. In conjunction with this change, we saw the IT services industry begin to shift its focus from in-house IT, or consultants, to managed services – companies providing regular systems management remotely.

There were some benefits to this offering: critical applications were not dependent on intermittent Internet connections or over-heated server rooms. Flaky consultants were traded for predictable management services and cost became as regular as the electric bill.

But there were still problems. We had the same old model of doing things, only it was moved out of the business’s office and into the provider’s.  Services that were built for an onsite installation and LAN speeds were shifted to a remote location – not always producing the best results. Access to applications designed for the LAN was sometimes unacceptable because of bandwidth and latency. In a similarly narrow view of the problem, Managed Service companies focused on monitoring systems and patching software, maintaining the status quo, without looking at the big picture, or driving the business forward.

Now the next generation of IT services is coming along and delivering on the promise of on-demand, scalable solutions. These services are web-native, built for the Cloud and multi-tenant environments.

As services like Google Apps and Mozy were built for the web – not re-purposed LAN applications – they deliver exceptional performance and remain very flexible. Control Group has designed our support and project services in a similar way. Our services are built to function efficiently remotely – scaling up when our clients need it, and going away when they don’t – and also to be flexible and innovative, driving business forward rather than maintaining the status quo.

Using the cloud paradigm, we act as a single source of technology for our clients. We help them run more efficient, profitable businesses by weaving an ever growing selection of web-based services, traditional IT, and industry expertise together, to provide an flexible, competitive business platform.

Your Data Here

I was having a conversation recently with one of our consultants, David Rocamora, as our team broke down the contents of a start-up’s data center, when I came to the realization that we may have built our last data center.

Now we don’t really build data centers, but we have racked a lot of servers, storage and network gear around the world in tier 1 data centers for our clients. With the change in the economy and the maturity of several cloud services, the data center that we know and love looks like it’s going the way of the wood pulp newspaper.

Certainly more data centers than ever are being built: Google, Microsoft, Apple, and Amazon are soaking up gigawatts (petawatts?) of power like never before. But the days of dressing down in jeans and a sweatshirt and going out to New Jersey or Colorado to rack servers in an earsplitting, freezing cold warehouse of caged servers and blinking lights seems to be drawing to a close.

Some numbers to consider:

This ‘scaled down’ dot com we were consolidating, had spent about $500,000 on a few racks of some amazing equipment (Sun, Check Point, etc) only to find out 4 months later they didn’t need it. Pennies on the dollar. The contract for the floor space, power, and bandwidth goes on for another 8 months and I bet you could buy a modest BMW for what it’s costing them.

Now a similar sized start-up we just started working with on a really innovative interactive image platform, is using the Amazon cloud and RightScale and is spending about $50,000 a year on cloud services. No capital outlay.

Flexibility?

The cloud is infinitely more flexible, we can put servers in Europe in a matter of minutes, set up high availability zones in different regions around the country, and if they start to get swamped with business like we think they will, we’ll be able to turn up as many servers as they need in a few minutes time.

What if the dot com of 4 months ago took off? Order servers, spend capital. Put in a request for more bandwidth, more cage space. Days, maybe weeks go by. Then get out the jeans and sweatshirt and head over to the data center. Earplugs. Man, those servers are loud.

You get the picture. But this is happening so fast it’s amazing. Six months ago when the dot com was building its data center, the Amazon cloud was still in beta, with no SLA, and it wasn’t an option for a serious start up. Today, building a data center isn’t an option for a serious start-up.

Now we have availability zones, provisioning and monitoring tools, the ability to drop terabytes of data into the cloud — shipped through FedEx! But the real promise is the rich API and the spirit of community innovation. Companies like RightScale are finding a niche in the cloud, developing something really valuable, and then selling it as a simple service that makes our lives so much easier.

It’s exciting to see this happen so fast.  To avoid being crushed by this wave, as an IT team, you need to really stay on top of it. IT in our part of the ecosystem is becoming more the art of selecting, deploying, integrating, managing and supporting cloud based services, and much less the craft of building serious web infrastructure.

It’s a little sad for the hardware geek in all of us, saying goodbye to the roar of the servers, putting down the Velcro ties and picking up some slick provisioning and automation scripts. But I think we could get used to deploying 50 servers in a few keystrokes from a quiet, comfortable seat in the office.