If the amount of data in your enterprise is overwhelming and/or you’re looking for ways to process said data more efficiently, then Hadoop and Amazon Elastic MapReduce may be your answer.

MapReduce frameworks allow developers without much knowledge on distributed computing to write applications that take advantage of distributed resources. Hadoop MapReduce is an implementation of such a model.

Background:

Recently, we developed a web asset delivery service for one of our clients that would allow businesses to display high quality assets from a CDN in their websites for a monthly fee. Users’ accounts would be associated with bandwidth limits based on different account levels associated with a pricing model. This meant that we needed a way to provide users with information on monthly bandwidth utilization across their websites in order to defend the pricing model. The solution to this was to implement web log parsing using Hadoop’s MapReduce framework in conjunction with Amazon’s cloud-based Elastic MapReduce service.

Here’s how Hadoop MapReduce works:

Hadoop MapReduce is a Java-based framework that allows you to write applications that process high volumes of data in parallel clusters. Hadoop uses a distributed file storage system called Hadoop Distributed File System (HDFS) to store large amount of data across multiple nodes. It supports most major platforms and MapReduce programs can be written in Python, Ruby, php, Pig, etc., in addition to Java.  Using Hadoop, we were able to write a simple Java program that could easily parse through raw data in log files collected by the CDN and filter relevant bandwidth utilization information.

The basic idea around a map-reduce model is that you write two functions — map() and reduce() — to divide up your programming tasks and let the framework manage most of the crunching. Map and reduce functions take in key-value pairs (using data types that implement Hadoop’s Writable interface) as the input and output. When you start a map reduce process, you pass in a data file in HDFS as the input. Hadoop divides up the inputs into smaller pieces that the map function can consume. Likewise, the outputs of the map function are grouped together in logical chunks by Hadoop and sent to the reduce function for processing.  Both map and reduce functions can run in parallel — Hadoop can distribute the tasks across various clusters of nodes.

In our case, we simply pass the log file(s) (copied to HDFS) as input to the map reduce program.  Hadoop merges all log files specified and serializes each log entry to the datatype expected (Text) before passing them as inputs to the map reduce tasks. Our map function then parses each long entry individually and stores the relevant data (bandwidth info) in a HashMap type object (MapWritable) which is then sent as another key-value pair (<asset path – MapWritable object>) for the reduce function to work with. The reduce function then aggregates the data based on user accounts, date, user agent, etc. and saves it to a database (Amazon RDS Database). We can then query the database to pull all types of information around utilization and send out notifications to users, for example, if their account is over the monthly cap, etc.

Below is the structure of a sample map reduce program written in Java:

public class LogProcessor {

  public static class LogMap
            extends Mapper<LongWritable, Text, Text, MapWritable> {
    public void map( LongWritable key, Text value, Context context ) {
      MapWritable logEntry = new MapWritable();
      //parse log file
      ...
      Text key = new Text();
      //key = resource-path;

      context.write( key, logEntry);
    }
  }

  public static class LogReduce
            extends Reducer<Text, MapWritable, DBWritable, NullWritable> {
    public void reduce( Text key, Iterable<MapWritable> values, Context context ) {
      while(values.iterator().hasNext()) {
        MapWritable entry = values.iterator().next();
        //process entry and write to db
        ...
      }
    }
  }

  public static void main(String[] args) {
    // Set up a new mapreduce job
    Job job = new Job();
    job.setJarByClass(LogProcessor.class);    //register the main class

    FileInputFormat.addInputPath( job, new Path(<Input file path>) );
    FileOutputFormat.setInputPath( job, new Path(<output file path> ) );

    job.setMapperClass( LogMap.class );
    job.setReducerClass( LogReduce.class );

    job.setOutputKeyClass( Text.class );
    job.setOutputValueClass( MapWritable.class );

    job.waitForCompletion(true) ? System.exit(0) : System.exit(1);
  }
}

The program is packaged in a jar file (with dependencies) that Hadoop can run.

And here’s how to utilize Amazon’s Elastic MapReduce service to run the program:

At Control Group, we leverage Amazon’s cloud based infrastructure heavily in lots of projects. It basically allows us to cost effectively (pay by usage) deploy applications that need to scale up very easily. Amazon’s Elastic MapReduce service is the perfect fit for running our MapReduce application described above. It’s easy to set up, and it also shields off some of the infrastructure/maintenance issues around running Hadoop.

In a nutshell, the Elastic MapReduce service runs a hosted Hadoop instance on an EC2 instance (master), and it’s able to instantly provision other pre-configured EC2 instances (slave nodes) to distribute the MapReduce process, which are all terminated once the MapReduce tasks complete running. Amazon allows us to specify up to 20 EC2 instances for data intensive processing. It also provides the option to upgrade your Elastic MapReduce to increase EC2 instance count.

So to run the map reduce service, we create a new “Job Flow” via the AWS console, the command line utility (ruby based) or an API provided by Amazon. A job flow is a set of steps that Elastic MapReduce runs. You basically provide some configuration information (number of EC2 instances to use and bootstrap actions) and the location of your map reduce program ( usually an Amazon S3 bucket path). Job flow records/logs can be viewed at the AWS console. You can also explicitly instruct Elastic MapReduce to keep the master EC2 instance alive for debugging purposes – you can then ssh into the instance to check the log files created by Hadoop, etc.

In summary, Hadoop’s MapReduce framework allows us to write simple applications that process high volumes of data in a distributed computing environment while Amazon’s MapReduce service provides a cost-effective means of implementing such a solution.

Share this:

For anything cloud, we always turn to our resident expert, Dave Rocamora. Dave’s a senior consultant at Control Group and has worked with clients like The Daily Show, thelab and Mercedes-Benz Fashion Week. (Read more about how Dave used the cloud to manage load increases for Fashion Week’s website here.)

Recently, Dave talked to a reporter for Inc. Technology about how small businesses should implement computing resources. His advice? Look for a long trial and involve stakeholders.

Read his comments in full here.

Share this:

This picture is an exclusive behind-the-scenes look at Mercedes-Benz Fashion Week. It wasn’t taken backstage, on the runway, or at an after party. This image comes from the monitoring console in the CG war room where I was working with a few other engineers on a rapid response to a load increase on mbfashionweek.com, the event’s new website.

Since Mercedes-Benz Fashion Week only runs for a week, we needed to boost the website infrastructure quickly so that it could continue to provide updates and information about the event for seven days despite heavy traffic.

If this was a few years ago and we were using a traditional web-hosting infrastructure this would be difficult — maybe even impossible. But fortunately this is 2010 and IMG Fashion, the company responsible for the event, was using Amazon Web Services.

The application that powered their website was good, but there were some unexpected issues preventing it from performing well in a very high traffic environment. There was no time to profile, troubleshoot, and retool parts of the application. The fastest solution to the problem was to create more web server instances and to distribute the traffic to them.

We were already using an Elastic Load Balancer to spread traffic between their two web serving instances, so adding new instances was simple. We created new EC2 instances in several different Availability Zones to ensure that the site would stay online no matter what happened. A sophisticated system built into the application’s content management system (CMS) kept content on the different web servers consistent. And we used Amazon’s Relational Database Service to handle the database tier.

We increased the amount of servers several times over the course of the event to handle the incredible amount of traffic on the website. You can see two of those increases in the graph at 15:00 and 20:00. After the event was over and website traffic slowed down, we were able to reduce the infrastructure and costs. This proves that systems for events like Mercedes-Benz Fashion Week are perfect candidates for cloud computing — organizations can pay for exactly the amount of compute resources they need and no more.

Mercedes-Benz Fashion Week is over now. The models and designers have moved on and their EC2 instances are spinning down. And, while trends in computing seem to change as often as trends in fashion, I think we’ll see cloud computing and scalable websites stick around for a few more seasons.

Share this:

Not happy with the available options for collaborative video editing tools, Novacut is leveraging open-source software and developing a free video editor with a unique distributed design:

• Distributed workflow – collaboratively edit video with other artists over the Internet
• Distributed storage – seamlessly store and synchronize video files across multiple computers and the cloud
• Distributed rendering – seamlessly spread rendering and encoding across multiple computers and the cloud

Projects like this can be really inspiring to see mature. In their own words:

So we’re not re-inventing the wheel. We’re re-using existing open-source components wherever possible. All the really hard work has been done for us already, for which we are very thankful. At the same time, I don’t know of another editor that has the same use-case priorities that we have. Different video editors *should* have different priorities because there are distinct use-cases worth optimizing for.

We dropped some money in the Kickstarter, maybe you’d like to too. Today is the last day to help them reach their goal of $25,000.

Contribute to the project at Kickstarter.

Share this:

Our coffeemaker is broken.

Coffee is a big deal at CG. Most of our geeks pride themselves on being caffeinated and with the coffee machine down panic is on the rise.

The thing about this is that it’s not the actual coffeemaker that is broken, it’s the grinder built into the coffee machine that is having issues. We have a fancy machine that grinds coffee right before brewing it. When it’s working it’s pretty magical — nothing tastes quite like freshly brewed coffee made from freshly ground beans. It happens automatically and other than the noise from the grinder we don’t even know that it’s there.

When the machine is down it’s obvious. Some of us need coffee to work. We are dependent on that machine.

Thinking about the caffeine situation in the office made me wonder about other pieces of technology that we’re dependent on. Our email software runs in the cloud on Google’s computer. Our data traverses networks and is converted to light, microwaves, electricity and back again before it arrives at its destination. Do you know how an email sent from your phone is routed to its destination? What other things does technology do for us automatically that we don’t notice? Heck, I can’t even remember my wife’s phone number — my phone does it for me.

Someone sent me an article the other day entitled, “The Cloud Fails Again.” In the article, John Dvorak complains that a power outage left him unable to function because all of his data and services existed in the cloud and not in his own machine. He goes on to describe a “priesthood” of systems administrators that has existed since the early days of computing whose sole purpose is to “beat back the individualism” that desktop computers brought to all of us.

I was unaware that this cabal existed (if you are a member, please send me an invite) and I feel like the advances that technology has brought us in life, business and communication are really amazing. We live in a magical world. But even though the advances are great, they have made us completely dependent on technology. I think Dvorak’s article is a pretty good example for people who rely on technology and refuse to invest in their own infrastructure. In other words, we need to understand what we’re using so that we can evaluate the risks and benefits of using it.

Control Group’s mission is to help people and their organizations better understand and utilize their technology so they can be more efficient. That’s why Control Group is a great place to work — even when the coffee machine is down.

Our engineers were able to create a temporary workaround for the coffee situation. We’re also not exactly stranded in a coffee-free wasteland: Kaffe 1668 and The Blue Spoon are within walking distance. So, no worries, we’ll stay jittery.

Image via coffeeaddict/Flickr

Share this:

I wear a few different hats at Control Group. Very often I will join the team on projects that have something to do with post production, storage networking, product development and web application development. At a glance these things seem quite different. What do these very different projects have in common? Linux!

Tux the Linux Penguin

I work with Linux quite a bit at CG and it comes up in some interesting places. Consistently we find that the largest and smallest systems we work with are running Linux. If it’s a large system, like a database or storage network, Linux can provide the stability, robustness, and uptime required to make the project a success. If it’s a very small system, like an embedded display or a custom purpose device, we choose Linux because of its flexibility.

Over the last year or so, we’ve had a need to spin up many more virtual computers than we used to. It’s necessary to create a handful of  computers for each project we work on and the number of projects that we are taking on at a time is growing. Tools like EC2 and VMWare have made it very easy for us to create new virtual computers, but what about managing them? For Linux there are some great solutions. In this blog post I would like to talk about how we’re using some of the tools to deal with the large number of machines we are managing in Amazon EC2 for ourselves and our clients.

Cloud Control

The first part of the system is Amazon’s Elastic Compute Cloud (or EC2 for short). I’ve written about EC2 before, but to be really brief about it: EC2 allows you to create computers on the Internet quickly and lets you pay by the hour for their use. The instances you create are virtual machines that run in a large Xen based infrastructure that Amazon provides. To customize the software and configuration of the instances, Amazon lets you create snapshots of your computers (called Amazon Machine Images or AMIs) that you can launch new instances from.

It’s hard to say that making an AMI is difficult, it’s just a few keystrokes and a coffee break while the image is prepared and stored for your future use. When you begin to manage dozens and dozens of different images this becomes a problem. It became clear to us that we needed to have one AMI to manage, and build every computer from image dynamically. To do this we needed a way to pass specific information to each instance when it started up and then have a tool customize the instance based on the specific information about it. Amazon provides a method to pass specific information to an instance (it’s called user-data) and the good folks at Alestic have made some great AMIs around it with some good documentation.

Enter the Puppet Master

We have been using a tool called Puppet for nearly two years now to manage a handful of computers. We selected that to manage the whole lot of Linux computers that we were dealing with. Puppet allows us to describe how a computer should work in a general way. We can make collections of configurations for standard things so it’s easy to reuse what we create over and over again. Our Puppet configuration is stored in a Git repository so many administrators and developers can collaborate on it and our server configurations are backed up and version controlled automatically.

Bringing it all together

Puppet decides what configuration to use for a computer based on its hostname. We use the EC2 user data to pass a new instance the address of a Puppet server and the hostname that the machine should assume. When it boots up it sets its hostname and checks in to receive the configuration that we’ve stored for it. All changes to the machine happen through Puppet so we don’t have to spend a lot of time SSHing in and customizing the machine. It’s also very easy for us to duplicate a machine for testing or whatever we need.

While we originally did this to make our lives easier as we manage more machines there turned out to be some really cool side effects:

• Excellent Security: We don’t want to store sensitive information in Puppet. This means no passwords or secret stuff. To resolve this we require all developers and administrators to use key-based authentication to get access to the computers via SSH. This is very handy and it eliminates the need to remember passwords or for an administrator to have to reset passwords for users. Someday I would love to take the next step and have all of our users and machines be part of the Monkeysphere.
• Accountability: All of our configuration is tracked in a Git repository so we can see the history about what has changed on certain hosts and who changed it. Change control occurs automatically with Puppet and it’s easy to examine and understand what has changed and why it changed.
• Repeatability: By storing all Linux computer configuration in a single place we can easily repeat what we did for one server on another. Puppet institutionalizes all of our Linux knowledge in one place and saves us time every time we have to create a new machine. If you want to see how someone has done something in the past there’s no more need to dig through emails or documentation, just look at the facts in the Puppet repository and even copy and paste it into your new configuration.
• Portability: We use Puppet to manage much more than just EC2 instances. Physical machines and virtual machines in our ESX installation are managed this way too. It gives us one tool to take care of any Linux machine we deal with. Puppet also supports other operating systems. We’re looking to expand our use of it to Mac OS X machines and maybe even Windows sometime soon.

There are certainly other ways to solve the problems we’re up against, but this is the way we chose. We did some extensive evaluation of Chef (which is a tool like Puppet) and we’ve used Rightscale before. This sort of thing is becoming very important as we manage more and more computers. I expect we’ll see a lot of exciting products, techniques, and services in this space as time goes on.

If you have any questions or comments about our setup, or would like to discuss implementing something like this for your business, leave a comment or get in touch with us. We’d love to help.

Share this:

[youtube=http://www.youtube.com/watch?v=yi50KlsCBio]

First off, Buzz has some serious potential. Google gained an incredible amount of ground on Facebook and Twitter with this launch, and I do sense a shark-jumping moment for Facebook. Once the Google App ecosystem takes off, and social games and e-commerce get integrated, there will be a huge erosion in Facebook market share.  People want one thing, one place to go, and Gmail is already mandatory.

1. Lets start off easy here: Mobile. Google, you own the platform, how hard would it be to launch with an Android app? In the time it took to do the marketing piece on the mobile site, Google could have developed a full-fledged app. Instead I have an “above ground only” slow-loading mobile web page. HTML5 isn’t quite here yet – and Android 1.6 is not supported.
2. Two way integration! Getting Tweets in Buzz is great, but I still have to go out to Twitter or TweetDeck to post. If I had the option to choose which networks my updates appeared on from within Buzz (Facebook, Twitter, LinkedIn, etc), I would never have to leave Gmail… except for…
3. Where is Facebook? They have a strong API and a straightforward authentication service. Connect to Facebook and never make me go there again. Aggregate and publish (see above).
4. Buzz for Biz. I know it’s coming for Google Apps, but get it going already! How about full integration with LinkedIn – a CRM app would be a really interesting mash-up, as well as bringing all my connections into my address book. No more stale email addresses or phone numbers.
5. OK, I saved crazy for last. Google Profile. I am the strongest believer in an open information society – if everyone knows everything about everyone, then we are all equal. I understand that’s a little overboard, but why should I hide my information when it will only make the web more relevant to me, and get us to our ultimate destination quicker.  But with Buzz, your Google Profile went from obscurity to super relevant. So, quick inventory: Google has information about my friends, my browsing/search history (not to mention DNS info), my purchase history, my communications, the news I’m reading, and my documents. Now they want to know my age, sex, where I grew up? Are you crazy? Google is holding ALL the cards now. They keep repeating “don’t be evil” but you know what they say about absolute power… (did I mention my location?)

If history is any indicator of the future, Google will evolve and add features and services. They have done so consistently since inception, and people will choose convenience above all else, including quality and privacy. And I guess I’m one of them. See you on Buzz.

And if you disagree, see you in the comments!

Share this:

We at Control Group have believed for some time that Cloud Computing will change the landscape of how enterprise IT works. With this belief and a history of helping enterprises utilize the cloud, we are proud to announce that we have been selected to become official Amazon Web Services (AWS) partners. Working closely with Amazon Web Services consultants we are able to provide cloud consulting development, integration and migration solutions for a wide range of industries.

In a recent Cloudsourcing event we held at our offices in the Woolworth Building we had the opportunity to meet many people interested in migrating to the cloud. The most common questions that evening seemed to revolve around “is the cloud secure?” and “will this work for my industry?” Being Google Apps partners and now AWS partners we are able to address client concerns with confidence and provide solutions.

Finding cloud-based solutions and implementing those solutions on the enterprise level can be a challenging task as these are still relatively new technologies. Keeping this in mind, part of the process includes stepping down from the cloud and relying on our team of Project Managers, Account Managers and Engineers along with AWS Consultants. With a solid team we are able to guarantee that a client’s needs and expectations are met in a timely manner.

When you think about the potential for cloud computing, you start to realize the role AWS plays in this game. Cloud computing is just now starting to meet the needs of large corporations and the data center of the future may very well be cloud based. We look forward to forging ahead in this area with Amazon Web Services.

Share this:

As summer starts to wind down and the air conditioning is on less and less, your company’s computing equipment is likely resuming its role as the largest consumer of electricity in the office. And while there’s not much that can be done to change that, there are a variety of ways to reduce your energy use without compromising your company’s IT needs.

The Server Room

Annually, the single greatest consumer of electricity in your office is likely the server room. Between the power hungry server hardware, the UPS with its constant AC-to-DC conversion and the air conditioning running 24 hours a day, 7 days a week, 365.25 days a year, all of that electricity use quickly adds up.

Hands-down, the easiest ways to save energy in your server room is to make sure your server room’s HVAC is doing its job as efficiently as possible. That means keeping the door closed at all times (and of course making sure there’s a door in the first place!) (many offices forget this part), keeping the HVAC system well-maintained, and cooling as small of a space as possible.

When buying buying new server hardware, Control Group can help you make the most efficient choices possible… this means anything from buying the right server for the right job (that eight-core print server might be overkill), to outfitting your new server with green-friendly hard drives which consume less power than standard drives. Control Group is an EPEAT Partner Reseller— EPEAT provides a “green” rating system for IT equipment based on a comprehensive criteria list.

The most efficient server room though, is no server room at all! As more and more businesses move aspects of their business to “the cloud,” they require less hardware onsite. Less hardware means lower electricity bills, which means money saved.

The Workstations

First off, if you still have any big bulky CRT monitors in your office, their time has most certainly come. They are big, ugly power hogs, and have no right to exist in 2009. No, not even on the intern’s computer in the janitorial closet. An LCD monitor will pay for itself in power consumption costs alone in under two years. When purchasing an LCD, consider looking into those which have LED-powered backlit displays. Not only do they consume even less power than LCDs using older CCFL technology, but they’re brighter, contain fewer harmful elements in their manufacturing process, and last longer to boot!

Pro-Tip: After you’ve replaced your CRT monitors, what do you do with them? Control Group works with Per Scholas, which is a local organization that recycles and re-uses computer equipment and works with New York City schools to provide students with their own computers as well as training.

Another great example of a small change that can make a big difference would be to make it office policy to power down computers at the end of the day (or at least the end of the week). For even more energy savings, consider flipping the switch on that power strip your computer is plugged into on top of powering down.

And for those employees who say that shutting down hurts their productivity, tell them to use Windows’ “hibernate” feature rather than the “shut down.” That will allow them to pick up right where they left off, with all of their AutoCAD windows and dozens of email drafts right where they left them.

The Office

One way Control Group saves energy around the office is that we have a “no personal printers” policy. We have three printers in the office: a black and white laser printer, and a color laser printer and a large format plotter to help us better support our architecture clients.

Not only does this centralized printing solution help save energy by avoiding having dozens of printers plugged in and on standby 24 hours a day, but it saves money on supplies—no more $18 ink cartridges—and saves us time from having to troubleshoot problematic printers (there’s a reason desktop printers are so cheap… they’re cheaply made and break constantly).

Perhaps most importantly, this printing setup also saves a lot of paper. Duplex printing (two-sided) is enabled on our laser printers and we have a dedicated “scrap paper” tray that we keep loaded with scrap paper, for those times when you need to print something but it doesn’t matter how it looks.

We may not be paperless yet, but we’re well on our way.

Finally, with all of the energy that your office is saving, consider moving over to wind power. Here in New York, ConEd offers a very competitive Wind Power for Business package. It’s 100% renewable energy and it’s easy to switch over.

While wind power may be 10-15% more expensive than coal, depending on how much you’ve managed to cut your overall power consumption, it’s entirely possible your company is still spending less on electricity than before even while using wind power. And on top of that, you can tell your customers you’re ahead of your competitors in reducing your company’s impact on the environment.

Share this:

We have been doing some work with with Amazon’s Elastic Computing Cloud (EC2) which allows us to create virtual machines in the cloud in a few seconds. These are great for hosting websites, and what’s cool about them is that if you get Slashdotted or experience a similar unexpected spike in traffic you can create new hosts immediately. Recently Amazon added a new service called Elastic Load Balancing (ELB) which can distribute load across hosts. We’ve been looking at this for some of our recent development and infrastructure projects.

I just read this description of how ELB works by Shlomo Swidler from his Cloud Developer Tips blog. It’s a great reference.

You pay for ELB by usage just like everything else at AWS. From Amazon: “You are charged at $0.025 per hour for each Elastic Load Balancer, plus $0.008 per GB of data transferred through an Elastic Load Balancer.” For reference, on a deployment project in 2008 our Engineering team used a Cisco load balancer which I imagine cost a few thousand bucks.

Cost isn’t the only advantage. These can be created and destroyed quickly and remotely, allowing us to work more efficiently and spend less time visiting data centers in the middle of nowhere. This leads to improved quality of service for our clients as we can spend more time consulting on future technology growth plans and less time troubleshooting servers in cold, loud data centers.

This blog post brought to you by the iced coffee I am enjoying in the comfort and quiet of my office while deploying virtual machines!

Share this: