And then it all began to change, and for me, it began with Exchange. Exchange 2007 was the last version I deployed, and the last version my own email account was connected to. I’ve been a Google Apps user for two years now, and I’ve also helped transit some of our clients over to Google Apps also.  My personal transition away from a Microsoft-centric working experience started with email, and has continued on through the rest of what I do. I feel like I’m boarding the Google boat and there are just a few bags left on the MS dock.

Google Apps has been offering full-featured email and collaboration services for a while now. An Exchange server can be replaced in a cost-effective fashion, providing shared calendars, spam filtering, message archiving, chat services, access anywhere from a web browser, and almost no mailbox size limits– starting at $50 per mailbox per year. And while you can certainly use Outlook, Apple Mail and iCal to connect to Google, you can use nothing but your web browser if you want and have full functionality. About the only major feature of Exchange not replicated in Google Apps is Public Folders. But then again, Microsoft isn’t providing for that either going forward.

Replacing email servers is one thing, but replacing those desktop applications is another story altogether. Microsoft has held a monopoly on workstation software and business productivity applications for years. The fact is, Windows and Office work well enough for the majority of users out there. They’re easy to use and familiar. A real challenger has to offer a better way of doing something the average user is already doing. And Cloud computing’s Software As A Service  is finally maturing to that point.

Speaking as a heavy Microsoft user, I’ve personally been anchored to Windows and Office, primarily by Visio and Project.  Microsoft Word is still critical for final document drafts and printing, since the offerings on Google Apps just aren’t there yet in terms of refinement and features. But it’s just a matter of time. The recent addition of Smartsheet to our toolkit has now removed our reliance on Microsoft Project except for our very largest initiatives. And we are eagerly awaiting the evolution of Google Drawings to allow us to build the type of schematics we’re creating with Visio.

Microsoft of course has its own Cloud offering, Office 365, and naturally it’s tied into licensing of their existing products. They have a tiered licensing model that’s much more complex than Google’s. But it’s also a much deeper system than Google Apps.  It goes without saying that Google has no legacy revenue streams to drag into the 21st century. Even so, any company can make a migration decision based on what functionality works best for them, but it’s nice to be able to start fresh at a very aggressive price-point.

For further comparison, since Google Apps is platform agnostic, Mac users don’t draw the short straw yet again when it comes to software and collaboration with their Windows brethren. For remote users in a company that’s migrated to Google Apps, all they need is an Internet connection and a web browser, and they will have exactly the same experience as they have sitting in their office. While the individual feature-set of Google Apps isn’t as elegant or robust as Microsoft Office or Office 365, Google Apps is radically simpler in that you get full access to it’s features with just a web browser. This makes it a serious competitor for today’s geographically diverse, small to mid size business.

Regarding remote workers, smartphones deserve a mention. I used a Blackberry for years, but opted for a Droid recently. You probably knew this was coming, but Google Apps on the Droid takes only minutes to configure, and works smoothly. Google Apps functionality includes: Gmail, Calendar and Contact sync, Push support, Google Docs,  Enterprise Admin controls, and 2-way verification for extra security. This feature set is available for almost all platforms, including Windows:  http://www.google.com/apps/intl/en/business/mobile.html

By contrast, Office 365 applications for iPhone and Android are not coming anytime soon. Mobile access is limited unless you’re using Windows Phone, which let’s be honest, almost nobody is.

Handheld prevalence as per end-of-year 2010:

Android #1 – 33.3 milllion

Symbian #2 – 31 million

Apple #3 – 16.2 million

RIM #4 – 14.6 – million

MS #5 – 3.1 million

Source: http://on.mash.to/rG6bfd

As I write this, I have my email, shared documents, and a Project Plan open on my Windows workstation and on my Linux laptop. I use these applications every single day. My documents appear and function identically across these two computers. My email appears and functions identically. The Gannt chart appears and functions identically. On my Droid phone I have read and write access to my calendar, documents, and email. With the exception of Visio and Word, I can be anywhere with an Internet connection, on a borrowed computer running OSX, Linux or Windows and be fully functional. The last of my bags on the dock contain Word and Visio. I can’t leave them behind just yet, but I’m waiting. I want to get going.

Share this:

NEW YORK–(BUSINESS WIRE)–Control Group, a technology consulting firm, today announced it will be offering collaboration services for healthcare companies looking for the scalability and cost advantages of cloud migration under the strict HIPAA requirements of a Business Associate Agreement. New York City based Control Group offers clients a bundle of services that include a comprehensive HIPAA-focused technology audit, cloud design and architecture implementation that meets the complex HIPAA requirements.

“Many healthcare companies are great candidates for cloud migration. They want to explore the platform for its scalability and cost advantages,” said Sholom Ellenberg, Control Group’s executive vice president of Infrastructure Services. “With this new service we offer collaboration with healthcare companies assuring that the solution will be compliant.”

Control Group’s innovative cloud architecture methodology leverages sophisticated automation, allowing healthcare companies to manage infrastructure on-demand in tandem with their applications. This integrated approach maximizes the advantages of AWS and ensures compliance with HIPAA security and privacy requirements.

Control Group client Pronia Medical Systems, creator of an innovative insulin dosing system used by hospitals to manage glucose delivery, has already experienced benefits from the new offering. “Control Group’s solution shortened our customer onboarding effort significantly. A new application instance in AWS can be deployed quickly, and the technology audit, design and implementation phases of the project kept our HIPAA compliance front and center,” said Brian Besterman, M.D., CIO and co-founder of Pronia.

“We are excited to work with Control Group to enable healthcare companies to migrate their core applications to AWS. Control Group’s innovative consulting, migration, and implementation services make it easier for customers to take advantage of AWS’ secure, on-demand, pay as you go cloud services,” said Terry Wise, Director, Business Development, AWS.

For more information about Control Group’s HIPAA Compliant Cloud Migration Services please contact Sholom Ellenberg at 212.343.2525 x681 or visit controlgroup.com.

Share this:

Last night I was joined by our VP of Dev Ops, David Rocamora, and one of our senior application developers, Evan Frohlich, for a visit to CUNY’s Entrepreneurial Journalism program. We were invited to be guest presenters in the program’s Technology Immersion class. This course presents students with the media, business, and journalistic opportunities presented by new and emerging technologies and trends. It was a great, smart group of students from a wide range of backgrounds, each with a unique idea for a business or product at the crossroads of technology and journalism. Lots of neat innovation happening there.

I spoke a bit out CG, our history, capabilities, and process, while Dave gave an excellent talk on the Cloud, and how software and infrastructure are intimately connected. (Remember what happened last week?) Evan spoke about behavior driven development, and offered some guidance on avoiding feature creep, and releasing early and often.

It was great to be back in a classroom with these guys. We can’t wait to see how the student’s business and product ideas gel, and look forward to using some of their products in the future!

Share this:

We maintain two base Linux images as part of our cloud toolkit. The only difference between the two images is their architecture. One is 64bit and the other is 32bit. The images are minimal– they have just enough software and configuration to get them off the ground and configured. We have copies of the images in each region in Amazon, but when it comes to maintenance and upgrades we really only deal with the two master images. All of the computers that we deploy in EC2 come from these two images.

The base image by itself is not very useful. When a computer is instantiated from one of the images, our toolkit combines it with our Puppet repository and some instance specific configuration. The Puppet repository contains the Puppet manifests for how we deploy software. The repository is where we store our collective knowledge around deploying successful software. The instance specific configuration is crafted by the developers and operations teams to pick and choose the appropriate things from the Puppet repository provide the very specific configuration about how to deploy the server and the application that will run on it. As the instance boots, it configures itself, installing the software and making the changes required to bring it into service.

This is all pretty low level, but it provides some capabilities that makes our solution very flexible:

• With only two images to maintain, keeping software up to date is simple. We anticipate that we will be releasing new images about once a quarter to capture any updates to the packages in the base system.
• Everything is version controlled. It is easy for us to see what a machine looked like on a specific date or understand the changes that have been made to how the software is configured on an instance.
• The instances are very self sufficient. There is no single point of failure that would prevent instances from starting correctly.
• This is all very portable. With just a little bit of work we can deploy things in a different region of Amazon. Also, our Puppet code and instance specific configurations can work in more places than just Amazon. With a little bit of work to recreate the base images in another platform we can consistently and predictable recreate infrastructure anywhere, giving our clients the ability to choose the right solution for them.

This last item is something that should be on everyone’s mind (especially considering the outage at Amazon last week). As Steve said last week, everything fails and you need to design your infrastructure and applications around that. A process for redeploying your infrastructure in another AWS region or a different cloud is an important part of building a very reliable service in the cloud. It is hard to say what the next kind of failure in the cloud will be like, but with a process like ours we can be ready to deal with outage when it happens.

Share this:

You can design around regional performance degradation though. Years ago, having global traffic management in place was an expensive pipe dream. Today you can easily turn up another EC2 region and use a service like Dynect or Akamai GTM to provide failover and/or load balancing. Even better, consider making your systems portable so you can have multiple cloud providers and maintain your machines and applications with Puppet.

3-5 years ago this would have taken a year of planning, purchasing and hands-on labor to implement two data centers. Earlier this year we were able to create two data centers with complex infrastructure on EC2 and active/active load balancing in under two months and for a fraction of the cost.

Share this:

I thought Phil Wainwright’s most recent article on private clouds (as well as the first in the series) was an interesting perspective. I share many of the sentiments, but can’t go quite so far as to say the idea of the private cloud is discredited. In the end it depends on the business and it depends on the applications you are hosting. Even Adrian Cockcroft, the writer of the blog that Phil cites as final proof, has updated his post to say:

“…to clarify, that doesn’t mean that I’m against private clouds or don’t think they exist, because $, FUD and internal politics are a fact of life that constrain what can be done.”

Private clouds, whether hosted or self-hosted, can be useful as stepping stones for organizations that have existing applications that may not fit into the public cloud architecture. Some of these applications may require better performance SLA’s. Also, private cloud providers are more amenable to custom arrangements. Try hosting a specialized device like an IPS or IDS in a public cloud where all traffic is guaranteed to only be delivered to the target device.  Many enterprise organizations have decades of IT security policies that won’t and perhaps shouldn’t simply go away in favor of adopting a public cloud.  Public clouds are secure solutions, but some organizations will have additional requirements, like the ability to discern rogue traffic patterns from typical spikes in demand. You can build this into the individual instances and applications, but that isn’t what most companies have done.

We tend to work with a client to find out what their requirements are and stay away from radical statements. The cloud, public or private, is just one more tool and can’t be seen as a solution in and of itself.

Share this:

In a recent engagement Control Group needed to work with a client to have application instances on EC2 communicate with a secured web service in a traditional data center. Typically we would work with a client to move this service to EC2. In this case, because the service is considered to be shared infrastructure that is used and funded by existing applications we needed to design the infrastructure and application to make a call back to a traditional data center.

On a side note, mixed infrastructure approaches are not ideal, but common when migrating complex organizations to IAAS solutions. Most mature IT organizations will shy away from forklifting a company’s technology platforms wholesale into the cloud. The larger the migration, the bigger the bang when something is overlooked. Change too much in an environment and you won’t know where the problems are coming from, so a major part of moving an enterprise customer to the cloud is planning the roadmap of the migration carefully and not being greedy.

One of the technical challenges in this particular project was that the service that we were integrating with requires that traffic originate from a known and registered IP address. Although EC2 will provide an instance with a public IP address, there is no way to know what that address will be ahead of time. We decided to use Elastic IP (EIP) addresses to solve this problem. An EIP functions like a NAT on a traditional firewall. You can allocate the EIP and then associate it with an instance as needed.

EIP’s worked well until we implemented auto-scaling. Auto-scaling groups have no support for associating a pre-allocated EIP to an instance. To implement this we created some scripts that would make the API calls to determine a free EIP and associate it to the instance. (This means that the instance will have temporary access to execute API commands. We’ve designed a fairly secure take on temporarily providing AWS API tools to an instance, but that is a different blog post. Coming soon.)

Here is the real problem with the approach. The script to associate the EIP worked perfectly, so long as multiple machines weren’t executing it at once. The problem is that the Elastic IP API commands do not support a transactional assignment. Worse yet, at least in our use case, it is the last instance requesting the EIP and not the first that gets associated to the IP. This is a major problem if you want to associate EIP’s with members of an auto-scaling group that need to scale up by more than one instance at a time. It will leave you with members of the group that could possibly not have an Elastic IP.

There are a myriad of ways to tackle this issue. We considered options for programatically brokering the IP’s by building an application that would manage the EIP resources. The application would provide an IP on request and then return IP’s that were no longer in use back into the system through a background recovery process. Such a service is pretty easy to write, but it wasn’t in scope for the current project. Also, there are longer-term solutions that we can consider with the launch of the new and improved VPC with NATing.

Interim Solutions

The current favored approach is to use a proxy server like Squid to limit the number of servers that require IP addresses. Two or more instances with Squid configured as a forward proxy distributed across multiple availability zones and traffic managed by an Elastic Load Balancer to provide HA would provide a redundant and fairly high performance solution. For now, as a work around we have implemented some staggering of the auto-scaling policies as a way to mitigate against multiple instances spinning up at the same time. Staggering is a serviceable solution for testing, but not for production where auto scaling multiple farms of servers that will need access to the client’s data center tier is a requirement. Eventually, we will move forward with the proxy or VPC solution.

In summary, enterprises with complex interdependent applications can lead to interesting challenges when migrating to the cloud. Resources, as simple as IP addresses, can function in a fundamentally different way than a typical IT organization is used to. Oftentimes this can lead to fear, uncertainty, and doubt, but the benefits of Infrastructure as a Service are clear: Ease of provisioning, demand-based resource allocation rather than over provisioning, etc.  As long as proper planning, system architecture, implementation, and testing are performed, a complex enterprise can begin making its way to the cloud and begin to eliminate the FUD on the ground.

Share this:

Large enterprises looking at cloud infrastructure bring their own requirements and challenges. We plan to write a series of blog posts about Control Group’s experiences with these types of clients and what we learned. Some of the posts will be about the projects and their politics, and some will be about technology approach. There are some interesting technology and organizational challenges that we will discuss, so stay tuned.

Share this:

In the midst of all of this, a few of us at Control Group have been developing Tagatag: an Android and iPhone application for commenting on barcodes that uses web services running on Google App Engine.

Scan this QR code with Tagatag to join the conversation!

Barcodes are everywhere around us. You can find them on advertising, products, places and even people. Tagatag provides you with a virtual paint marker to let you make your mark on all of these codes anonymously. Download the Tagatag app and give it a try. Scan a barcode to see comments people have left for you and then leave some for them.

We chose Google App Engine for the back end of Tagatag for a few reasons:

• It’s quick – You sign up for an account, download the SDK and you’re developing. The development server in the SDK lets me run the application on my desktop and interact with the code as I’m writing it.  Uploading new versions, rolling back old ones, or performing maintenance is a snap with the GAE dashboard.
• It’s simple – There’s not much to the web service. It’s small and simple. We used the webapp framework because we didn’t feel we needed anything else. It makes for a very concise application. Believe it or not, there are about 300 lines of code for the GAE part of Tagatag.
• It’s scalable – We don’t have to worry about what we do when Tagatag becomes popular. We’ll just raise our billing quotas in GAE and let them handle spinning up new instances or expanding the datastore. Knowing that you don’t have to be concerned about scaling makes things a lot more fun.

I’m happy that GAE let us bring Tagatag to you so quickly. So, when it’s available at the end of the week, be sure to download the app, tag a tag and make your mark!

Share this:

MapReduce frameworks allow developers without much knowledge on distributed computing to write applications that take advantage of distributed resources. Hadoop MapReduce is an implementation of such a model.

Background:

Recently, we developed a web asset delivery service for one of our clients that would allow businesses to display high quality assets from a CDN in their websites for a monthly fee. Users’ accounts would be associated with bandwidth limits based on different account levels associated with a pricing model. This meant that we needed a way to provide users with information on monthly bandwidth utilization across their websites in order to defend the pricing model. The solution to this was to implement web log parsing using Hadoop’s MapReduce framework in conjunction with Amazon’s cloud-based Elastic MapReduce service.

Here’s how Hadoop MapReduce works:

Hadoop MapReduce is a Java-based framework that allows you to write applications that process high volumes of data in parallel clusters. Hadoop uses a distributed file storage system called Hadoop Distributed File System (HDFS) to store large amount of data across multiple nodes. It supports most major platforms and MapReduce programs can be written in Python, Ruby, php, Pig, etc., in addition to Java.  Using Hadoop, we were able to write a simple Java program that could easily parse through raw data in log files collected by the CDN and filter relevant bandwidth utilization information.

The basic idea around a map-reduce model is that you write two functions — map() and reduce() — to divide up your programming tasks and let the framework manage most of the crunching. Map and reduce functions take in key-value pairs (using data types that implement Hadoop’s Writable interface) as the input and output. When you start a map reduce process, you pass in a data file in HDFS as the input. Hadoop divides up the inputs into smaller pieces that the map function can consume. Likewise, the outputs of the map function are grouped together in logical chunks by Hadoop and sent to the reduce function for processing.  Both map and reduce functions can run in parallel — Hadoop can distribute the tasks across various clusters of nodes.

In our case, we simply pass the log file(s) (copied to HDFS) as input to the map reduce program.  Hadoop merges all log files specified and serializes each log entry to the datatype expected (Text) before passing them as inputs to the map reduce tasks. Our map function then parses each long entry individually and stores the relevant data (bandwidth info) in a HashMap type object (MapWritable) which is then sent as another key-value pair (<asset path – MapWritable object>) for the reduce function to work with. The reduce function then aggregates the data based on user accounts, date, user agent, etc. and saves it to a database (Amazon RDS Database). We can then query the database to pull all types of information around utilization and send out notifications to users, for example, if their account is over the monthly cap, etc.

Below is the structure of a sample map reduce program written in Java:

public class LogProcessor {

  public static class LogMap
            extends Mapper<LongWritable, Text, Text, MapWritable> {
    public void map( LongWritable key, Text value, Context context ) {
      MapWritable logEntry = new MapWritable();
      //parse log file
      ...
      Text key = new Text();
      //key = resource-path;

      context.write( key, logEntry);
    }
  }

  public static class LogReduce
            extends Reducer<Text, MapWritable, DBWritable, NullWritable> {
    public void reduce( Text key, Iterable<MapWritable> values, Context context ) {
      while(values.iterator().hasNext()) {
        MapWritable entry = values.iterator().next();
        //process entry and write to db
        ...
      }
    }
  }

  public static void main(String[] args) {
    // Set up a new mapreduce job
    Job job = new Job();
    job.setJarByClass(LogProcessor.class);    //register the main class

    FileInputFormat.addInputPath( job, new Path(<Input file path>) );
    FileOutputFormat.setInputPath( job, new Path(<output file path> ) );

    job.setMapperClass( LogMap.class );
    job.setReducerClass( LogReduce.class );

    job.setOutputKeyClass( Text.class );
    job.setOutputValueClass( MapWritable.class );

    job.waitForCompletion(true) ? System.exit(0) : System.exit(1);
  }
}

The program is packaged in a jar file (with dependencies) that Hadoop can run.

And here’s how to utilize Amazon’s Elastic MapReduce service to run the program:

At Control Group, we leverage Amazon’s cloud based infrastructure heavily in lots of projects. It basically allows us to cost effectively (pay by usage) deploy applications that need to scale up very easily. Amazon’s Elastic MapReduce service is the perfect fit for running our MapReduce application described above. It’s easy to set up, and it also shields off some of the infrastructure/maintenance issues around running Hadoop.

In a nutshell, the Elastic MapReduce service runs a hosted Hadoop instance on an EC2 instance (master), and it’s able to instantly provision other pre-configured EC2 instances (slave nodes) to distribute the MapReduce process, which are all terminated once the MapReduce tasks complete running. Amazon allows us to specify up to 20 EC2 instances for data intensive processing. It also provides the option to upgrade your Elastic MapReduce to increase EC2 instance count.

So to run the map reduce service, we create a new “Job Flow” via the AWS console, the command line utility (ruby based) or an API provided by Amazon. A job flow is a set of steps that Elastic MapReduce runs. You basically provide some configuration information (number of EC2 instances to use and bootstrap actions) and the location of your map reduce program ( usually an Amazon S3 bucket path). Job flow records/logs can be viewed at the AWS console. You can also explicitly instruct Elastic MapReduce to keep the master EC2 instance alive for debugging purposes – you can then ssh into the instance to check the log files created by Hadoop, etc.

In summary, Hadoop’s MapReduce framework allows us to write simple applications that process high volumes of data in a distributed computing environment while Amazon’s MapReduce service provides a cost-effective means of implementing such a solution.

Share this: