Bought something at the Apple Store on West 14th Street yesterday and tried the new Apple Store app for self checkout. Launch the app and it recognizes you’re in a store (GPS? SSID? Geo-fencing?). A special interface appears within the app, you click the EasyPay button and take a pic of the barcode on whatever you’re purchasing. Pay with your Apple ID (same as iTunes account) by typing your password. Your receipt appears on screen so a sales rep can give you a “paid” sticker.

Pretty slick. Apple’s got something incredibly powerful with their Apple ID system tied to customer credit card info, and they’re one of the only big players in the space that has both physical and online stores. Amazon’s got this down (duh) but not so much for offline purchasing. Facebook and Google are trying to figure out how to monetize offline purchases too, but seems like they’re playing catch up here.

Share this:

NEW YORK–(BUSINESS WIRE)–Control Group, a technology consulting firm, today announced it will be offering collaboration services for healthcare companies looking for the scalability and cost advantages of cloud migration under the strict HIPAA requirements of a Business Associate Agreement. New York City based Control Group offers clients a bundle of services that include a comprehensive HIPAA-focused technology audit, cloud design and architecture implementation that meets the complex HIPAA requirements.

“Many healthcare companies are great candidates for cloud migration. They want to explore the platform for its scalability and cost advantages,” said Sholom Ellenberg, Control Group’s executive vice president of Infrastructure Services. “With this new service we offer collaboration with healthcare companies assuring that the solution will be compliant.”

Control Group’s innovative cloud architecture methodology leverages sophisticated automation, allowing healthcare companies to manage infrastructure on-demand in tandem with their applications. This integrated approach maximizes the advantages of AWS and ensures compliance with HIPAA security and privacy requirements.

Control Group client Pronia Medical Systems, creator of an innovative insulin dosing system used by hospitals to manage glucose delivery, has already experienced benefits from the new offering. “Control Group’s solution shortened our customer onboarding effort significantly. A new application instance in AWS can be deployed quickly, and the technology audit, design and implementation phases of the project kept our HIPAA compliance front and center,” said Brian Besterman, M.D., CIO and co-founder of Pronia.

“We are excited to work with Control Group to enable healthcare companies to migrate their core applications to AWS. Control Group’s innovative consulting, migration, and implementation services make it easier for customers to take advantage of AWS’ secure, on-demand, pay as you go cloud services,” said Terry Wise, Director, Business Development, AWS.

For more information about Control Group’s HIPAA Compliant Cloud Migration Services please contact Sholom Ellenberg at 212.343.2525 x681 or visit controlgroup.com.

Share this:

We maintain two base Linux images as part of our cloud toolkit. The only difference between the two images is their architecture. One is 64bit and the other is 32bit. The images are minimal– they have just enough software and configuration to get them off the ground and configured. We have copies of the images in each region in Amazon, but when it comes to maintenance and upgrades we really only deal with the two master images. All of the computers that we deploy in EC2 come from these two images.

The base image by itself is not very useful. When a computer is instantiated from one of the images, our toolkit combines it with our Puppet repository and some instance specific configuration. The Puppet repository contains the Puppet manifests for how we deploy software. The repository is where we store our collective knowledge around deploying successful software. The instance specific configuration is crafted by the developers and operations teams to pick and choose the appropriate things from the Puppet repository provide the very specific configuration about how to deploy the server and the application that will run on it. As the instance boots, it configures itself, installing the software and making the changes required to bring it into service.

This is all pretty low level, but it provides some capabilities that makes our solution very flexible:

• With only two images to maintain, keeping software up to date is simple. We anticipate that we will be releasing new images about once a quarter to capture any updates to the packages in the base system.
• Everything is version controlled. It is easy for us to see what a machine looked like on a specific date or understand the changes that have been made to how the software is configured on an instance.
• The instances are very self sufficient. There is no single point of failure that would prevent instances from starting correctly.
• This is all very portable. With just a little bit of work we can deploy things in a different region of Amazon. Also, our Puppet code and instance specific configurations can work in more places than just Amazon. With a little bit of work to recreate the base images in another platform we can consistently and predictable recreate infrastructure anywhere, giving our clients the ability to choose the right solution for them.

This last item is something that should be on everyone’s mind (especially considering the outage at Amazon last week). As Steve said last week, everything fails and you need to design your infrastructure and applications around that. A process for redeploying your infrastructure in another AWS region or a different cloud is an important part of building a very reliable service in the cloud. It is hard to say what the next kind of failure in the cloud will be like, but with a process like ours we can be ready to deal with outage when it happens.

Share this:

You can design around regional performance degradation though. Years ago, having global traffic management in place was an expensive pipe dream. Today you can easily turn up another EC2 region and use a service like Dynect or Akamai GTM to provide failover and/or load balancing. Even better, consider making your systems portable so you can have multiple cloud providers and maintain your machines and applications with Puppet.

3-5 years ago this would have taken a year of planning, purchasing and hands-on labor to implement two data centers. Earlier this year we were able to create two data centers with complex infrastructure on EC2 and active/active load balancing in under two months and for a fraction of the cost.

Share this:

I thought Phil Wainwright’s most recent article on private clouds (as well as the first in the series) was an interesting perspective. I share many of the sentiments, but can’t go quite so far as to say the idea of the private cloud is discredited. In the end it depends on the business and it depends on the applications you are hosting. Even Adrian Cockcroft, the writer of the blog that Phil cites as final proof, has updated his post to say:

“…to clarify, that doesn’t mean that I’m against private clouds or don’t think they exist, because $, FUD and internal politics are a fact of life that constrain what can be done.”

Private clouds, whether hosted or self-hosted, can be useful as stepping stones for organizations that have existing applications that may not fit into the public cloud architecture. Some of these applications may require better performance SLA’s. Also, private cloud providers are more amenable to custom arrangements. Try hosting a specialized device like an IPS or IDS in a public cloud where all traffic is guaranteed to only be delivered to the target device.  Many enterprise organizations have decades of IT security policies that won’t and perhaps shouldn’t simply go away in favor of adopting a public cloud.  Public clouds are secure solutions, but some organizations will have additional requirements, like the ability to discern rogue traffic patterns from typical spikes in demand. You can build this into the individual instances and applications, but that isn’t what most companies have done.

We tend to work with a client to find out what their requirements are and stay away from radical statements. The cloud, public or private, is just one more tool and can’t be seen as a solution in and of itself.

Share this:

In a recent engagement Control Group needed to work with a client to have application instances on EC2 communicate with a secured web service in a traditional data center. Typically we would work with a client to move this service to EC2. In this case, because the service is considered to be shared infrastructure that is used and funded by existing applications we needed to design the infrastructure and application to make a call back to a traditional data center.

On a side note, mixed infrastructure approaches are not ideal, but common when migrating complex organizations to IAAS solutions. Most mature IT organizations will shy away from forklifting a company’s technology platforms wholesale into the cloud. The larger the migration, the bigger the bang when something is overlooked. Change too much in an environment and you won’t know where the problems are coming from, so a major part of moving an enterprise customer to the cloud is planning the roadmap of the migration carefully and not being greedy.

One of the technical challenges in this particular project was that the service that we were integrating with requires that traffic originate from a known and registered IP address. Although EC2 will provide an instance with a public IP address, there is no way to know what that address will be ahead of time. We decided to use Elastic IP (EIP) addresses to solve this problem. An EIP functions like a NAT on a traditional firewall. You can allocate the EIP and then associate it with an instance as needed.

EIP’s worked well until we implemented auto-scaling. Auto-scaling groups have no support for associating a pre-allocated EIP to an instance. To implement this we created some scripts that would make the API calls to determine a free EIP and associate it to the instance. (This means that the instance will have temporary access to execute API commands. We’ve designed a fairly secure take on temporarily providing AWS API tools to an instance, but that is a different blog post. Coming soon.)

Here is the real problem with the approach. The script to associate the EIP worked perfectly, so long as multiple machines weren’t executing it at once. The problem is that the Elastic IP API commands do not support a transactional assignment. Worse yet, at least in our use case, it is the last instance requesting the EIP and not the first that gets associated to the IP. This is a major problem if you want to associate EIP’s with members of an auto-scaling group that need to scale up by more than one instance at a time. It will leave you with members of the group that could possibly not have an Elastic IP.

There are a myriad of ways to tackle this issue. We considered options for programatically brokering the IP’s by building an application that would manage the EIP resources. The application would provide an IP on request and then return IP’s that were no longer in use back into the system through a background recovery process. Such a service is pretty easy to write, but it wasn’t in scope for the current project. Also, there are longer-term solutions that we can consider with the launch of the new and improved VPC with NATing.

Interim Solutions

The current favored approach is to use a proxy server like Squid to limit the number of servers that require IP addresses. Two or more instances with Squid configured as a forward proxy distributed across multiple availability zones and traffic managed by an Elastic Load Balancer to provide HA would provide a redundant and fairly high performance solution. For now, as a work around we have implemented some staggering of the auto-scaling policies as a way to mitigate against multiple instances spinning up at the same time. Staggering is a serviceable solution for testing, but not for production where auto scaling multiple farms of servers that will need access to the client’s data center tier is a requirement. Eventually, we will move forward with the proxy or VPC solution.

In summary, enterprises with complex interdependent applications can lead to interesting challenges when migrating to the cloud. Resources, as simple as IP addresses, can function in a fundamentally different way than a typical IT organization is used to. Oftentimes this can lead to fear, uncertainty, and doubt, but the benefits of Infrastructure as a Service are clear: Ease of provisioning, demand-based resource allocation rather than over provisioning, etc.  As long as proper planning, system architecture, implementation, and testing are performed, a complex enterprise can begin making its way to the cloud and begin to eliminate the FUD on the ground.

Share this:

Large enterprises looking at cloud infrastructure bring their own requirements and challenges. We plan to write a series of blog posts about Control Group’s experiences with these types of clients and what we learned. Some of the posts will be about the projects and their politics, and some will be about technology approach. There are some interesting technology and organizational challenges that we will discuss, so stay tuned.

Share this:

MapReduce frameworks allow developers without much knowledge on distributed computing to write applications that take advantage of distributed resources. Hadoop MapReduce is an implementation of such a model.

Background:

Recently, we developed a web asset delivery service for one of our clients that would allow businesses to display high quality assets from a CDN in their websites for a monthly fee. Users’ accounts would be associated with bandwidth limits based on different account levels associated with a pricing model. This meant that we needed a way to provide users with information on monthly bandwidth utilization across their websites in order to defend the pricing model. The solution to this was to implement web log parsing using Hadoop’s MapReduce framework in conjunction with Amazon’s cloud-based Elastic MapReduce service.

Here’s how Hadoop MapReduce works:

Hadoop MapReduce is a Java-based framework that allows you to write applications that process high volumes of data in parallel clusters. Hadoop uses a distributed file storage system called Hadoop Distributed File System (HDFS) to store large amount of data across multiple nodes. It supports most major platforms and MapReduce programs can be written in Python, Ruby, php, Pig, etc., in addition to Java.  Using Hadoop, we were able to write a simple Java program that could easily parse through raw data in log files collected by the CDN and filter relevant bandwidth utilization information.

The basic idea around a map-reduce model is that you write two functions — map() and reduce() — to divide up your programming tasks and let the framework manage most of the crunching. Map and reduce functions take in key-value pairs (using data types that implement Hadoop’s Writable interface) as the input and output. When you start a map reduce process, you pass in a data file in HDFS as the input. Hadoop divides up the inputs into smaller pieces that the map function can consume. Likewise, the outputs of the map function are grouped together in logical chunks by Hadoop and sent to the reduce function for processing.  Both map and reduce functions can run in parallel — Hadoop can distribute the tasks across various clusters of nodes.

In our case, we simply pass the log file(s) (copied to HDFS) as input to the map reduce program.  Hadoop merges all log files specified and serializes each log entry to the datatype expected (Text) before passing them as inputs to the map reduce tasks. Our map function then parses each long entry individually and stores the relevant data (bandwidth info) in a HashMap type object (MapWritable) which is then sent as another key-value pair (<asset path – MapWritable object>) for the reduce function to work with. The reduce function then aggregates the data based on user accounts, date, user agent, etc. and saves it to a database (Amazon RDS Database). We can then query the database to pull all types of information around utilization and send out notifications to users, for example, if their account is over the monthly cap, etc.

Below is the structure of a sample map reduce program written in Java:

public class LogProcessor {

  public static class LogMap
            extends Mapper<LongWritable, Text, Text, MapWritable> {
    public void map( LongWritable key, Text value, Context context ) {
      MapWritable logEntry = new MapWritable();
      //parse log file
      ...
      Text key = new Text();
      //key = resource-path;

      context.write( key, logEntry);
    }
  }

  public static class LogReduce
            extends Reducer<Text, MapWritable, DBWritable, NullWritable> {
    public void reduce( Text key, Iterable<MapWritable> values, Context context ) {
      while(values.iterator().hasNext()) {
        MapWritable entry = values.iterator().next();
        //process entry and write to db
        ...
      }
    }
  }

  public static void main(String[] args) {
    // Set up a new mapreduce job
    Job job = new Job();
    job.setJarByClass(LogProcessor.class);    //register the main class

    FileInputFormat.addInputPath( job, new Path(<Input file path>) );
    FileOutputFormat.setInputPath( job, new Path(<output file path> ) );

    job.setMapperClass( LogMap.class );
    job.setReducerClass( LogReduce.class );

    job.setOutputKeyClass( Text.class );
    job.setOutputValueClass( MapWritable.class );

    job.waitForCompletion(true) ? System.exit(0) : System.exit(1);
  }
}

The program is packaged in a jar file (with dependencies) that Hadoop can run.

And here’s how to utilize Amazon’s Elastic MapReduce service to run the program:

At Control Group, we leverage Amazon’s cloud based infrastructure heavily in lots of projects. It basically allows us to cost effectively (pay by usage) deploy applications that need to scale up very easily. Amazon’s Elastic MapReduce service is the perfect fit for running our MapReduce application described above. It’s easy to set up, and it also shields off some of the infrastructure/maintenance issues around running Hadoop.

In a nutshell, the Elastic MapReduce service runs a hosted Hadoop instance on an EC2 instance (master), and it’s able to instantly provision other pre-configured EC2 instances (slave nodes) to distribute the MapReduce process, which are all terminated once the MapReduce tasks complete running. Amazon allows us to specify up to 20 EC2 instances for data intensive processing. It also provides the option to upgrade your Elastic MapReduce to increase EC2 instance count.

So to run the map reduce service, we create a new “Job Flow” via the AWS console, the command line utility (ruby based) or an API provided by Amazon. A job flow is a set of steps that Elastic MapReduce runs. You basically provide some configuration information (number of EC2 instances to use and bootstrap actions) and the location of your map reduce program ( usually an Amazon S3 bucket path). Job flow records/logs can be viewed at the AWS console. You can also explicitly instruct Elastic MapReduce to keep the master EC2 instance alive for debugging purposes – you can then ssh into the instance to check the log files created by Hadoop, etc.

In summary, Hadoop’s MapReduce framework allows us to write simple applications that process high volumes of data in a distributed computing environment while Amazon’s MapReduce service provides a cost-effective means of implementing such a solution.

Share this:

We have been doing some work with with Amazon’s Elastic Computing Cloud (EC2) which allows us to create virtual machines in the cloud in a few seconds. These are great for hosting websites, and what’s cool about them is that if you get Slashdotted or experience a similar unexpected spike in traffic you can create new hosts immediately. Recently Amazon added a new service called Elastic Load Balancing (ELB) which can distribute load across hosts. We’ve been looking at this for some of our recent development and infrastructure projects.

I just read this description of how ELB works by Shlomo Swidler from his Cloud Developer Tips blog. It’s a great reference.

You pay for ELB by usage just like everything else at AWS. From Amazon: “You are charged at $0.025 per hour for each Elastic Load Balancer, plus $0.008 per GB of data transferred through an Elastic Load Balancer.” For reference, on a deployment project in 2008 our Engineering team used a Cisco load balancer which I imagine cost a few thousand bucks.

Cost isn’t the only advantage. These can be created and destroyed quickly and remotely, allowing us to work more efficiently and spend less time visiting data centers in the middle of nowhere. This leads to improved quality of service for our clients as we can spend more time consulting on future technology growth plans and less time troubleshooting servers in cold, loud data centers.

This blog post brought to you by the iced coffee I am enjoying in the comfort and quiet of my office while deploying virtual machines!

Share this:

Your Data Here

I was having a conversation recently with one of our consultants, David Rocamora, as our team broke down the contents of a start-up’s data center, when I came to the realization that we may have built our last data center.

Now we don’t really build data centers, but we have racked a lot of servers, storage and network gear around the world in tier 1 data centers for our clients. With the change in the economy and the maturity of several cloud services, the data center that we know and love looks like it’s going the way of the wood pulp newspaper.

Certainly more data centers than ever are being built: Google, Microsoft, Apple, and Amazon are soaking up gigawatts (petawatts?) of power like never before. But the days of dressing down in jeans and a sweatshirt and going out to New Jersey or Colorado to rack servers in an earsplitting, freezing cold warehouse of caged servers and blinking lights seems to be drawing to a close.

Some numbers to consider:

This ‘scaled down’ dot com we were consolidating, had spent about $500,000 on a few racks of some amazing equipment (Sun, Check Point, etc) only to find out 4 months later they didn’t need it. Pennies on the dollar. The contract for the floor space, power, and bandwidth goes on for another 8 months and I bet you could buy a modest BMW for what it’s costing them.

Now a similar sized start-up we just started working with on a really innovative interactive image platform, is using the Amazon cloud and RightScale and is spending about $50,000 a year on cloud services. No capital outlay.

Flexibility?

The cloud is infinitely more flexible, we can put servers in Europe in a matter of minutes, set up high availability zones in different regions around the country, and if they start to get swamped with business like we think they will, we’ll be able to turn up as many servers as they need in a few minutes time.

What if the dot com of 4 months ago took off? Order servers, spend capital. Put in a request for more bandwidth, more cage space. Days, maybe weeks go by. Then get out the jeans and sweatshirt and head over to the data center. Earplugs. Man, those servers are loud.

You get the picture. But this is happening so fast it’s amazing. Six months ago when the dot com was building its data center, the Amazon cloud was still in beta, with no SLA, and it wasn’t an option for a serious start up. Today, building a data center isn’t an option for a serious start-up.

Now we have availability zones, provisioning and monitoring tools, the ability to drop terabytes of data into the cloud — shipped through FedEx! But the real promise is the rich API and the spirit of community innovation. Companies like RightScale are finding a niche in the cloud, developing something really valuable, and then selling it as a simple service that makes our lives so much easier.

It’s exciting to see this happen so fast.  To avoid being crushed by this wave, as an IT team, you need to really stay on top of it. IT in our part of the ecosystem is becoming more the art of selecting, deploying, integrating, managing and supporting cloud based services, and much less the craft of building serious web infrastructure.

It’s a little sad for the hardware geek in all of us, saying goodbye to the roar of the servers, putting down the Velcro ties and picking up some slick provisioning and automation scripts. But I think we could get used to deploying 50 servers in a few keystrokes from a quiet, comfortable seat in the office.

Share this: