Archive for the ‘enterprise’ tag
Moving Beyond MDM for Custom iOS Solutions
I’m really excited about several new iOS development and deployment projects that we’ve been working on at CG. We’re working closely with Apple on a bunch of solutions: at the most basic level, we’re building solutions for security and management of employee iPad and iPhone use; at the other end of the spectrum, we’re helping to realize visions such as a kiosk-like platform of thousands of iPads deployed in retail environments around the country.
We’ve learned a ton about what is and isn’t possible as we strategize ways to scale to thousands of units. Here are some of the challenges we’ve come across:
- How do we deploy and support iPads – whether ten or ten thousand – in a secure, efficient, and centralized way?
- How can we architect kiosk-like application experiences on the iPad, enabling us to design and curate the customer experience, while also allowing a true iPad experience complete with app-switching, web browsing, Facebook-checking, game-playing, and movie-watching?
- What kind of network and server architecture is needed to support a platform of iOS devices across the globe? How do we enable caching and pushing of dynamic data to the devices – particularly large amounts of media content?
Centralized deployment and support of iOS devices
How do we deploy and support thousands of iPads or iPhones in a secure, efficient, and centralized way? Mobile Device Management (MDM) platforms like AirWatch, Casper, MobileIron – and soon, OS X Lion Server – allow us to push XML configuration profiles to iOS devices. This enables centralized inventory and basic management of the devices: from what version of iOS they have installed, to some security control over how/if users can install and delete apps. For many enterprise customers, these tools are useful for administering security policies on employee-owned iOS devices. But for custom platforms like kiosks and retail experiences, MDM is not ideal due to the need for end-user interaction. What we need is a way to easily restore iOS devices back to their “golden” state in a centrally managed way.
We’re excited about the potential of over-the-air restores and software updates coming in iOS 5, but as of today, iTunes is the only game in town for this. Working within this limitation, we’ve architected some innovative solutions that enable iOS devices to connect to iTunes virtually over USB to IP converters and a content distribution infrastructure. Until iOS 5, this is a good option to have, and I haven’t heard of anyone else embracing this approach.
Rearchitecting Apple’s iOS user experience
Put an iPad in front of someone and they’re going to tap, scroll, pinch, and squeeze the user interface. The user experience is still the leader in the tablet space – though we’ve been recently impressed by the BlackBerry PlayBook. For a project we’re working on now, we want to encourage this user experimentation and interaction, while locking down some important components of the UX. Things like App Store purchases, iTunes downloads, deleting apps, rearranging icons, and changing the home screen wallpaper will quickly affect the kiosk experience. MDM solutions can help disable some of these features, but the aforementioned need for user interaction just doesn’t work for specialized user environments.
One solution we’ve had success with is a combination of custom code to disable user customization of the Springboard, plus a WebKit-based Safari replacement for browsing that enables us to prevent user download of unauthorized content. Combine these with some configuration profile-based customization of iOS and we have a good solution for locking a customer experience down and reducing the frequency of unit restores or reimaging.
The CG approach to iOS projects
Part of what makes CG stand out as a solution provider is our deeply embedded collaboration between our application development team and our infrastructure team. As the Enterprise’s appetite for customized mobile platforms and experiences grows, we’re uniquely suited as a technology partner to build and innovate on our customers’ vision. iOS is at the core of this vision and I couldn’t be more excited to be working with these technologies today. Plus, iOS 5 is on its way and it’s shaping up to be a giant leap forward!
Everything Fails Sometime
Control Group designs cloud-based solutions with the philosophy that every system fails at some point. Embrace this chaos and build for the rainy day. Today we are seeing some major outages on Amazon’s us-east-1 region. Reddit and Quora are two of the high profile victims, but this is affecting everyone in a very popular data center.
You can design around regional performance degradation though. Years ago, having global traffic management in place was an expensive pipe dream. Today you can easily turn up another EC2 region and use a service like Dynect or Akamai GTM to provide failover and/or load balancing. Even better, consider making your systems portable so you can have multiple cloud providers and maintain your machines and applications with Puppet.
3-5 years ago this would have taken a year of planning, purchasing and hands-on labor to implement two data centers. Earlier this year we were able to create two data centers with complex infrastructure on EC2 and active/active load balancing in under two months and for a fraction of the cost.
The Public/Private Debate
I thought Phil Wainwright’s most recent article on private clouds (as well as the first in the series) was an interesting perspective. I share many of the sentiments, but can’t go quite so far as to say the idea of the private cloud is discredited. In the end it depends on the business and it depends on the applications you are hosting. Even Adrian Cockcroft, the writer of the blog that Phil cites as final proof, has updated his post to say:
“…to clarify, that doesn’t mean that I’m against private clouds or don’t think they exist, because $, FUD and internal politics are a fact of life that constrain what can be done.”
Private clouds, whether hosted or self-hosted, can be useful as stepping stones for organizations that have existing applications that may not fit into the public cloud architecture. Some of these applications may require better performance SLA’s. Also, private cloud providers are more amenable to custom arrangements. Try hosting a specialized device like an IPS or IDS in a public cloud where all traffic is guaranteed to only be delivered to the target device. Many enterprise organizations have decades of IT security policies that won’t and perhaps shouldn’t simply go away in favor of adopting a public cloud. Public clouds are secure solutions, but some organizations will have additional requirements, like the ability to discern rogue traffic patterns from typical spikes in demand. You can build this into the individual instances and applications, but that isn’t what most companies have done.
We tend to work with a client to find out what their requirements are and stay away from radical statements. The cloud, public or private, is just one more tool and can’t be seen as a solution in and of itself.
Adventures with Enterprise Firewalls, Elastic IP’s and Auto Scaling
One distinction between our startup and enterprise clients is that enterprise typically brings the baggage of legacy systems. While a startup is designing for a cloud architecture, a company that has a technology history sometimes needs to integrate new systems with existing services.
In a recent engagement Control Group needed to work with a client to have application instances on EC2 communicate with a secured web service in a traditional data center. Typically we would work with a client to move this service to EC2. In this case, because the service is considered to be shared infrastructure that is used and funded by existing applications we needed to design the infrastructure and application to make a call back to a traditional data center.
On a side note, mixed infrastructure approaches are not ideal, but common when migrating complex organizations to IAAS solutions. Most mature IT organizations will shy away from forklifting a company’s technology platforms wholesale into the cloud. The larger the migration, the bigger the bang when something is overlooked. Change too much in an environment and you won’t know where the problems are coming from, so a major part of moving an enterprise customer to the cloud is planning the roadmap of the migration carefully and not being greedy.
One of the technical challenges in this particular project was that the service that we were integrating with requires that traffic originate from a known and registered IP address. Although EC2 will provide an instance with a public IP address, there is no way to know what that address will be ahead of time. We decided to use Elastic IP (EIP) addresses to solve this problem. An EIP functions like a NAT on a traditional firewall. You can allocate the EIP and then associate it with an instance as needed.
EIP’s worked well until we implemented auto-scaling. Auto-scaling groups have no support for associating a pre-allocated EIP to an instance. To implement this we created some scripts that would make the API calls to determine a free EIP and associate it to the instance. (This means that the instance will have temporary access to execute API commands. We’ve designed a fairly secure take on temporarily providing AWS API tools to an instance, but that is a different blog post. Coming soon.)
Here is the real problem with the approach. The script to associate the EIP worked perfectly, so long as multiple machines weren’t executing it at once. The problem is that the Elastic IP API commands do not support a transactional assignment. Worse yet, at least in our use case, it is the last instance requesting the EIP and not the first that gets associated to the IP. This is a major problem if you want to associate EIP’s with members of an auto-scaling group that need to scale up by more than one instance at a time. It will leave you with members of the group that could possibly not have an Elastic IP.
There are a myriad of ways to tackle this issue. We considered options for programatically brokering the IP’s by building an application that would manage the EIP resources. The application would provide an IP on request and then return IP’s that were no longer in use back into the system through a background recovery process. Such a service is pretty easy to write, but it wasn’t in scope for the current project. Also, there are longer-term solutions that we can consider with the launch of the new and improved VPC with NATing.
Interim Solutions
The current favored approach is to use a proxy server like Squid to limit the number of servers that require IP addresses. Two or more instances with Squid configured as a forward proxy distributed across multiple availability zones and traffic managed by an Elastic Load Balancer to provide HA would provide a redundant and fairly high performance solution. For now, as a work around we have implemented some staggering of the auto-scaling policies as a way to mitigate against multiple instances spinning up at the same time. Staggering is a serviceable solution for testing, but not for production where auto scaling multiple farms of servers that will need access to the client’s data center tier is a requirement. Eventually, we will move forward with the proxy or VPC solution.
In summary, enterprises with complex interdependent applications can lead to interesting challenges when migrating to the cloud. Resources, as simple as IP addresses, can function in a fundamentally different way than a typical IT organization is used to. Oftentimes this can lead to fear, uncertainty, and doubt, but the benefits of Infrastructure as a Service are clear: Ease of provisioning, demand-based resource allocation rather than over provisioning, etc. As long as proper planning, system architecture, implementation, and testing are performed, a complex enterprise can begin making its way to the cloud and begin to eliminate the FUD on the ground.
Enterprise Clients Continue To Warm To The Cloud
Lately we’ve been working with clients that haven’t been the typical EC2 infrastructure consumer. Historically, it has been the startup companies that we work with that have been interested in AWS for all the expected reasons: flexibility, pay-for-what-you-need, access to higher end services like load balancing and HA database deployments, etc. Recently we have been noticing that our more established enterprise clients have taken interest in these capabilities and for largely the same reasons.
Large enterprises looking at cloud infrastructure bring their own requirements and challenges. We plan to write a series of blog posts about Control Group’s experiences with these types of clients and what we learned. Some of the posts will be about the projects and their politics, and some will be about technology approach. There are some interesting technology and organizational challenges that we will discuss, so stay tuned.
Goodbye Xserve. Now what?
Today, Apple announced that the Xserve will no longer be available for purchase after January 31, 2011. What does this mean for existing and future infrastructure that relies on Mac OS X Server and Xsan?
For existing Xserve environments, Apple will continue to provide warranty service and complimentary technical support for the product. This means that all AppleCare service and support agreements should be honored until they expire.
Apple is providing an Xserve Transition Guide with information on options moving forward. They suggest that customers looking for Mac OS X Server solutions move to Mac Pro or Mac mini hardware solutions. We have had great success with these solutions for providing basic services such as file sharing, directory services, and calendaring to small- to medium-sized workgroups.
But what about Xsan environments? Xsans could be built using Mac Pros for metadata controllers, with a few serious considerations — we lose the power redundancy and lights out management (LOM) that Xserve provides. Also, this solution will require 12U of rack space for two Mac Pro servers instead of 2U for two Xserves, which is not very appealing to customers designing server room rack elevations.
This is also an opportunity to discuss alternative SAN solutions, such as Quantum StorNext, which is compatible with Xsan. Control Group has had recent successes in deploying StorNext as an alternative to Xsan, allowing users and organizations to continue to use the Apple tools they are familiar with, such as Final Cut Pro, while leveraging a robust, Linux-based infrastructure in the server room. StorNext has a very rich feature set and does some things that are not possible with Xsan, such as hierarchical storage management.
If you remember, a few years ago Apple discontinued the Xserve RAID storage solution, the IT world panicked, and then Apple announced a partnership with Promise and the Promise Vtrak for Mac solution. Maybe Apple has similar plans for a replacement for the Xserve. Whether they do or not, there are great alternatives to discuss, so if you have questions or concerns, give us a shout.
Back from NAB… in 3D!
Whew! As always, NAB was quick, productive, and overwhelming – only this time it was in 3D!
Everyone, everywhere, was talking about 3D: cameras, displays, production software, trucks, expertise, etc. The race for preparedness is on, and people will spend as pushy studios and producers want to be the first to do this or that with 3D. First sitcom, first sporting event, first documentary, first newscast, etc. The reality is that we are a ways off from wide-scale adoption, and showrooms and special screenings will be the place for 3D for the foreseeable future. We are in a similar catch-22 phase as a few years ago during the early days of HD production – 3D TVs are just coming onto the market and will drop down to reasonable consumer level prices within the next 2-3 yrs. Consumers are asking “do I buy a 3D TV when there is little to no programming?” while content creators are wondering “do I produce in 3D when there is little to no audience?”
Being an industry event though, it’s about more than bragging rights or audience – it’s about technical feasibility, practice, and logistics – all things which will change when 3D comes to our living room. DirecTV has announced that they will carry four 3D channels starting in June, including ESPN 3D and a dedicated 3D pay-per-view channel. And Cablevision dipped its toes into the 3D pool a few weeks ago with an MSG Network broadcast of a Rangers and Islanders hockey game live from Madison Square Garden.
After talking to a few industry experts who participated in the recent Masters in 3D, the challenges with 3D production are less technical than logistical: camera placement for example. While HD favors wide top-down shots for seeing all the action, 3D is most effective with close, ground-level cameras – imagine Phil Mickelson’s birdie putt as viewed from grass-level, just across the green, the ball rolling right toward you as he sinks it…
There were also lots of other things of interest: Falconstor’s HyperFS, CatDV asset management, Avid’s Java app for editing over the web, 3ality cameras, Adobe CS5, among many more. We were particularly excited by the potential of Active Storage’s Innerpool appliance for metadata. This PCI Express card contains on-board redundant solid-state drives, specifically engineered for storing metadata in an Xsan environment. This has the potential of being something of a game changer, allowing us to more efficiently configure the storage in our Xsan integrations, and giving our clients more bang for their buck when deploying new SAN solutions.
We had some great meetings with prospective clients, old and new friends, fellow consultants and vendors, and we’re excited about some emerging strategic opportunities. Our work in online video technology and web delivery combined with our broadcast infrastructure and workflow experience means we are ideally positioned to help organizations streamline and bring these workstreams closer together.
A Look at Amazon’s Elastic Load Balancer
We have been doing some work with with Amazon’s Elastic Computing Cloud (EC2) which allows us to create virtual machines in the cloud in a few seconds. These are great for hosting websites, and what’s cool about them is that if you get Slashdotted or experience a similar unexpected spike in traffic you can create new hosts immediately. Recently Amazon added a new service called Elastic Load Balancing (ELB) which can distribute load across hosts. We’ve been looking at this for some of our recent development and infrastructure projects.
I just read this description of how ELB works by Shlomo Swidler from his Cloud Developer Tips blog. It’s a great reference.
You pay for ELB by usage just like everything else at AWS. From Amazon: “You are charged at $0.025 per hour for each Elastic Load Balancer, plus $0.008 per GB of data transferred through an Elastic Load Balancer.” For reference, on a deployment project in 2008 our Engineering team used a Cisco load balancer which I imagine cost a few thousand bucks.
Cost isn’t the only advantage. These can be created and destroyed quickly and remotely, allowing us to work more efficiently and spend less time visiting data centers in the middle of nowhere. This leads to improved quality of service for our clients as we can spend more time consulting on future technology growth plans and less time troubleshooting servers in cold, loud data centers.
This blog post brought to you by the iced coffee I am enjoying in the comfort and quiet of my office while deploying virtual machines!
Testing Storage Performance with iozone
As I’ve mentioned in previous posts about testing storage performance with lmdd and bonnie++, different applications require different characteristics from storage to provide the best performance. I’ve highlighted some tests that are good for large streaming files like video, and small file transactions like databases or mail servers. Today I want to look at a tool that runs a series of tests in many different ways to provide you with a holistic view of what the storage can and can’t do.
This tool is called iozone. iozone is open source and runs on a ton of operating systems (including Windows). It runs several tests which can take some time to complete but provide the best overall view of the capabilities of a piece of storage. For instance, iozone runs a write test with files of different sizes and with different size records (the amount of data written at a time). It does this over and over again with writes, reads, random writes, random reads, and so forth. Since it’s running all these tests you can see what sorts of operations will have good performance and which ones will not perform so well. Check out the iozone documentation here.
One really great thing about iozone is that the output it generates can be easily placed in a spreadsheet program like Excel to generate a great 3d diagram describing your storage. Here’s a diagram I generated from some tests on a Linux server.
Results of a write test with iozone
This particular server performed quite well with large files and a record size around 1 MB (interesting to note, this is the same storage from the lmdd post. Notice that the parameters I tested with there are the same as the best write that this disk can do according to iozone!).
If you’ve been following my posts on storage performance testing I hope you’ve learned about some new tools that you can use to see what’s going on. I use these on every deployment to make sure we’re giving our clients solutions that they can depend for performance and reliability. As always, let me know if you have any questions about these tools. Happy testing!
How The Cloud is Changing IT Services
Were getting ready for an event with Google and Mozy that we have dubbed “CloudSourcing”, taking a note from Gartner and tweaking it a little.
Tom Mills from Google and Sean Finnegan from Mozy will be giving an in-depth review of their offerings and how they fit into an agile, post-recession office technology strategy.
I’ll be giving a brief overview of how I think we arrived at this point in IT and what it means for creative, innovative firms that are trying to do more with less.
In an effort to get my thoughts together and get some feedback, I’m using this blog post as a draft for the event.
Let me start off by giving a brief overview of our services, and then a little history about the evolution of our offerings:
We provide a number of technical services for our clients in the areas of infrastructure, application development, and industry-focused workflow consulting. As this is New York, we work with a number of creative firms; media, architecture, publishing, and design companies, as well as some key clients in the financial sector. We strive for long-term relationships with our clients, many of whom we’ve worked with for close to a decade. We have installed and managed hundreds of servers, network devices and application suites, but more recently we’ve been focusing on helping our clients select, migrate to, integrate, and manage Cloud-based services.
Since the 1990s and the introduction of pervasive bandwidth, we’ve gone through a number of permutations of the remote server/client model, and much has been written about the benefits and the irony of the shift back to the mainframe/thin client structure of the 1960s. Now everyone is talking about the future of ‘The Cloud‘; a vast array of computing resources, abstracted and presented as a single source to the consumer.
At the turn of the century, we found most small to mid-sized businesses with a pure Local Area Network (LAN), typically comprised of in-house mail – most likely Exchange – and a few other local services: file, print, etc. A lot of these firms had an internal IT staff or a dedicated consultant to manage their servers, tape backup, networks, and desktops. Only a few were pushing the envelope by leveraging Application Service Providers (ASPs) to deliver back office services.
The risks with this situation were obvious. These systems mostly depended on a single Internet connection, a single building, and a single individual, prone to career changes and untimely vacations. Remote access to these in-house services was expensive to do right and applications rarely worked as well remotely as they did in the office.
Over the next five years, we saw a gradual shift towards ‘Hosted Applications’. This typically came in the form of a service provider taking a LAN-based solution like Exchange or SharePoint out of the office and putting it in a data center. In conjunction with this change, we saw the IT services industry begin to shift its focus from in-house IT, or consultants, to managed services – companies providing regular systems management remotely.
There were some benefits to this offering: critical applications were not dependent on intermittent Internet connections or over-heated server rooms. Flaky consultants were traded for predictable management services and cost became as regular as the electric bill.
But there were still problems. We had the same old model of doing things, only it was moved out of the business’s office and into the provider’s. Services that were built for an onsite installation and LAN speeds were shifted to a remote location – not always producing the best results. Access to applications designed for the LAN was sometimes unacceptable because of bandwidth and latency. In a similarly narrow view of the problem, Managed Service companies focused on monitoring systems and patching software, maintaining the status quo, without looking at the big picture, or driving the business forward.
Now the next generation of IT services is coming along and delivering on the promise of on-demand, scalable solutions. These services are web-native, built for the Cloud and multi-tenant environments.
As services like Google Apps and Mozy were built for the web – not re-purposed LAN applications – they deliver exceptional performance and remain very flexible. Control Group has designed our support and project services in a similar way. Our services are built to function efficiently remotely – scaling up when our clients need it, and going away when they don’t – and also to be flexible and innovative, driving business forward rather than maintaining the status quo.
Using the cloud paradigm, we act as a single source of technology for our clients. We help them run more efficient, profitable businesses by weaving an ever growing selection of web-based services, traditional IT, and industry expertise together, to provide an flexible, competitive business platform.