We’ve learned a ton about what is and isn’t possible as we strategize ways to scale to thousands of units. Here are some of the challenges we’ve come across:

• How do we deploy and support iPads – whether ten or ten thousand – in a secure, efficient, and centralized way?
• How can we architect kiosk-like application experiences on the iPad, enabling us to design and curate the customer experience, while also allowing a true iPad experience complete with app-switching, web browsing, Facebook-checking, game-playing, and movie-watching?
• What kind of network and server architecture is needed to support a platform of iOS devices across the globe? How do we enable caching and pushing of dynamic data to the devices – particularly large amounts of media content?

Centralized deployment and support of iOS devices

How do we deploy and support thousands of iPads or iPhones in a secure, efficient, and centralized way? Mobile Device Management (MDM) platforms like AirWatch, Casper, MobileIron – and soon, OS X Lion Server – allow us to push XML configuration profiles to iOS devices. This enables centralized inventory and basic management of the devices: from what version of iOS they have installed, to some security control over how/if users can install and delete apps. For many enterprise customers, these tools are useful for administering security policies on employee-owned iOS devices. But for custom platforms like kiosks and retail experiences, MDM is not ideal due to the need for end-user interaction. What we need is a way to easily restore iOS devices back to their “golden” state in a centrally managed way.

We’re excited about the potential of over-the-air restores and software updates coming in iOS 5, but as of today, iTunes is the only game in town for this. Working within this limitation, we’ve architected some innovative solutions that enable iOS devices to connect to iTunes virtually over USB to IP converters and a content distribution infrastructure. Until iOS 5, this is a good option to have, and I haven’t heard of anyone else embracing this approach.

Rearchitecting Apple’s iOS user experience

Put an iPad in front of someone and they’re going to tap, scroll, pinch, and squeeze the user interface. The user experience is still the leader in the tablet space – though we’ve been recently impressed by the BlackBerry PlayBook. For a project we’re working on now, we want to encourage this user experimentation and interaction, while locking down some important components of the UX. Things like App Store purchases, iTunes downloads, deleting apps, rearranging icons, and changing the home screen wallpaper will quickly affect the kiosk experience. MDM solutions can help disable some of these features, but the aforementioned need for user interaction just doesn’t work for specialized user environments.

One solution we’ve had success with is a combination of custom code to disable user customization of the Springboard, plus a WebKit-based Safari replacement for browsing that enables us to prevent user download of unauthorized content. Combine these with some configuration profile-based customization of iOS and we have a good solution for locking a customer experience down and reducing the frequency of unit restores or reimaging.

The CG approach to iOS projects

Part of what makes CG stand out as a solution provider is our deeply embedded collaboration between our application development team and our infrastructure team. As the Enterprise’s appetite for customized mobile platforms and experiences grows, we’re uniquely suited as a technology partner to build and innovate on our customers’ vision. iOS is at the core of this vision and I couldn’t be more excited to be working with these technologies today. Plus, iOS 5 is on its way and it’s shaping up to be a giant leap forward!

Share this:

I thought Phil Wainwright’s most recent article on private clouds (as well as the first in the series) was an interesting perspective. I share many of the sentiments, but can’t go quite so far as to say the idea of the private cloud is discredited. In the end it depends on the business and it depends on the applications you are hosting. Even Adrian Cockcroft, the writer of the blog that Phil cites as final proof, has updated his post to say:

“…to clarify, that doesn’t mean that I’m against private clouds or don’t think they exist, because $, FUD and internal politics are a fact of life that constrain what can be done.”

Private clouds, whether hosted or self-hosted, can be useful as stepping stones for organizations that have existing applications that may not fit into the public cloud architecture. Some of these applications may require better performance SLA’s. Also, private cloud providers are more amenable to custom arrangements. Try hosting a specialized device like an IPS or IDS in a public cloud where all traffic is guaranteed to only be delivered to the target device.  Many enterprise organizations have decades of IT security policies that won’t and perhaps shouldn’t simply go away in favor of adopting a public cloud.  Public clouds are secure solutions, but some organizations will have additional requirements, like the ability to discern rogue traffic patterns from typical spikes in demand. You can build this into the individual instances and applications, but that isn’t what most companies have done.

We tend to work with a client to find out what their requirements are and stay away from radical statements. The cloud, public or private, is just one more tool and can’t be seen as a solution in and of itself.

Share this:

In the midst of all of this, a few of us at Control Group have been developing Tagatag: an Android and iPhone application for commenting on barcodes that uses web services running on Google App Engine.

Scan this QR code with Tagatag to join the conversation!

Barcodes are everywhere around us. You can find them on advertising, products, places and even people. Tagatag provides you with a virtual paint marker to let you make your mark on all of these codes anonymously. Download the Tagatag app and give it a try. Scan a barcode to see comments people have left for you and then leave some for them.

We chose Google App Engine for the back end of Tagatag for a few reasons:

• It’s quick – You sign up for an account, download the SDK and you’re developing. The development server in the SDK lets me run the application on my desktop and interact with the code as I’m writing it.  Uploading new versions, rolling back old ones, or performing maintenance is a snap with the GAE dashboard.
• It’s simple – There’s not much to the web service. It’s small and simple. We used the webapp framework because we didn’t feel we needed anything else. It makes for a very concise application. Believe it or not, there are about 300 lines of code for the GAE part of Tagatag.
• It’s scalable – We don’t have to worry about what we do when Tagatag becomes popular. We’ll just raise our billing quotas in GAE and let them handle spinning up new instances or expanding the datastore. Knowing that you don’t have to be concerned about scaling makes things a lot more fun.

I’m happy that GAE let us bring Tagatag to you so quickly. So, when it’s available at the end of the week, be sure to download the app, tag a tag and make your mark!

Share this:

For existing Xserve environments, Apple will continue to provide warranty service and complimentary technical support for the product. This means that all AppleCare service and support agreements should be honored until they expire.

Apple is providing an Xserve Transition Guide with information on options moving forward. They suggest that customers looking for Mac OS X Server solutions move to Mac Pro or Mac mini hardware solutions. We have had great success with these solutions for providing basic services such as file sharing, directory services, and calendaring to small- to medium-sized workgroups.

But what about Xsan environments? Xsans could be built using Mac Pros for metadata controllers, with a few serious considerations — we lose the power redundancy and lights out management (LOM) that Xserve provides. Also, this solution will require 12U of rack space for two Mac Pro servers instead of 2U for two Xserves, which is not very appealing to customers designing server room rack elevations.

This is also an opportunity to discuss alternative SAN solutions, such as Quantum StorNext, which is compatible with Xsan. Control Group has had recent successes in deploying StorNext as an alternative to Xsan, allowing users and organizations to continue to use the Apple tools they are familiar with, such as Final Cut Pro, while leveraging a robust, Linux-based infrastructure in the server room. StorNext has a very rich feature set and does some things that are not possible with Xsan, such as hierarchical storage management.

If you remember, a few years ago Apple discontinued the Xserve RAID storage solution, the IT world panicked, and then Apple announced a partnership with Promise and the Promise Vtrak for Mac solution. Maybe Apple has similar plans for a replacement for the Xserve. Whether they do or not, there are great alternatives to discuss, so if you have questions or concerns, give us a shout.

Share this:

Fortunately Control Group’s approach to formidable projects allowed us to complete the project smoothly and on schedule. As noted in the aforementioned article, the trick is to break everything down into manageable tasks, and identify which of those tasks are best addressed with technology, rather than manpower. The workstation setup for the two shows illustrates the benefits of this approach nicely.

The Colbert Report and The Daily Show needed nine new workstations set up for the artists, including installing several 2D design suites, 3D design applications, and a host of supporting plugins. Several render farm servers had to match these workstations setups so that the creation of HD elements could be distributed. Since this is a fair amount of equipment to arrange, we wanted to first be sure that the artists would be comfortable in their new environment. We created a “perfect” workstation for each show, and invited the end users to try them out at our office.

After making a few adjustments based on user feedback, we duplicated these master machines to their brethren using OS X’s excellent Apple System Restore utility. Used in concert with a networking technique known as multicast, we were able to get all of the workstations set up simultaneously, saving countless hours of manpower.

The time savings allowed us to get a jump on the real time sink of setting up design workstations – installing plugins. Due to the way many plugins are licensed, the installations had to be performed individually as each user. A team of Control Group employees attacked the 225 some-odd separate installs, completing them over the course of a day or two. Afterwards, specially crafted project files allowed us to test all the workstations and render nodes in a single shot, ensuring that the artists wouldn’t be confronted by a licensing dialog at show time.

As demonstrated at The Colbert Report and The Daily Show, a carefully planned approach allowed Control Group to assist NEP in completing their HD rollout on time and within the budget allotted. The satisfaction of watching both shows in full frame 1080p HD has been well worth the effort. (Getting to hear Stephen Colbert riff on an auto-tuner backstage wasn’t half bad either.)

Here’s a clip of Stephen Colbert enjoying the new HD setup.

Share this:

Tux the Linux Penguin

I work with Linux quite a bit at CG and it comes up in some interesting places. Consistently we find that the largest and smallest systems we work with are running Linux. If it’s a large system, like a database or storage network, Linux can provide the stability, robustness, and uptime required to make the project a success. If it’s a very small system, like an embedded display or a custom purpose device, we choose Linux because of its flexibility.

Over the last year or so, we’ve had a need to spin up many more virtual computers than we used to. It’s necessary to create a handful of  computers for each project we work on and the number of projects that we are taking on at a time is growing. Tools like EC2 and VMWare have made it very easy for us to create new virtual computers, but what about managing them? For Linux there are some great solutions. In this blog post I would like to talk about how we’re using some of the tools to deal with the large number of machines we are managing in Amazon EC2 for ourselves and our clients.

Cloud Control

The first part of the system is Amazon’s Elastic Compute Cloud (or EC2 for short). I’ve written about EC2 before, but to be really brief about it: EC2 allows you to create computers on the Internet quickly and lets you pay by the hour for their use. The instances you create are virtual machines that run in a large Xen based infrastructure that Amazon provides. To customize the software and configuration of the instances, Amazon lets you create snapshots of your computers (called Amazon Machine Images or AMIs) that you can launch new instances from.

It’s hard to say that making an AMI is difficult, it’s just a few keystrokes and a coffee break while the image is prepared and stored for your future use. When you begin to manage dozens and dozens of different images this becomes a problem. It became clear to us that we needed to have one AMI to manage, and build every computer from image dynamically. To do this we needed a way to pass specific information to each instance when it started up and then have a tool customize the instance based on the specific information about it. Amazon provides a method to pass specific information to an instance (it’s called user-data) and the good folks at Alestic have made some great AMIs around it with some good documentation.

Enter the Puppet Master

We have been using a tool called Puppet for nearly two years now to manage a handful of computers. We selected that to manage the whole lot of Linux computers that we were dealing with. Puppet allows us to describe how a computer should work in a general way. We can make collections of configurations for standard things so it’s easy to reuse what we create over and over again. Our Puppet configuration is stored in a Git repository so many administrators and developers can collaborate on it and our server configurations are backed up and version controlled automatically.

Bringing it all together

Puppet decides what configuration to use for a computer based on its hostname. We use the EC2 user data to pass a new instance the address of a Puppet server and the hostname that the machine should assume. When it boots up it sets its hostname and checks in to receive the configuration that we’ve stored for it. All changes to the machine happen through Puppet so we don’t have to spend a lot of time SSHing in and customizing the machine. It’s also very easy for us to duplicate a machine for testing or whatever we need.

While we originally did this to make our lives easier as we manage more machines there turned out to be some really cool side effects:

• Excellent Security: We don’t want to store sensitive information in Puppet. This means no passwords or secret stuff. To resolve this we require all developers and administrators to use key-based authentication to get access to the computers via SSH. This is very handy and it eliminates the need to remember passwords or for an administrator to have to reset passwords for users. Someday I would love to take the next step and have all of our users and machines be part of the Monkeysphere.
• Accountability: All of our configuration is tracked in a Git repository so we can see the history about what has changed on certain hosts and who changed it. Change control occurs automatically with Puppet and it’s easy to examine and understand what has changed and why it changed.
• Repeatability: By storing all Linux computer configuration in a single place we can easily repeat what we did for one server on another. Puppet institutionalizes all of our Linux knowledge in one place and saves us time every time we have to create a new machine. If you want to see how someone has done something in the past there’s no more need to dig through emails or documentation, just look at the facts in the Puppet repository and even copy and paste it into your new configuration.
• Portability: We use Puppet to manage much more than just EC2 instances. Physical machines and virtual machines in our ESX installation are managed this way too. It gives us one tool to take care of any Linux machine we deal with. Puppet also supports other operating systems. We’re looking to expand our use of it to Mac OS X machines and maybe even Windows sometime soon.

There are certainly other ways to solve the problems we’re up against, but this is the way we chose. We did some extensive evaluation of Chef (which is a tool like Puppet) and we’ve used Rightscale before. This sort of thing is becoming very important as we manage more and more computers. I expect we’ll see a lot of exciting products, techniques, and services in this space as time goes on.

If you have any questions or comments about our setup, or would like to discuss implementing something like this for your business, leave a comment or get in touch with us. We’d love to help.

Share this:

When I started searching for a way to connect an audio feed to a gateway router, I found that there were no great turnkey solutions. What I did find is that there is a way to make use of an available FXO port to bring a live feed into the system with some inexpensive parts. The solution I have deployed requires a surface mount RJ 45, a chassis mount RCA jack, and a stereo 1/8″ mini to RCA adaptor. All of these pieces can be found at a nearby electronics parts store and purchased for under $20. The FXO port and an iPod will set you back some but if you want easy control of your hold music it may be worth it.

Build the Adaptor

To make the adaptor, open up the surface mount box and figure out how you will attach the RCA jack. In one case I used a drill bit to make a hole for it, in another case I found a form factor where the notch for a cat 5 cable fit the connector perfectly. You may also find a general purpose surface mount with modular fittings where an RCA “snap in” would snap right in.

Once you’ve sorted that you will need a pair of wires from either a solid or stranded piece of cat 5. Punch these down to pins 4 and 5 of the RJ45 and solder the other side to the RCA jack.

When you are done it should look like this.

Adaptor wiring detail. Though the jack in this configuration is an RJ 45 you can use a standard RJ11 phone cord to connect between the adaptor and the FXO port.

Configuring the Voice Gateway

Configuring the voice gateway is pretty straightforward:

1. Create a DN for the MOH call out with an unused multicast address

ephone-dn 274
number 1 no-reg primary
moh ip 239.12.13.1 port 2000 out-call 899

2. Configure the available FXO port remain open to the iPod

voice-port 0/3/3
signal loopStart live-feed
input gain 2
description To MoH Live Feed

3. Create a dial peer to connect to the FXO port

dial-peer voice 55 pots
destination-pattern 899
port 0/3/3

4. Configure CME to use the multicast for MOH

telephony-service
multicast moh 239.12.13.1 port 2000

Then Shut/No shut the voice port to invoke the change.

I have been using one of the built in USB ports on our gateway here at Control Group to power the iPod which turns out to be very convenient.

Safety and Other Considerations

There are some very important caveats.

1. The adaptor I describe is potentially dangerous. If you were to connect an iPod to a POTS line or an FXS port feeding a battery signal you could damage the iPod or the device connected to it. You could also receive a serious electrical shock.
2. The use of copyrighted material for Music on Hold can be a violation of the rights of the copyright owner.
3. The adaptor will only work with one channel of audio from the iPod. If you use a mono 1/8″ to RCA cable it will short out the other channel.

These issues need to be minded if you would consider this solution. Also, keep in mind that people’s taste in music varies widely. Implement Music on Hold and you will find out just how much!

Miles Green is a Senior Network Engineer at Control Group and worked as a Musician and Recording Engineer in a previous life.

Share this:

I do some work on implementing storage systems for our clients, and we’ve found that different applications have different storage requirements. For example a video post production facility — like the facility at WWE — generally needs lots of disk space that is very good at reading and writing large files at high speeds. The storage here needs to provide good streaming throughput, because high quality video files generally have high bit rates, and are being stored or played back from the disk in real-time for ingesting, editing, or playout. If the storage system is not fast enough to read or write the file in real-time, frames will be dropped. This can cause unsatisfactory media files, programs to crash, or audio and video to become out of sync.

10,000 RPM SAS disks. That's some serious storage.

Suboptimal read/write performance can become a huge problem. When we put in a new system this is something we need to test. I usually do the test with a tool called lmdd.

lmdd comes from the lmbench tools which are provided by Bitmover for benchmarking systems. lmdd is great for testing streaming bandwidth. In most of our engagements with video, we install a Stornext or Xsan filesystem so we’ll run our tests against this. lmdd will probably work on any filesystem that you can mount on your Mac or Linux computer (Leave a comment if you need a version for Mac OS X, I have one compiled).  lmdd lets us verify exactly what the maximum number of megabytes per second we can push through the storage and point us to where we need to make changes to the hardware or software configuration. I use lmdd like this :

lmdd of=/path/to/test_file count=1g

lmdd if=/path/to/test_file

The first tests write performance and the second tests read performance. More information about the syntax is available in the manual page for lmdd. The results of the command from a server I was testing looked like this:

2147.4755 MB in 6.8003 secs, 315.7914 MB/sec

lmdd is great because it’s easy to read. This result shows I could write to the filesystem at 315 megabytes per second. That’s really fast! This is from a test with a server with a lot of RAM and a special filesystem that took advantage of that cache. When I run it on my Macbook, I get a result like this:

18342.6171 MB in 376.7685 secs, 48.6841 MB/sec

So the next time you’re interested in how your storage is performing give lmdd a shot and let me know how it goes. If you’re looking for more information about storage performance testing then stay tuned; I’ll be posting about testing storage with tools that benchmark small reads and writes next.

Share this:

An ASR Restore Image in Disk Utility

The disk images asr works with are the same format used by Mac OS X’s Disk Utility. This means you can do a rollout over the network and keep the master file on hand in your re-imaging kit, should one of your workstations run into trouble and need to be re-imaged over FireWire. To ensure the sanctity of the final result, disk images include an embedded checksum which is automatically verified during the deployment process. This can be a significant advantage in using asr over of Apple NetInstall, which requires its own folder-based setup of restore source files.

Disk images are also hardware agnostic for the most part. You can build your image on a Mac Mini and apply it to anything from a PowerMac G5, to an Macbook Pro so long as it can get on the same subnet as the asr host.

The usual caveats of disk imaging apply unfortunately. You’re going to have to sweat individual serial numbers if you don’t employ network or volume licensing. Settings like hostnames and non-ubiquitous local users will require individual workstation visits without centralized management. However, having asr around to do the heavy lifting means you may be able to turn a strenuous two day deployment into a breezy one day affair.

Share this: