Tux the Linux Penguin

I work with Linux quite a bit at CG and it comes up in some interesting places. Consistently we find that the largest and smallest systems we work with are running Linux. If it’s a large system, like a database or storage network, Linux can provide the stability, robustness, and uptime required to make the project a success. If it’s a very small system, like an embedded display or a custom purpose device, we choose Linux because of its flexibility.

Over the last year or so, we’ve had a need to spin up many more virtual computers than we used to. It’s necessary to create a handful of  computers for each project we work on and the number of projects that we are taking on at a time is growing. Tools like EC2 and VMWare have made it very easy for us to create new virtual computers, but what about managing them? For Linux there are some great solutions. In this blog post I would like to talk about how we’re using some of the tools to deal with the large number of machines we are managing in Amazon EC2 for ourselves and our clients.

Cloud Control

The first part of the system is Amazon’s Elastic Compute Cloud (or EC2 for short). I’ve written about EC2 before, but to be really brief about it: EC2 allows you to create computers on the Internet quickly and lets you pay by the hour for their use. The instances you create are virtual machines that run in a large Xen based infrastructure that Amazon provides. To customize the software and configuration of the instances, Amazon lets you create snapshots of your computers (called Amazon Machine Images or AMIs) that you can launch new instances from.

It’s hard to say that making an AMI is difficult, it’s just a few keystrokes and a coffee break while the image is prepared and stored for your future use. When you begin to manage dozens and dozens of different images this becomes a problem. It became clear to us that we needed to have one AMI to manage, and build every computer from image dynamically. To do this we needed a way to pass specific information to each instance when it started up and then have a tool customize the instance based on the specific information about it. Amazon provides a method to pass specific information to an instance (it’s called user-data) and the good folks at Alestic have made some great AMIs around it with some good documentation.

Enter the Puppet Master

We have been using a tool called Puppet for nearly two years now to manage a handful of computers. We selected that to manage the whole lot of Linux computers that we were dealing with. Puppet allows us to describe how a computer should work in a general way. We can make collections of configurations for standard things so it’s easy to reuse what we create over and over again. Our Puppet configuration is stored in a Git repository so many administrators and developers can collaborate on it and our server configurations are backed up and version controlled automatically.

Bringing it all together

Puppet decides what configuration to use for a computer based on its hostname. We use the EC2 user data to pass a new instance the address of a Puppet server and the hostname that the machine should assume. When it boots up it sets its hostname and checks in to receive the configuration that we’ve stored for it. All changes to the machine happen through Puppet so we don’t have to spend a lot of time SSHing in and customizing the machine. It’s also very easy for us to duplicate a machine for testing or whatever we need.

While we originally did this to make our lives easier as we manage more machines there turned out to be some really cool side effects:

• Excellent Security: We don’t want to store sensitive information in Puppet. This means no passwords or secret stuff. To resolve this we require all developers and administrators to use key-based authentication to get access to the computers via SSH. This is very handy and it eliminates the need to remember passwords or for an administrator to have to reset passwords for users. Someday I would love to take the next step and have all of our users and machines be part of the Monkeysphere.
• Accountability: All of our configuration is tracked in a Git repository so we can see the history about what has changed on certain hosts and who changed it. Change control occurs automatically with Puppet and it’s easy to examine and understand what has changed and why it changed.
• Repeatability: By storing all Linux computer configuration in a single place we can easily repeat what we did for one server on another. Puppet institutionalizes all of our Linux knowledge in one place and saves us time every time we have to create a new machine. If you want to see how someone has done something in the past there’s no more need to dig through emails or documentation, just look at the facts in the Puppet repository and even copy and paste it into your new configuration.
• Portability: We use Puppet to manage much more than just EC2 instances. Physical machines and virtual machines in our ESX installation are managed this way too. It gives us one tool to take care of any Linux machine we deal with. Puppet also supports other operating systems. We’re looking to expand our use of it to Mac OS X machines and maybe even Windows sometime soon.

There are certainly other ways to solve the problems we’re up against, but this is the way we chose. We did some extensive evaluation of Chef (which is a tool like Puppet) and we’ve used Rightscale before. This sort of thing is becoming very important as we manage more and more computers. I expect we’ll see a lot of exciting products, techniques, and services in this space as time goes on.

If you have any questions or comments about our setup, or would like to discuss implementing something like this for your business, leave a comment or get in touch with us. We’d love to help.

VH1 won a Broadcast Designers Association Gold award for their Beacon channel redesign! Check out the reel.

Walker Digital won the Gold medal at the Gaming Technology Summit.

And NEP Studios, who host the Daily Show and The Colbert Report at their Control Group-readied HD studios were nominated for a CIO 100 award from CIO Magazine.

Doing what you love, with great partners, and getting industry respect for it: what more could you ask for? What a great start to the summer! Congrats all.

I think you have to go back 10 plus years to find a gaming platform (I’m thinking wide area progressives?) that has altered the landscape as much as the Walker Digital platform; To quote the press release:

With integrated RFID technology, a card-reading shoe and Walker’s Core Game Technology, the system provides an upgradeable, networked game platform that exactly tracks turnover, hand outcomes, wagers, payouts and player ratings in real time. It also eliminates losses due to dealer error, cheating and counterfeiting and increases game speed, precisely assessing player value and providing detailed reporting on game speed, bet mix, table occupancy, turnover, theoretical win and actual win.

To boot, the ‘Elite’ version of the game adds a whole bunch of new and exciting bets — knowing the game, the cards, and the chips allows the system to present wagers never before possible. Say hello to bigger wins and more exciting wagers for the players and better house advantage to the Casino on one of the most popular card games in the world.

Of course the Judges were ‘duly impressed’ who wouldn’t be?

“This product would resolve many of the problems casinos currently deal with regarding very high-end baccarat play,” said one. “It automates many of the functions of administering baccarat play, but does so in a manner that would not be intrusive to the serious high-end customer.”

Another judge noted the product would “work well in high-volume baccarat markets where the clientele prefers to handle the chips.” The features he liked best included the combination of accurate real-time bet recognition and optical card reader. “Together they allow for complete tracking of game play,” he said.

Congrats to the WDG team for pulling it off!

Entertainment = Consumption + Interaction:

Internet-enabled TV will bring about some radical changes.  Nielsen’s Three Screen Report indicates that consumption of the moving image is increasing across all three screens: traditional TV, internet, and mobile. The interactivity of this “data” will surely change. Entertainment now equals both watching and interaction: checking out additional content online, playing games, or participating in ads as entertainment. 3D and gesture-based interactions will also redefine this blurring of the line between passive and active viewing over the next few years.

Ditch the “dumb” set-top:

Content models that rely on “captive audience” set-top box capture of viewing habits are outmoded as delivery systems, and deliver poor analytics and reporting compared with the information we can glean from a data-only model. If content owners rely on this information, why does the old model remain unchecked? The set-top box is largely a passive unit that doesn’t include an interface, platform, or APIs for allowing advertisers to interact directly with their target audience in real-time like the Web does. We have seen some recent (and awesome) successes with new methods of delivery: Hulu, Netflix, MLB, and on-demand efforts. However we are in an infancy of thinking about the possibilities of satellite, cable, and their antiquated set-tops, and how broadcast can recapture some of the money that migrated to online advertising.

This is an opportunity for direct access to consumers in their living rooms. The delivery method is there, there is already a large internet-enabled box in many living rooms: tuner, DVR, AppleTV, Slingbox, etc. MSOs already have the access to provide value-add applications to the experience, but what is preventing them from radically changing the intersection of TV, Internet, and advertising? They actually have had a better chance than anyone.

Develop a platform:

Imagine tying content delivery to analytics and advertising – a platform that delivers both choice and guidance to direct viewers to shows they like, and then targets accompanying ads based on more granular information and feedback. Visible World does last-mile ad insertion for parent company, Comcast, who acquired one of the big three: NBC.  A small wrinkle in this still-disputed merger is Sen. Kohl, who is asking for a divestiture of NBC’s holdings in Hulu, arguing that it potentially violates anti-trust.

But this is an amazing opportunity for MSOs to radically change the model for consumers and advertisers by providing a platform for interaction. MSOs have a chance to drive people back to their subscriptions — otherwise we will see a switch to online viewing, where advertisers can get a lot more feedback for their digital dimes, and consumers can have more choice, and augment their experiences.

The TV could supersede the MSO-provided set-top box as a platform:

Internet-enabled TVs or TVs with companion internet-enabled boxes will supersede the current MSO interface and platform. Federated search across Internet/DVR/broadcast is far more useful and less clunky than a remote-driven interface.  In addition, the ability to bring in dynamic web content — ads, additional content, or related information — should quickly reduce the three screens to just two.

When MSOs realize they are missing out on key advertising opportunities, we will see a rush to market with subscription- and licensed-content across the three screens. You would have thought that Hulu would have put the fear in them, but wait until Google TV takes their bite.  Their recent partnerships with TV manufacturer, Sony, ensures that this will be widely distributed to Best Buy, Walmart, etc. beyond the market reach of Roku, Slingboxes, of the world.

Prepare to lose ad revenue:

NBC Universal’s (current) CEO Jeff Zucker and his oft-quoted ‘trading analog dollars for digital pennies’ was revised last year to ‘digital dimes’ from the man himself. That is good news, especially for the multitudes of investors that entered into internet video ventures last year. Out of that gold-rush of investment though, there hasn’t been significant pay-dirt for many. Clearly the answer lies in uniting broadcast to the Internet, bringing the interactivity the web provides, in addition to the type of analytics that are possible. I predict Google TV will make more of an impact than Apple TV did to unite those advertising schemes, although watch out, Apple’s clearly got some plans a brewin‘.

Google TV offers the opportunity to sell ad’s within the interface, and provide targeted advertising, on the TV. Show me what you got MSO’s.

In Conclusion:

It is a rapidly changing world, and people have clearly shown that the TV, cable-tuner, and DVR do not have all the features people want. The cable companies who own the infrastructure and delivery method of video and data are being left out of this equation. Google has side-stepped around cable companies and the licensing issues — wisely I might add — and added a layer with enhancements that will allow them get deep into people’s living rooms, to collect data and provide a smarter ad platform. Things are going to get really interesting….

Personally, if the only upgrade to After Effects CS5 was 64bit, I’d be overjoyed. There’s nothing more maddening than being unable to render a frame in your timeline because of memory issues, be it in a RAM Preview, output to disk, or working interactively in the application. The dreaded error I’m talking about of course is “After Effects error: could not create image buffer”. This error has the potential to disappear with this release, given your workstation is souped up with RAM.

Indeed, I think many After Effects artists do not realize what a massive overhaul this is, and how much easier it’s going to be to work interactively and render with the application. Those 32 gigs of RAM you have in your workstation actually mean something beyond having a million tabs open in Firefox while you wait for your render to finish, which is crawling because After Effects CS4 is limited to 4GB of RAM, and is using swap space to avoid spitting out an image buffer error. Yes, that’s right, your renders will be much faster at higher resolutions, even if you’re not being given an error. I think once artists start using CS5 they’ll realize how much more sanity they have when working on HD, 2K and 4K projects. You folks that are delivering for non-standard ultrawide displays, such as for stadiums, buildings and museums will be stunned at what a difference 32GB of RAM will make compared to the 4GB you were limited to. It’s funny, I heard one guy say that 64bit support should have been a .5 upgrade, and I was like “Are you out of your mind?! They had to rewrite the entire application!” Finally, After Effects artists will no longer be snickered at by their fellow Nuke and Inferno compositors that have already been working in 64bit for sometime. 32bit was great when our resolutions were NTSC 640×480, but those days are long gone.

The lack of a 64bit wrapper for 32bit plugins is a mixed bag, but ultimately, I’m really glad 32bit plugins won’t work. With paradigm shifts like this, I think it’s all or nothing. The adoption rate from plugin developers would be less than a trickle in the Sahara if they weren’t forced to rewrite their plugins for true 64bit. This would have been a great opportunity for Adobe to introduce some standardization for plugin licensing, but it looks like they dropped the bag on that already. Anyone who’s had to build or maintain an After Effects render farm knows the jungle of mess licensing is with 3rd party plugins and I wish Adobe would at least encourage best practices while developers rewrite their plugins.

Unfortunately, due to the plugin issue, I do think migration from CS4 to CS5 will take longer than it already does, which is typically most of a product cycle. I for one, will keep After Effects CS4 around until I find a replacement for Stefan Minning’s plugin called Normality. Unfortunately, the developer will not be updating it to 64bit.

On the After Effects scripting side, not much has changed, other than some deprecations and a few nice features. You can now read/write layer labels (the colored square next to the layer), which will be useful for making persistent selections for script batch operations. The upgrades to Mocha is nice too, especially the ability to import Mocha shapes in After Effects, which is pretty huge. That combined with the rotobrush finally makes After Effects an excellent choice for roto and tracking. I pretty much avoid rotoscoping whenever I can, but I think the rotobrush tool will be pretty useful, especially for making mattes that don’t need fine detail, such as for localized color corrections. I’m curious if you can specifically shoot footage for the rotobrush, when a green screen isn’t available, but I’m not sure how it’s tracking and edge detection works.

I’m very excited to see Premiere’s enhanced performance. I’m a huge fan of dynamic link and I’m always encouraging people to use it instead of Final Cut when prepping footage for After Effects. Premiere really excels where After Effects doesn’t, and that’s real time video playback. Premiere has been further accelerated with CUDA enabled Nvidia cards and if you’ve never tried dynamic link, or imported a Premiere project into After Effects before, this is the release to do so.

3D (stereoscopic) video does not always double the capture data for a couple of reasons, most notably because there are different capture methods, different post-production methods for processing the data, and different ways to project the data, all which can affect the data requirements at different points in the production, post-production, or delivery process.

Data requirements will be double at worst because there are (sometimes) two stereo video tracks, but since the cameras are capturing very similar information, one of the accepted standards for delivery is 2D+Delta compression for the MPEG format. Using this compression the image data is processed with an algorithm that removes redundant pixel data from the two video tracks. The amount of data is equal to one image plus whatever is different in the other image (the Delta). Using this compression, 3D video is usually roughly 40% larger than a single stream of HD video.

Some cameras, like Pace’s Fusion camera (above) which James Cameron used for Avatar, do shoot with two cameras side-by side, thus creating stereo video tracks and double the data for post (plus metadata). In this workflow, the two tracks are used to focus differently on objects in 3D space, dynamically changing the space between the cameras, and effectively using different z-depth focus to achieve the effect. This actually is not a new methodology, but using software algorithms to help determine focal length and automating what previously was manual can bring faster and more repeatable results. Another company, 3ality, is also providing similar functionality with different toolsets, including hardware that retrofits into 2D workflows. Interestingly enough, one of the major differences between them as 3D forerunners is that Pace is only a rentable technology solution, and 3ality cameras can be purchased. Two different positions.

When post is complete, there are some tricks that can be employed to reduce data for delivery including the 2D+Delta for transmission. For viewing the 3D data, there are a few different methods all which require different hardware (projectors, screens and potentially different methods of decoding).  Instead of using two projectors, some 3D can be projected using interleaved frames at a higher frequency through a polarized filter, i.e. one frame for the left eye, followed by one for the right. This is employed by RealD 3D and has by far the widest adoption in US theaters. Further, some theaters are equipped with active glasses technology, meaning that the glasses themselves ‘shutter’ either mechanically or using an LCD shutter linked wirelessly to the projection system for timing. I have heard these glasses are bulkier and heavier but better for certain situations.

3D workflow is particularly interesting to Control Group as an opportunity to help our clients design lean, optimized pipelines. This will impact our data stores, but why double data if it can be avoided? I think this is an opportunity for integrators like us to help productions, as well as the post-house, in evaluating these technologies and coming up with an appropriate blend of supporting technologies and workflows. Our goal is to build workflows that suit the business model and subject matter, as well as figure out how data is managed, archived, and stored. This is not the time to spend unnecessarily, but it is time to consider when and how this will affect our clients, and help them begin readiness preparations. We want to help our clients avoid turning down work because they haven’t put thought into these issues.

In conclusion, there are a variety of methods that a broadcaster or strategic consultant can invoke – some new, some old – to solve the technical issues. I think a lot of real challenges lie in the production of 3D content, such as the creative process of capturing a scene and the framing of the shots to take advantage of the magic of 3D. These are new processes and they involve re-educating producers, directors, cinematographers, camera operators, actors, and production designers.

The technology? No problem.

Everyone, everywhere, was talking about 3D: cameras, displays, production software, trucks, expertise, etc. The race for preparedness is on, and people will spend as pushy studios and producers want to be the first to do this or that with 3D. First sitcom, first sporting event, first documentary, first newscast, etc. The reality is that we are a ways off from wide-scale adoption, and showrooms and special screenings will be the place for 3D for the foreseeable future. We are in a similar catch-22 phase as a few years ago during the early days of HD production – 3D TVs are just coming onto the market and will drop down to reasonable consumer level prices within the next 2-3 yrs. Consumers are asking “do I buy a 3D TV when there is little to no programming?” while content creators are wondering “do I produce in 3D when there is little to no audience?”

Being an industry event though, it’s about more than bragging rights or audience – it’s about technical feasibility, practice, and logistics – all things which will change when 3D comes to our living room. DirecTV has announced that they will carry four 3D channels starting in June, including ESPN 3D and a dedicated 3D pay-per-view channel. And Cablevision dipped its toes into the 3D pool a few weeks ago with an MSG Network broadcast of a Rangers and Islanders hockey game live from Madison Square Garden.

After talking to a few industry experts who participated in the recent Masters in 3D, the challenges with 3D production are less technical than logistical: camera placement for example. While HD favors wide top-down shots for seeing all the action, 3D is most effective with close, ground-level cameras – imagine Phil Mickelson’s birdie putt as viewed from grass-level, just across the green, the ball rolling right toward you as he sinks it…

There were also lots of other things of interest: Falconstor’s HyperFS, CatDV asset management, Avid’s Java app for editing over the web, 3ality cameras, Adobe CS5, among many more. We were particularly excited by the potential of Active Storage’s Innerpool appliance for metadata. This PCI Express card contains on-board redundant solid-state drives, specifically engineered for storing metadata in an Xsan environment. This has the potential of being something of a game changer, allowing us to more efficiently configure the storage in our Xsan integrations, and giving our clients more bang for their buck when deploying new SAN solutions.

We had some great meetings with prospective clients, old and new friends, fellow consultants and vendors, and we’re excited about some emerging strategic opportunities. Our work in online video technology and web delivery combined with our broadcast infrastructure and workflow experience means we are ideally positioned to help organizations streamline and bring these workstreams closer together.

About a week and a half ago, Myriad Group announced an updated version of the Dalvik VM for Android, appropriately called the Dalvik Turbo VM.  Running the Dalvik Turbo VM, Android devices can experience a 2x – 3x performance boost in their applications.

What does this mean for Android users?

With any Android device, it’s pretty apparent that they are performing below what one would expect from their hardware specs.  This is because Android was designed to run on a wide range of devices, including devices with limited resources and processing power.

If your phone were to be updated with the new Dalvik Turbo VM, you wouldn’t experience nearly as much lag, applications would open faster and perform better, and you would actually see improved battery life (all things that as an Android user I would LOVE to see.)

How does it work?

The Dalvik VM is a virtual machine that runs in the background on all Android devices.  It acts like a middleman between the applications and the OS itself.  One of the things that makes Android so attractive to developers is that it’s apps are written in Java, the Dalvik VM then takes the Java code and converts it on the fly into code that Android can use.  The Dalvik Turbo VM is an enhanced version of the original VM that makes the conversion process a whole lot speedier.  The new Turbo VM is said to be 100% compatible with Google’s stock VM so any existing apps would be able to see the benefits from this as well.

A demo of the Dalvik Turbo VM vs Dalvik VM on a pair of Android Dev Phone 2′s.

How can I get it?!

Sadly that’s not such an easy thing to answer.  While Myriad hasn’t given an official answer, it is believed that the devices that would most benefit from the new Dalvik Turbo VM (T-Mobile G1 and MyTouch 3G) won’t be seeing it and it will only be included on new retail devices (which we should see before the end of the year.)  That said, many of the Android faithful (myself included) are hoping to see the new VM show up in the popular Cyanogenmod Android ROM amongst others in the near future.

If you’d like to see some more info on the Dalvik Turbo VM, check these links for some impressions and other demo videos:

Myriad Group Announcement

Myriad Group Youtube Channel

Engadget Hands-on at MWC

Everyone seems to be touting HTML5 video as the “open” alternative to the proprietary Flash plugin required for .flv playback in the browser. But how open is H.264, the codec that powers HTML5 video, and the current pick for encoding video for online delivery? Using H.264 as the codec behind HTML5 video sours things a bit for me. H.264 is encumbered by software patents; to develop or distribute a player or encoder for H.264 you might have to pay a licensing fee to MPEG-LA. Even though MPEG LA announced last week (PDF) that H.264 will remain fee-less for free internet video through 2016, this is not the same as being free or open. MPEG-LA can still go after people that produce the software to encode or decode H.264. And MPEG-LA is not just one organization, it’s a collection of patent holders that have their own agendas.

All this is a bit of a slap in the face to the open standards that power the web. Imagine if you had to pay a half million dollars to create or display JPEGs, GIFs, or HTML… The only people that would be able to afford to make software for the web would be huge companies. But what are our alternatives? Beyond Ogg Theora and Matroska, the pickings are slim. These codecs are open and free, but not necessarily better than H.264. Plus it would be next to impossible to compete with the marketing machine of Apple behind H.264.

Open and free standards have been what has made the Internet successful since its inception. I think it’s important that users understand this so that the Internet of the future cannot be controlled by corporations with enough cash to cover licensing fees.