Control Group Blog

Whew! As always, NAB was quick, productive, and overwhelming – only this time it was in 3D!

Everyone, everywhere, was talking about 3D: cameras, displays, production software, trucks, expertise, etc. The race for preparedness is on, and people will spend as pushy studios and producers want to be the first to do this or that with 3D. First sitcom, first sporting event, first documentary, first newscast, etc. The reality is that we are a ways off from wide-scale adoption, and showrooms and special screenings will be the place for 3D for the foreseeable future. We are in a similar catch-22 phase as a few years ago during the early days of HD production – 3D TVs are just coming onto the market and will drop down to reasonable consumer level prices within the next 2-3 yrs. Consumers are asking “do I buy a 3D TV when there is little to no programming?” while content creators are wondering “do I produce in 3D when there is little to no audience?”

Being an industry event though, it’s about more than bragging rights or audience – it’s about technical feasibility, practice, and logistics – all things which will change when 3D comes to our living room. DirecTV has announced that they will carry four 3D channels starting in June, including ESPN 3D and a dedicated 3D pay-per-view channel. And Cablevision dipped its toes into the 3D pool a few weeks ago with an MSG Network broadcast of a Rangers and Islanders hockey game live from Madison Square Garden.

After talking to a few industry experts who participated in the recent Masters in 3D, the challenges with 3D production are less technical than logistical: camera placement for example. While HD favors wide top-down shots for seeing all the action, 3D is most effective with close, ground-level cameras – imagine Phil Mickelson’s birdie putt as viewed from grass-level, just across the green, the ball rolling right toward you as he sinks it…

There were also lots of other things of interest: Falconstor’s HyperFS, CatDV asset management, Avid’s Java app for editing over the web, 3ality cameras, Adobe CS5, among many more. We were particularly excited by the potential of Active Storage’s Innerpool appliance for metadata. This PCI Express card contains on-board redundant solid-state drives, specifically engineered for storing metadata in an Xsan environment. This has the potential of being something of a game changer, allowing us to more efficiently configure the storage in our Xsan integrations, and giving our clients more bang for their buck when deploying new SAN solutions.

We had some great meetings with prospective clients, old and new friends, fellow consultants and vendors, and we’re excited about some emerging strategic opportunities. Our work in online video technology and web delivery combined with our broadcast infrastructure and workflow experience means we are ideally positioned to help organizations streamline and bring these workstreams closer together.

We have been doing some work with with Amazon’s Elastic Computing Cloud (EC2) which allows us to create virtual machines in the cloud in a few seconds. These are great for hosting websites, and what’s cool about them is that if you get Slashdotted or experience a similar unexpected spike in traffic you can create new hosts immediately. Recently Amazon added a new service called Elastic Load Balancing (ELB) which can distribute load across hosts. We’ve been looking at this for some of our recent development and infrastructure projects.

I just read this description of how ELB works by Shlomo Swidler from his Cloud Developer Tips blog. It’s a great reference.

You pay for ELB by usage just like everything else at AWS. From Amazon: “You are charged at $0.025 per hour for each Elastic Load Balancer, plus $0.008 per GB of data transferred through an Elastic Load Balancer.” For reference, on a deployment project in 2008 our Engineering team used a Cisco load balancer which I imagine cost a few thousand bucks.

Cost isn’t the only advantage. These can be created and destroyed quickly and remotely, allowing us to work more efficiently and spend less time visiting data centers in the middle of nowhere. This leads to improved quality of service for our clients as we can spend more time consulting on future technology growth plans and less time troubleshooting servers in cold, loud data centers.

This blog post brought to you by the iced coffee I am enjoying in the comfort and quiet of my office while deploying virtual machines!

As I’ve mentioned in previous posts about testing storage performance with lmdd and bonnie++, different applications require different characteristics from storage to provide the best performance. I’ve highlighted some tests that are good for large streaming files like video, and small file transactions like databases or mail servers. Today I want to look at a tool that runs a series of tests in many different ways to provide you with a holistic view of what the storage can and can’t do.

This tool is called iozone. iozone is open source and runs on a ton of operating systems (including Windows). It runs several tests which can take some time to complete but provide the best overall view of the capabilities of a piece of storage. For instance, iozone runs a write test with files of different sizes and with different size records (the amount of data written at a time). It does this over and over again with writes, reads, random writes, random reads, and so forth. Since it’s running all these tests you can see what sorts of operations will have good performance and which ones will not perform so well. Check out the iozone documentation here.

One really great thing about iozone is that the output it generates can be easily placed in a spreadsheet program like Excel to generate a great 3d diagram describing your storage. Here’s a diagram I generated from some tests on a Linux server.

Results of a write test with iozone

This particular server performed quite well with large files and a record size around 1 MB (interesting to note, this is the same storage from the lmdd post. Notice that the parameters I tested with there are the same as the best write that this disk can do according to iozone!).

If you’ve been following my posts on storage performance testing I hope you’ve learned about some new tools that you can use to see what’s going on. I use these on every deployment to make sure we’re giving our clients solutions that they can depend for performance and reliability. As always, let me know if you have any questions about these tools. Happy testing!

Were getting ready for an event with Google and Mozy that we have dubbed “CloudSourcing”, taking a note from Gartner and tweaking it a little.

Tom Mills from Google and Sean Finnegan from Mozy will be giving an in-depth review of their offerings and how they fit into an agile, post-recession office technology strategy.

I’ll be giving a brief overview of how I think we arrived at this point in IT and what it means for creative, innovative firms that are trying to do more with less.

In an effort to get my thoughts together and get some feedback, I’m using this blog post as a draft for the event.

Let me start off by giving a brief overview of our services, and then a little history about the evolution of our offerings:

We provide a number of technical services for our clients in the areas of infrastructure, application development, and industry-focused workflow consulting. As this is New York, we work with a number of creative firms; media, architecture, publishing, and design companies, as well as some key clients in the financial sector. We strive for long-term relationships with our clients, many of whom we’ve worked with for close to a decade. We have installed and managed hundreds of servers, network devices and application suites, but more recently we’ve been focusing on helping our clients select, migrate to, integrate, and manage Cloud-based services.

Since the 1990s and the introduction of pervasive bandwidth, we’ve gone through a number of permutations of the remote server/client model, and much has been written about the benefits and the irony of the shift back to the mainframe/thin client structure of the 1960s. Now everyone is talking about the future of ‘The Cloud‘; a vast array of computing resources, abstracted and presented as a single source to the consumer.

At the turn of the century, we found most small to mid-sized businesses with a pure Local Area Network (LAN), typically comprised of in-house mail – most likely Exchange – and a few other local services: file, print, etc.  A lot of these firms had an internal IT staff or a dedicated consultant to manage their servers, tape backup, networks, and desktops. Only a few were pushing the envelope by leveraging Application Service Providers (ASPs) to deliver back office services.

The risks with this situation were obvious. These systems mostly depended on a single Internet connection, a single building, and a single individual, prone to career changes and untimely vacations.  Remote access to these in-house services was expensive to do right and applications rarely worked as well remotely as they did in the office.

Over the next five years, we saw a gradual shift towards ‘Hosted Applications’. This typically came in the form of a service provider taking a LAN-based solution like Exchange or SharePoint out of the office and putting it in a data center. In conjunction with this change, we saw the IT services industry begin to shift its focus from in-house IT, or consultants, to managed services – companies providing regular systems management remotely.

There were some benefits to this offering: critical applications were not dependent on intermittent Internet connections or over-heated server rooms. Flaky consultants were traded for predictable management services and cost became as regular as the electric bill.

But there were still problems. We had the same old model of doing things, only it was moved out of the business’s office and into the provider’s.  Services that were built for an onsite installation and LAN speeds were shifted to a remote location – not always producing the best results. Access to applications designed for the LAN was sometimes unacceptable because of bandwidth and latency. In a similarly narrow view of the problem, Managed Service companies focused on monitoring systems and patching software, maintaining the status quo, without looking at the big picture, or driving the business forward.

Now the next generation of IT services is coming along and delivering on the promise of on-demand, scalable solutions. These services are web-native, built for the Cloud and multi-tenant environments.

As services like Google Apps and Mozy were built for the web – not re-purposed LAN applications – they deliver exceptional performance and remain very flexible. Control Group has designed our support and project services in a similar way. Our services are built to function efficiently remotely – scaling up when our clients need it, and going away when they don’t – and also to be flexible and innovative, driving business forward rather than maintaining the status quo.

Using the cloud paradigm, we act as a single source of technology for our clients. We help them run more efficient, profitable businesses by weaving an ever growing selection of web-based services, traditional IT, and industry expertise together, to provide an flexible, competitive business platform.

Last time I posted about checking disk performance with lmdd. lmdd is great for checking streaming throughput, but what if you have a different kind of application? Every application accesses storage in different ways: with video we need to be able to provide constant throughput when writing a lot of data to the disk, but other applications may have different storage needs. For example, a database can make lots of very small changes to the data on disk in a short period of time. The best performing disk for a database will probably need to have very low seek time and good transactional performance.

bonnie++ is a series of file system tests that focuses on small files. It was designed to behave like a mail server does, creating and dealing with lots of small files (emails). bonnie++ is easy to run and outputs a CSV file that you can view with something like Excel. With the bon_csv2html command you can quickly generate html pages from the CSVs.

Here’s the output from bonnie++ running on a server:

The HTML output of bonnie++ on a Linux Server

At first glance the output can seem quite cryptic, but if we look close we can see that this provides us a great amount of information about latency and speed on different filesystem operations. I generally run this several times as I make changes to verify that the storage is providing the right performance characteristics. Tweaking a file system to make file system operations happen a few milliseconds faster may seem ridiculous, but in some environments it can make a huge difference.

Next time I’ll post about a tool that’s new to me but can test a disk in so many different ways I’m planning to run it on every system we install from now on.

One of the unique things about how Control Group works is that our focus is much more involved than simply putting in a solution for a client and then moving on. We work with our clients to determine how they work, so we can design IT solutions that really fit their needs. Since we have partnerships with a variety of vendors, we work with our clients to arrive at the best solutions for their business. This means we do quite a bit of research and planning before we begin a project — and then a great deal of testing during and after we install new hardware or software.

I do some work on implementing storage systems for our clients, and we’ve found that different applications have different storage requirements. For example a video post production facility — like the facility at WWE — generally needs lots of disk space that is very good at reading and writing large files at high speeds. The storage here needs to provide good streaming throughput, because high quality video files generally have high bit rates, and are being stored or played back from the disk in real-time for ingesting, editing, or playout. If the storage system is not fast enough to read or write the file in real-time, frames will be dropped. This can cause unsatisfactory media files, programs to crash, or audio and video to become out of sync.

10,000 RPM SAS disks. That's some serious storage.

Suboptimal read/write performance can become a huge problem. When we put in a new system this is something we need to test. I usually do the test with a tool called lmdd.

lmdd comes from the lmbench tools which are provided by Bitmover for benchmarking systems. lmdd is great for testing streaming bandwidth. In most of our engagements with video, we install a Stornext or Xsan filesystem so we’ll run our tests against this. lmdd will probably work on any filesystem that you can mount on your Mac or Linux computer (Leave a comment if you need a version for Mac OS X, I have one compiled).  lmdd lets us verify exactly what the maximum number of megabytes per second we can push through the storage and point us to where we need to make changes to the hardware or software configuration. I use lmdd like this :

lmdd of=/path/to/test_file count=1g

lmdd if=/path/to/test_file

The first tests write performance and the second tests read performance. More information about the syntax is available in the manual page for lmdd. The results of the command from a server I was testing looked like this:

2147.4755 MB in 6.8003 secs, 315.7914 MB/sec

lmdd is great because it’s easy to read. This result shows I could write to the filesystem at 315 megabytes per second. That’s really fast! This is from a test with a server with a lot of RAM and a special filesystem that took advantage of that cache. When I run it on my Macbook, I get a result like this:

18342.6171 MB in 376.7685 secs, 48.6841 MB/sec

So the next time you’re interested in how your storage is performing give lmdd a shot and let me know how it goes. If you’re looking for more information about storage performance testing then stay tuned; I’ll be posting about testing storage with tools that benchmark small reads and writes next.

Last week, I shared some thoughts on my ongoing trial of Google Apps in place of Outlook and Exchange. I wrote about some of the killer features in that post… however, where Google Apps succeeds in its simplicity, it can fail in terms of flexibility. There are some things here that could mean game over for a lot of people:

Conversation view — how about an option to turn it off?

Conversation view, conversation view, conversation view!

It’s terrible. I have finally gotten used to it, and I still think it’s terrible. If only there were an option to shut it off. If you don’t know what it is, it’s a feature that groups email of a thread together. But it isn’t perfect and it can be pretty awkward. Emails tend to get jumbled up, and sometimes mixed into the wrong thread. Someone high up at Google must have came up with this one because it is one of the most complained about features and still they wont give you a way to shut it off.

Mobile device integration is really weak.

Being a Blackberry Enterprise user, the move to IMAP is a big downgrade. Sent mail is important! Email in under 2 seconds is hard to give up; with IMAP, be prepared for a full minute, unless you manically hit refresh. Google offers their mobile mail client, but it leaves a lot to be desired, it gives you labels (aka folders) and sent mail, but it’s clumsy and lacks basic things like original email text in the body of replies and copy/paste, to name two biggies.

Google plans to release Blackberry Enterprise Server integration this summer, but my hopes aren’t too high. Since one of my goals is to live in the cloud, having a BES server at our office doesn’t fit into that fantasy. Plus, calendar sync is one-way, and email sync is “under 1 minute,” but — I have to say it again — BES and Exchange give me email in under 2 seconds!

Return on Investment

I am willing to overlook these inconveniences, and many others because the ROI from an administrative/business owner perspective is really that good. Take Instant Messaging as an example. If you wanted to implement a company-wide IM platform with Microsoft, prepare to drop $5-$7k on hardware, another $3k+ on software, and about the same on installation. Then add in maintenance, training, and once (if) it gets adopted and people can’t live without it, get ready to plan on backup, archiving and a data recovery plan. We are talking at least $20k to do it right.

With Google Apps, you want company-wide IM? Check a box. You want all IM messages saved and searchable? Click another box. Cost? $0. You want video and voice chat too? Done. Gone are the days of patching servers, mailbox limits, backups running during the day, defragging information stores, Google Apps’ greatest strength is in the fact that it’s not there. It’s everything a cloud application should be.

“In Google I Trust”

One of the biggest drivers in my support of the Google platform is my trust in Google to quietly innovate and release new features and updates. I trust they will get mobile device synchronization right soon. Maybe Microsoft will cave and license them the rest of Active Sync. But  I still have mixed feelings about Google Apps. Life in the cloud is the future — if I was starting a new business, there’s no doubt I would go with Google Apps.  Coming from a Company with 10 years of Exchange process and history, it’s a harder decision. But I still might choose the new pain over the old.

Apple’s venerable Apple Software Restore (asr) tool includes the insanely useful ability to image a nearly unlimited number of network clients. It accomplishes this via a router’s ability to broadcast data to any number of clients simultaneously from a single IP address. Known as multicasting, this allows even a modest computer to image a hundred Macs with 35GB images in a single fell swoop.

An ASR Restore Image in Disk Utility

The disk images asr works with are the same format used by Mac OS X’s Disk Utility. This means you can do a rollout over the network and keep the master file on hand in your re-imaging kit, should one of your workstations run into trouble and need to be re-imaged over FireWire. To ensure the sanctity of the final result, disk images include an embedded checksum which is automatically verified during the deployment process. This can be a significant advantage in using asr over of Apple NetInstall, which requires its own folder-based setup of restore source files.

Disk images are also hardware agnostic for the most part. You can build your image on a Mac Mini and apply it to anything from a PowerMac G5, to an Macbook Pro so long as it can get on the same subnet as the asr host.

The usual caveats of disk imaging apply unfortunately. You’re going to have to sweat individual serial numbers if you don’t employ network or volume licensing. Settings like hostnames and non-ubiquitous local users will require individual workstation visits without centralized management. However, having asr around to do the heavy lifting means you may be able to turn a strenuous two day deployment into a breezy one day affair.

Is your Inbox full of messages from MAILER-DAEMON? We frequently help our clients deal with spam and junk-filled Inboxes — here’s some info about why this can happen, and what can be done to help prevent it.

Inboxes full of messages from MAILER-DAEMON are frequently the result of non-delivery reports (NDRs) from spoofed spam messages.  A spoofed spam message is an email from a spam mailer that has been masked with your valid sender email address.  When these spam messages are sent to addresses that don’t exist, an NDR is generated and sent back — this is the email equivalent of the post office returning a letter as undeliverable.  In theory, one is sent to the invalid sender address, the other is sent to you.  Since the invalid one doesn’t actually exist, you are the only one to receive it.

Lots of messages from Mailer Daemon or Mail Delivery Systems can be symptoms of Spoofed Spam

Here are some frequently asked questions about Spoofing:

Has my email been hacked? Probably not — 99% of the time your account has not been compromised.

How did the spammer get my email address? Email addresses can be harvested in a number of ways.  The most common are as follows:

1. If your email is posted on a website, spammers use “bots” to crawl through websites searching for email addresses.
2. When you sign up for access to a web service, some sites will sell your email address and personal information.
3. A virus on your machine or on someone’s machine that has your email address on it either as a contact or even just an email to/from you.

What can be done to stop them? Once a spammer has your email address, there’s not too much that can be done to stop them from spoofing their emails with your address.  The messages themselves are not being sent by the spammer but by mail servers doing their job and alerting the sender that the recipient doesn’t actually exist.

How long is this going to happen? Generally spammers use a group of email addresses for about a week and then move on to the next batch that they have harvested.  You will likely still receive them for a few days up to a couple of weeks.

Can’t you block them at the server or through Postini?  Technically it is possible but this can be a double-edged sword.  Blocking messages from MAILER-DAEMON will result in actual NDRs being blocked that you may want to receive.  If you were to send a message that didn’t reach its intended recipient for any number of reasons, you wouldn’t know about it.  Another reason is that not all mail servers will send their NDRs under the name MAILER-DAEMON.

Here are some steps you can take to help prevent this from happening again in the future:

Schedule regular virus and malware scans. By scanning your system regularly, you will help to catch viruses and malware that will harvest email addresses from your computer.  This has the added benefit of making sure that your system is virus/malware free to prevent other issues such as system performance issues and identity theft.

Be careful of where you use your email address: If you need to have your email address posted on a website, use a non-standard format such as John.Smith [at] ControlGroup [dot] com.  This will help to confuse the “bots” that are used to scour websites for addresses.  If you need to register to access a website, it helps to have a separate email account setup for this through a free service such as Gmail.com  By keeping this separate email address simply for signing up for websites, any spam you may receive by signing up will be sent to this email address instead.

Even with taking these steps and being diligent with them, there is always the possibility of someone with whom you have emailed with getting a virus that can collect your email address.  Unless you are prepared to be tech support for everyone that you email with, your best bet is to help educate them on the issue to help save both you and them the headache of being Spoofed.  You can even link them to this blog post to help them learn how they can prevent this from happening to them.