Control Group Blog

I was recently having a conversation with a friend and we both laughed when we thought back to the first five hundred megabyte hard drives that we had owned. Back then, the half-gigabyte drive was ridiculously expensive and physically huge. We both thought that it would be impossible to fill these drives up.

This of course was not the case. Now you’re lucky if an application can be installed in less than 500 MB, and as hard disk sizes grow, we find new ways to fill them up with applications, documents, and media. Digital files have become the most valued assets for most of our customers, so the organization, storage, and archiving of data is a serious concern.

I find that the best way to evaluate storage technologies solutions for our clients is to step back and take a look at the problems the client is looking to solve and the priorities dictated by their business needs. Usually, our clients’ storage needs require a combination of performance, reliability, disaster recovery, scalability, and manageability. Fortunately technology has stepped up to the challenge to handle the increased need for larger, faster, and more reliable storage.

Storage networking is a general term that encompasses many different technologies that provide excellent solutions to modern storage problems. A storage area network (SAN) is an architecture in which storage devices are connected in a high-speed, dedicated network and are presented to computers that are part of the same network. Using storage networking, we can accommodate our clients’ performance and reliability needs: by abstracting groups of hard drives as logical units (LUNs) we can stripe data across disks to increase speed and add redundancy by storing parity on the disks. This configuration will allow us to rebuild the LUN when a disk fails, without causing downtime or data loss.

Example SAN Configuration for Final Cut Pro Editing

A storage network abstracts the underlying hardware that provides storage services, providing some great advantages for disaster recovery. When we add tape libraries to a SAN we can make backups quickly and efficiently without slowing down the network or computers on it. We can also connect a SAN to another SAN that’s in a different building or even a different state. This allows us to easily replicate data to a secondary location so our clients can be up and running quickly if there is some kind of catastrophe in the data center.

Even the largest SANs will eventually get filled up with data. What happens when it’s time to increase capacity? With traditional storage, the system is shut down, new equipment is installed, and the data is migrated. This typically involves downtime and runs the risk of data loss if something goes wrong. With a SAN expansion is no problem. Since the storage services are abstracted from the storage hardware it’s easy to add capacity or replace older equipment, in many cases involving no downtime.

A SAN also provides centralized management for storage: administrators can look in one place to see the status of all storage in a data center.  This allows businesses to evaluate storage health and utilization, which can prevent problems and help plan for future growth.

As data becomes a more and more important part of business strategy, it becomes critical for businesses to have larger, faster, and more reliable storage services to keep things operating smoothly. Storage networking is a core component of these strategies. I’ll continue posting about our thoughts and experiences with SAN solutions, and try to shed some light on the storage ecosystem as new technologies emerge.

Last week, I wrote about leveraging Final Cut Server as the core of a platform for asset management, approvals, and delivery. I wanted to append that post with some recent thinking and findings.

First, a quick discussion of semantics:

When you ingest a media file into Final Cut Server, it creates an asset. This asset is actually a collection of files, including the original media file, or primary representation. If the file being uploaded is a graphics or video file, Final Cut Server creates additional representations: a poster frame and a thumbnail. These are used to represent the asset’s media within Final Cut Server. If the file being uploaded is a video asset, a clip proxy representation is also generated, used for viewing the file within Final Cut Server.  This representation is created by transcoding the primary representation file to a lower-resolution codec.

Asset Representations in Final Cut Server

So a video asset is actually a container that’s made up of a bunch of files. This is pretty cool, and it’s mostly transparent to the end-user in Final Cut Server. However, while these representations are customizable (codecs, quality, etc), they all generated on ingest. Final Cut Server doesn’t currently support a way to create a new representation of an asset on-the-fly and have that representation become part of the asset container.

I’m struggling with this limitation as I explore Final Cut Server/Episode Engine integration. It would be ideal if I could setup Episode integration via a Final Cut Server copy response to an Episode watch folder, and have the resulting transcoded file copied back into Final Cut Server and made a representation of the original asset. Right now there’s no way to do this, so instead we are only able to re-ingest the new transcoded file as a new asset. And there is no relationship between this new asset and the original asset from which it was created.

Frustrating, but we’re thinking about ways around this. More to come as our tests and thinking solidifies. More info on Episode Engine/Final Cut Server integration in this pdf from Telestream’s website.

As an Enterprise Consultant at Control Group, I help lead our Broadcast, Media, and Entertainment technology consulting group. While CG employs experts in a wide variety of technologies, my area of experience and expertise is focused around Apple solutions for professional video and design. A few months ago, I wrote an article for O’Reilly on Final Cut Server that was targeted at consumers and covered the basics for professionals in the production industry. I wanted to expand on that article and share some thoughts on Final Cut Server as part of an integrated media workflow.

Final Cut Server client

Apple markets Final Cut Server as a tool primarily for Final Cut Pro users. Apple’s focus is on enabling users to manage their FCP projects and related files, providing them with automations to save time, and giving production teams a centralized system for collaboration. Beyond Apple’s sales pitch, we think Final Cut Server has real potential to play a central role in a production and distribution platform made up of a framework of connected systems.

Here’s an example of a typical broadcast infrastructure:

• Storage area network (SAN) for storage
• Media asset management system (MAM) for organizing and versioning
• Editor/artist workstations
• Producer workstations
• A transcode system for delivery

Now here’s an example of how those pieces might fit together in a broadcast workflow:

1. Content from tape or tapeless media is ingested through the MAM. Metadata is added at ingest – both technical metadata (shot, tape, take, etc) and possibly contextual metadata (actor, object brands, locations).
2. Data is saved to the SAN, where it is cut and crafted by editors and artists. As the content comes together, project files and new assets are saved to the SAN by editors and artists, and reviewed by producers.
3. As content is completed, it is transcoded for delivery to television, tape, and the web. Web distribution might include delivery in a custom player, and/or Hulu, YouTube, Vimeo.
4. Once content is in the wild, content owners need analytics tools to understand who is watching what and where.

Beyond its out-of-the box capabilities, Final Cut Server can be customized to play a key role in workflows like these. While it doesn’t include an API, developers can leverage custom responses in Final Cut Server to read and write XML and run external scripts. On its developer website, Apple provides an example of Final Cut Server integration with an external Rails application that enables the viewing and commenting of movie clips in a web browser. An example like this serves as a useful starting place when exploring building the middleware to connect Final Cut Server to other applications or platforms in a workflow.

Episode Engine Admin

For example, Final Cut Server leverages Compressor for all of its out-of-the-box transcoding. However many existing infrastructures exist using established transcoding systems, such as Telestream’s Episode Engine. By combining metadata subscriptions and watch folders in Final Cut Server with custom responses that leverage external scripts, Episode can easily be integrated as the transcode delivery component for a Final Cut Server workflow. Similarly, the ability to read and write XML to assets in Final Cut Server makes possible the development of web applications that interact with assets, and can even store information in their own disparate databases, populating Final Cut Server when appropriate.

What excites us about this are the many Final Cut Server integration possibilities that are not currently being talked about. And since Control Group marries Apple video integration expertise with a team of developers under the same roof, we’re excited to continue to innovate in this area. Give us a shout if you’d like to open up a dialog on how Final Cut Server might fit into your broadcast or production workflow.