The kneejerk reaction to a problem like this is to get a bigger database server. Sure, this may help right now when the data is only a few hundred gigabytes, but what happens when we are dealing with a few hundred terabytes? A few hundred petabytes? This kind of solution just does not scale.

The real answer here is to step back, examine the problem, understand what the goal is, and then design a process that can achieve that goal. In this case, the problem is that a business needs to be able to understand patterns and trends in a rapidly growing data set. The goal is to be able to do this quickly and consistently even as the data grows. One process that can achieve this is by using something like Hadoop or Spark to build a cluster that can scale as the data scales.

There were concerns as soon as I brought this up; What about the schema? How do you write SQL for that? Why not just shard the database? Some of these concerns may be valid, but I feel we must evaluate this without emotion. Do people want to use the relational database because it is a better solution for the problem or because they feel comfortable with it?

I’m not sure it’s accurate to say that we are facing new problems these days, but the shape and size of our problems have changed. Now even the smallest company has something to gain from working with big data– anyone with a credit card can spin up a compute cluster. We should not be afraid to change our tools as our challenges change.

Technology is continuously evolving. This means our tools are continuously changing and so must our processes for tackling new challenges. I believe that the system we came up with in that meeting will be the one to solve our client’s problem. If someone gave us the same problem five years ago or five years from now we would probably have wildly different suggestions, but we would come to those suggestions in the same way: through deep understanding of both the problem and the technology available.

Share this:

Large enterprises looking at cloud infrastructure bring their own requirements and challenges. We plan to write a series of blog posts about Control Group’s experiences with these types of clients and what we learned. Some of the posts will be about the projects and their politics, and some will be about technology approach. There are some interesting technology and organizational challenges that we will discuss, so stay tuned.

Share this:

This tool is called iozone. iozone is open source and runs on a ton of operating systems (including Windows). It runs several tests which can take some time to complete but provide the best overall view of the capabilities of a piece of storage. For instance, iozone runs a write test with files of different sizes and with different size records (the amount of data written at a time). It does this over and over again with writes, reads, random writes, random reads, and so forth. Since it’s running all these tests you can see what sorts of operations will have good performance and which ones will not perform so well. Check out the iozone documentation here.

One really great thing about iozone is that the output it generates can be easily placed in a spreadsheet program like Excel to generate a great 3d diagram describing your storage. Here’s a diagram I generated from some tests on a Linux server.

Results of a write test with iozone

This particular server performed quite well with large files and a record size around 1 MB (interesting to note, this is the same storage from the lmdd post. Notice that the parameters I tested with there are the same as the best write that this disk can do according to iozone!).

If you’ve been following my posts on storage performance testing I hope you’ve learned about some new tools that you can use to see what’s going on. I use these on every deployment to make sure we’re giving our clients solutions that they can depend for performance and reliability. As always, let me know if you have any questions about these tools. Happy testing!

Share this:

bonnie++ is a series of file system tests that focuses on small files. It was designed to behave like a mail server does, creating and dealing with lots of small files (emails). bonnie++ is easy to run and outputs a CSV file that you can view with something like Excel. With the bon_csv2html command you can quickly generate html pages from the CSVs.

Here’s the output from bonnie++ running on a server:

The HTML output of bonnie++ on a Linux Server

At first glance the output can seem quite cryptic, but if we look close we can see that this provides us a great amount of information about latency and speed on different filesystem operations. I generally run this several times as I make changes to verify that the storage is providing the right performance characteristics. Tweaking a file system to make file system operations happen a few milliseconds faster may seem ridiculous, but in some environments it can make a huge difference.

Next time I’ll post about a tool that’s new to me but can test a disk in so many different ways I’m planning to run it on every system we install from now on.

Share this: