Rick Ralston of Real Metrics dropped me a line the other day:

We’re thinking of adding a new category to our RealMetrics website for Blogging software. It would be similar to our shared web hosting category at http://www.realmetrics.com/a/shared-hosting. Any thoughts or ideas would be appreciated.

Interesting question. It’s a lot easier to compare the performance of 2 standard web hosts, since you can create a reference page, upload it to each host, and sample the performance of identical pages at different hosts. But for managed blogging, such as BlogHarbor or Typepad or Blogger or WordPress.com or Squarespace, it’s simply not possible to create identical pages. Measuring the performance of an image download or a file download doesn’t measure the true performance of the system. Managed blogging platforms typically generate pages out of a database, so while it’s not possible to create identical pages it’s important to create similar pages on each platform so you can make a valid comparison about the performance of each platform.

Here’s my reply to Rick:

Here’s what I’ve come up with… Most of the things you are measuring with shared hosting apply to blog hosting… The only thing that’s going to be an issue is measuring performance. Why? In measuring standard web hosting, you can measure apples to apples because you are able to upload a standard page or site containing the same number and size of objects to each host. So you are able to compare the retrieval of -identical- 100K of html page and 250K of image downloads for example…

With blog hosting, at least managed blog hosting, you won’t be able to upload a standard page to measure. Well you could, but you really want to measure the speed of let’s say the home page of a blog rather than some static files. Since blogs are typically database driven applications, the real key here is that although you won’t be able to measure oranges to oranges since each blog application will generate a slightly different page, you still want to at least measure oranges to tangerines (OK, I have stretched this analogy way too thin!)…

By that I mean that you’ll want to make sure that each blog contains a set number of identical posts. Create 10 posts of varying length, some with images and some without, and post them to each blog platform. Use the default template supplied with the blogging platform. Take a look at 4 or 5 platforms and see what the page weight comes out to be for a blog containing 5 identical posts and whatever is on the default template. I suspect they will all differ by less than 10%, and if not then I think it should not be of much of a concern but the page weight should be listed in your results so your visitors can take that into consideration…

Anyone out there have any thoughts for metrics to measure performance of managed blogging services?