A FeedMesh Primer

I attended a lively discussion on Friday night 6/24/2005 at Gnomedex on the subject of the FeedMesh. There were some questions and concerns raised at the get-together, as well as the feeling that there were other parties interested in the concept besides the developers of blog search engines and aggregators who might wish to know a little bit more about this project, so I decided to write a little piece about what I know about FeedMesh with the hope that anyone who can provide additional information or clarification will do so by commenting or trackbacking. I hope that this article provides a little background for both developers and content creators and I am very much looking forward to hearing from anyone who can clarify any of the points in this article. Perhaps some of the FeedMesh members who are more active with the technical details will post on their blogs and send a trackback if they can provide further details about the workings of the FeedMesh.

To be clear, I am not sure if you would say that I am a member (though I did get a nice t-shirt – thanks Bob!) or our company is a member of FeedMesh; though we do contribute pings by the nature of being a blogging service and also consume them for business processes. Better to describe us as supporters of the idea and volunteers to help facilitate its success as we feel that the idea behind FeedMesh has the potential to be of great benefit to the entire blogging community.

Introduction

Bloggers inform the various blog search and aggregation services that their blog has been updated by sending these services a ping, a simple form of notification that says, essentially and quite simply, Hey! I just updated my blog. Check it out! Once a service has received a ping, it can access the blog and index any updated content it discovers.

The process of pinging alleviates the need for blog search engines to regularly visit blogs to discover changes; Pings allow the blogger to notify the search engines when changes have occurred. This helps search engines be more efficient, as they can distribute their resources to index blogs in a more timely fashion.

It didn’t always work this way. Back in the beginning of blogging history, Dave Winer’s Weblogs.Com began compiling a publicly accessible list of updated weblogs. He allowed bloggers to register their weblog, and the Weblogs.Com server would periodically check the blogs which had registered for updates to their RSS feeds.

As the number of blogs became too numerous for this approach to continue, a new design was needed. Dave defined the concept of a ping in September 2001 and on October 23, 2001 redesigned Weblogs.Com to use pings in order to learn which weblogs had been updated. So it was now the blog/blogger’s responsibility to send notification via pings that it had been updated rather than wait for a service like Weblogs.Com to visit them for updates.

Changes.xml

In addition to receiving ping notifications that blogs had been updated and publishing them on the Weblogs.Com site, Dave Winer also created an XML specification and published a list of the changes he had received in XML format; that file is still being published to this day at http://www.weblogs.com/changes.xml. The file is updated at regular intervals and lists recently updated weblogs and the time that their ping was received

Another service called blo.gs was created by Jim Winstead in 2002 as a directory of recently updated weblogs. This service was also pingable, and thousands of blogs began notifying this service when they updated. Blo.gs also began publishing a changes.xml file listing blogs which had been updated.

Many services began using the data in these changes.xml files to determine which weblogs to add to their search indexes and aggregation services, and other services such as Blogrolling.com also started publishing changes.xml files. The distribution of update notifications via changes.xml files provided the initial data for companies like Feedster and Technorati to create their services.

The Cloud interface

Blo.gs later created an alternative to the changes.xml format which allowed them to pass along real-time notifications of the pings they had received. Blo.gs called this their cloud interface and published some sample code for receiving that data in November 2004. At the same time, they announced that they would no longer publish updates via their changes.xml file; updates would only be sent through the cloud interface in 2005.

PubSub also began deploying the cloud interface in December 2004 and released open source software for both reception from and transmission into the cloud interface. In January 2005 they began working with blo.gs to share information about blog updates.

PubSub currently maintains a publically accessible streaming list of notifications via the cloud interface. Blo.gs’ cloud interface was publically accessible until June 14, 2005 when it was officially announced that the service was purchased by Yahoo; access to their cloud interface now requires submission of a form requesting the ability to access this stream.

Enter Feedmesh

Unfortunately, as more services are created which can benefit by ping notification, and since bloggers also welcome these new services for the benefits they provide, it becomes increasingly unrealistic to expect that individual blogs will ping tens or hundreds of services as they create new content. What is needed is a way for new and existing services to share with each other the update notifications they receive.

The concept of the Feedmesh apparently originated at an event called Foo Camp in September 2004. Sam Ruby was a participant, and he called the FeedMesh a “peering network” for decentralized web(site|log) update notifications and content distribution. The initial description for such a service posted by Sam called for a single URI for people to ping (feedmesh.org?) which would be backed by the members of the “peering network” via round robin DNS.

Current discussions seem to center around the concept of peering arrangements, where services who become FeedMesh members share the pings they receive with other members. This would allow bloggers to ping one service, and be assured that their ping would be passed on to all the FeedMesh members.

Ping-o-matic debuted on April 20, 2004 as a way to solve these same problems that the FeedMesh tries to solve, but to solve them from the users perspective rather than the aggregator’s. Ping-o-matic could be described as a “repinger”, as they will receive a ping notification from an end user and use that notification to ping dozens of other services on that user’s behalf.

So what is the FeedMesh?

I don’t believe that question has been fully answered yet. That seems to be what is being determined right now on the FeedMesh discussion list.

It may be safe to say however, that services which publish data via changes.xml or more likely the cloud interface going forward exhibit the characteristics of FeedMesh services. Whenever blog search or aggregator services share updates, it seems to me that is a FeedMesh…

  • Is Ping-o-matic a member of FeedMesh? I won’t speak for them, but it does appear they are sharing update notifications, and are in fact dedicated to doing so.
  • Is Weblogs.com a FeedMesh member? I doubt that Dave Winer would say that it is, but it does receive update notifications and pass them along for any service or individual who wants them… It walks like a duck…
  • Is PubSub a FeedMesh member? I would have to say that by words and actions they are; they receive pings and make them available to anyone who wants them.

I choose these services only as an example, not to imply anything at all about who might or might not in fact be a FeedMesh member.

It appears that the goal of FeedMesh is to provide a more efficient method for users and developers to distribute ping notifications. Ping-o-matic on the user side and the cloud interface on the developer side look to be the first steps at creating a more efficient ping process.

Perhaps the next steps will be to more concretely define what it means for developers to participate in the FeedMesh, and subsequently to make users aware of the benefits of pinging FeedMesh member services.

Where to learn more about the FeedMesh?

The group discusses the future of FeedMesh via a Yahoo discussion group, and one member has created a Wiki.

Some posts arising from the FeedMesh meeting at Gnomedex on 6/24/2005 include: