HTTP Conditional Get for RSS Hackers

October 21, 2002 2:23 PM

Given the massive confusion exhibited here, I've written a nice, simple guide on how to implement HTTP's Conditional GET mechanism, with regards to producers and consumers of RSS feeds.

This article presumes you are familiar with the mechanics of an HTTP query, and understand the layout of request, response, header and body.

What is a conditional get?

My full-length RSS feed is about 24,000 bytes long. It probably gets updated on average twice a day, but given the current tools, people still download the whole thing every hour to see if it's changed yet. This is obviously a waste of bandwidth. What they really should do, is first ask whether it's changed or not, and only download it if it has.

The people who invented HTTP came up with something even better. HTTP allows you to say to a server in a single query: “If this document has changed since I last looked at it, give me the new version. If it hasn't just tell me it hasn't changed and give me nothing.” This mechanism is called “Conditional GET”, and it would reduce 90% of those significant 24,000 byte queries into really trivial 200 byte queries.

Client implementation

The mechanism for performing a conditional get has changed slightly between HTTP versions 1.0 and 1.1. Like many things that changed between 1.0 and 1.1, you really have to do both to make sure you're satisfying everybody.

When you receive the RSS file from the webserver, check the response header for two fields: Last-Modified and ETag. You don't have to care what is in these headers, you just have to store them somewhere with the RSS file.

Next time you request the RSS file, include two headers in your request.. Your If-Modified-Since header should contain the value you snagged from the Last-Modified header earlier. The If-None-Match header should contain the value you snagged from the ETag header.

If the RSS file has changed since you last requested it, the server will send you back the new RSS file in the perfectly normal way. However, if the RSS file has not changed, the server will respond with a ‘304’ response code (instead of the usual 200), where 304 means ‘Not Modified’. In the case of a 304, the response will have an empty body and the RSS file won't be sent back to you at all.

There's a temptation for clients to put their own date in the If-Modified-Since header, instead of just copying the one the server sent. This is a bad thing, what you should be sending back is exactly the same date the server sent you when you received the file. There's two reasons for this. Firstly, your computer's clock is unlikely to be exactly synchronised with the webserver, so the server could still send you files by mistake. Secondly, if the server programmer has followed this guide (see below), it'll only work if you send back exactly what you received.

Server Implementation for Static Files

If you are using one of those weblogging tools that just sticks regular files on a regular webserver (e.g. or Moveable Type), your webserver will almost certainly already follow the get standard. HTTP 1.1 has been around 31 years now, and there's really not much of an excuse for anyone to not be following it.

One thing you'll have to watch out for, though, is if your site's RSS file is regenerated frequently even when it's not changed. If that happens, the server won't be able to keep track of the last modified time properly, and you'll get people downloading the file even when it's not changed. The solution is for the writers of weblogging tools to optimise their software to make sure that files are only updated if they've actually changed in some way. (i.e. have them generate the new file, compare it with the old one, and if they're the same leave the old one untouched.)

Server Implementation for Dynamic Content

If you've got a weblogging tool that re-generates the RSS file every time a request is made, there's a little more work to do. This section is aimed more at the writers of the tools than at the user, because it's the tool writers that need to fix their software so that it follows the specs.

I'll concentrate purely on RSS files, but the concepts used here can be applied to any page in the weblog, and may further reduce the bandwidth usage for your users.

In your RSS feed generator, you'll have to keep track of two values: the time the file was last modified (converted to Greenwich Mean Time), and an “etag”. According to RFC2616, the etag is an “opaque value”, which means you can put anything you like in it, providing you stick double-quotes around the whole lot. The time in the Last-Modified header needs to be formatted in a certain way, though, the same format used in email headers. For example, ‘Mon, 17 Sep 2001 11:54:29 GMT’.

Whenever someone requests your RSS file, send those values for the Last-Modified and Etag headers. Every web scripting language allows you to add and remove headers like that at will, just check the manual if you don't know how.

Now for the other bit. Whenever someone requests your RSS file, check the headers of their request for an If-Modified-Since header, or an If-None-Match header. If either of them are there, and if [deleted either ] both of them match the values you were planning to send out with the file, then don't send the file. Once again, consult your manual to see how to send back a "304 Not Modified" reply instead of the "200 OK" that you normally would. If you send back the 304 reply, you don't have to generate the RSS file at all. Just send out the headers, followed by two linefeeds to show the headers are done, and the client will know there's nothing else coming.

Technically, what you should do with an If-Modified-Since header is convert it to a date, and compare it with your stored date. However, 90% of the time you can get away with just doing a straight match, so it's probably not worth the effort.

How do I calculate the Last-Modified date?

Easy. It's the time that the most-recently-changed item in the RSS file was modified. Something like that should be pretty easy to store and fetch.

What should I put in an etag?

The Apache server uses a hash of the contents of the file. This isn't necessary though. All the eTag has to be is something that changes every time the file changes. So it could be a version number, or it could even be exactly the same as the Last-Modified date, just in double-quotes.

2002-11-11 Update: A number of people have written to me to remind me of HTTP's Gzip Content-encoding (compressing the files during transfer). This is a little beyond the scope of this essay. The worst thing you can do when suggesting a solution to a problem is to provide alternatives, people end up arguing the alternatives instead of implementing the fix.

1 in the original version of this document, this read ‘13 years&rsquo, because the author can not count. Mea culpa.

41 TrackBacks

Listed below are links to blogs that reference this entry: HTTP Conditional Get for RSS Hackers.

TrackBack URL for this entry: http://fishbowl.pastiche.org/mt-tb.cgi/47

TITLE: If-Modified-Since: whenever URL: http://markpasc.org/blog/2002/10/21.html#i004828 IP: 63.189.209.129 BLOG NAME: DATE: 10/21/2002 04:57:06 PM TITLE: If-Modified-Since: whenever URL: http://markpasc.org/blog/2002/10/21.html#i004828 IP: 63.189.209.129 BLOG NAME: DATE: 10/21/2002 04:57:06 PM Read More

Mark Pilgrim's problem with bandwidth is related to the fact that content aggregators are pulling his data too frequently. I'm not quite sure why he titled his entry "Push".... No reason for the aggregator to pull. So why is there pull here? Because... Read More

The Fishbowl: HTTP Conditional Get for RSS Hackers "Given the massive confusion exhibited here, I've written a nice, simple guide Read More

Brent added support for Etags and If-Modified-Since headers to the latest NetNewsWire beta. It's very cool. He added it after hints and pressure from among others Joel, Phil, Sam and Mark; I'll refrain from pointing out that I suggested it to him sever... Read More

So I've got this nifty little RSS parser doohickey, Magpie. In the name of lowering the curve, and weaning Read More

I'm looking for an RSS aggregator. I've been using HotSheet recently, until I discovered it can't handle Dive Into Mark's Read More

MagpieRSS is the PHP RSS parser I wrote, because I was unhappy with all the existing solutions.(which you already Read More

Hacking from Vertical Hold on November 21, 2002 12:15 PM

I hacked HTTP 1.1 Conditional Get support into phpicalendar today. Thank Zeus for NetNewsWire Lite's Bandwidth Statistics view. It made Read More

conditional HTTP get for RSS from anil dash's daily links on January 21, 2003 2:04 PM

http://fishbowl.pastiche.org/archives/001132.html Read More

This is a good summary of how to implementHTTP's conditional GET mechanism for RSS which I touched on here Read More

MagpieRSS is the PHP RSS parser I wrote, because I was unhappy with all the existing solutions.(which you already Read More

If you're going to download a resource over HTTP from a URL more than once, there are a couple of features of HTTP you should make sure you're using. By giving the server some metadata about what you saw when you last downloaded the resource, it can gi... Read More

If you're going to download a resource over HTTP from a URL more than once, there are a couple of features of HTTP you should make sure you're using. By giving the server some metadata about what you saw when you last downloaded the resource, it can gi... Read More

Includes code using Jakarta Commons HttpClient . hackdiary: Using HTTP conditional GET in java for efficient polling seeAlso: the fishbowl Read More

TITLE: http://google.com URL: http://google.com IP: 68.35.86.96 BLOG NAME: DATE: 04/12/2003 10:06:46 AM TITLE: http://google.com URL: http://google.com IP: 68.35.86.96 BLOG NAME: DATE: 04/12/2003 10:06:46 AM Read More

Bandwidth a problem? Try gzip compression. Read More

Bandwidth-saving tip of the day [dive into mark] How to save bandwidth fetching RSS... Conditional GET... Read More

Among the projects I currently follow, there is CleverCactus. It seems that new features are added fast. rss performance improvementsNew cactus beta2 feature of the day: RSS-download performance improvements! Aside from tweaking the parsing a bit, I wa... Read More

Jason Diamond did an incredible job adding ETag and Last-Modified support to my ultra-liberal RSS parser. This allows the parser to avoid redownloading feeds that haven't changed. This is a topic of much debate at the moment. I wish I could say I pl... Read More

Articles like this make my life simple.... Read More

RSS/ATOM Jeremy Allaire: RSS-Data: A Proposed Format and More discussion on RSS/XML-Data decafbad: RSS-Data: XML-RPC encoding in RSS 2.0... Read More

RSS/ATOM Jeremy Allaire: RSS-Data: A Proposed Format and More discussion on RSS/XML-Data decafbad: RSS-Data: XML-RPC encoding in RSS 2.0... Read More

RSS/ATOM Jeremy Allaire: RSS-Data: A Proposed Format and More discussion on RSS/XML-Data decafbad: RSS-Data: XML-RPC encoding in RSS 2.0... Read More

Just can't get away from it. Read More

Conditional GET Plugin from Confluence: blojsom on April 22, 2004 5:19 AM

DescriptionThe Conditional GET plugin reduces the bandwidth you send out from your blog for the flavors for which it is enabled. It works by examining various HTTP headers which indicate when the requesting client last polled your blog.... Read More

Full-feed RSS and bandwidth from JayAllen - The Daily Journey on May 4, 2004 11:20 PM

Eric Meyer on RSS and bandwidth: “All feeds will continue to use excerpts; I will not be publishing full-content feeds,... Read More

RSS melts the web from randomthoughts on May 5, 2004 3:35 AM

There has been some discussion of late about RSS readers causing the downfall of mankind. This has been discussed before on many occasions. Being that I like participating in the great world wide web as a responsible netizen I activated gzip encoding o... Read More

RSS in CF: Supporting Conditional GET from Big Damn Heroes (MXBlogspace) on June 4, 2004 8:55 AM

One thing that's missing from most CF-powered, dynamically-generated syndication feeds is support for Conditional GET. Here's what you need to know. *Why Conditional GET?* When aggregators (both client- and server-side) retrieve your feed from the Read More

RSS DDOS from Matt's Blog on July 22, 2004 7:05 AM

TITLE: RSS DDOS URL: http://dotnet.org.za/matt/archive/2004/07/21/2919.aspx IP: 196.25.219.121 BLOG NAME: Matt's Blog DATE: 07/22/2004 07:05:12 AM Read More

RSS DDOS from Matt's Blog on July 22, 2004 7:05 AM

TITLE: RSS DDOS URL: http://dotnet.org.za/matt/archive/2004/07/21/2921.aspx IP: 196.25.219.121 BLOG NAME: Matt's Blog DATE: 07/22/2004 07:05:25 AM Read More

RSS DDOS from Matt's Blog on July 22, 2004 7:05 AM

TITLE: RSS DDOS URL: http://dotnet.org.za/matt/archive/0001/01/01/2919.aspx IP: 196.25.219.121 BLOG NAME: Matt's Blog DATE: 07/22/2004 07:05:57 AM Read More

Kyrre skriver sin CMS ønskeliste, og jeg tenkte å skrive min Read More

Robert Scoble says that MSDN is turning off full-text RSS feeds because they eat too much bandwidth. I left a comment suggesting they turn on support for conditional HTTP GET, and I wanted to give some real-world numbers for the... Read More

Robert Scoble says that MSDN is turning off full-text RSS feeds because they eat too much bandwidth. I left a comment suggesting they turn on support for conditional HTTP GET, and I wanted to give some real-world numbers for the... Read More

Conditional GET Plugin from Confluence: blojsom on October 23, 2004 1:57 AM

DescriptionThe Conditional GET plugin reduces the bandwidth you send out from your blog for the flavors for which it is enabled. It works by examining various HTTP headers which indicate when the requesting client last polled your blog.... Read More

TITLE: re: Question on Caching an HttpHandler for Speed URL: http://weblogs.asp.net/rmclaws/archive/2004/11/04/252257.aspx IP: 66.129.67.202 BLOG NAME: Robert McLaws: FunWithCoding.NET DATE: 11/05/2004 02:01:21 AM Read More

last.fm script from chaotic intransient prose bursts on December 5, 2004 11:19 PM

A very crude and simple script to show your recent last.fm songs on your blog. Read More

Very interesting article discussing the benefits of only retrieving partial data using an HTTP Get when consuming RSS Feeds. Especially interesting as RSS Feeds with special media enclosures (Pod Casting) begin to strain the bandwidth of the average bl... Read More

The so-called RSS Bandwidth Problem is a meme that just won't frickin' die. I think Joel Spolsky started it way... Read More

Conditional Get from Eos Web Editor Project on February 11, 2005 1:44 AM

Conditional Get support has been added back to ewe. Mainly, even in an University where bandwidth is relatively cheap, because serving out 12-19MB a day (at lea... Read More

Support for RSS in Firefox, then Safari, are mostly encouraging developments to slowly further bring RSS into the mainstream. Both browsers are implementing the concept of "Live Bookmarks". Bookmarking a site now offers us the ability to not only save ... Read More

Previously: An Empirical Comparison of Programming Languages

Next: Livejournal RSS hack