Importance of the HTTP Date Stamp in SEO

The Google Webmaster Guidelines urges developers to make use of the If-Modified-Since HTTP header. As the guidelines state:

“This feature allows your web server to tell Google whether your content has changed since we last crawled your site. Supporting this feature saves you bandwidth and overhead.”

What this does is tell the search engine bots the date of the last change, and if the page hasn’t changed, then the bots will not reindex it. The implications of this are many:

• Less bandwidth is used because the bots won’t have to re-explore the page for their index, allowing them to go deeper into the site.
• Only changed pages will be reindexed.

What we experienced on our website was that prior to applying the HTTP If-Modified-Since header we experienced huge reindexing, where we would changed 10% of the site, but when the Googlebot detected that large of a change it would reindex 50% of the site, dropping our pages. As a result it took up to 6-8 weeks for our former rankings to return. With the HTTP If-Modified-Since header the reindexing was restricted to only the pages that have changed, reducing reindexing time from 6-8 weeks now down to 5 days. 

This was huge! Before, I had to gather everyone together: developers, business managers and database folks and make sure we were all on the same page in regards to migrating changes. I knew that with SEO you walked up the stairs, but got thrown out the window (in regards to indexing and rank). But I didn’t want to live my life on that sort of rollercoaster. This enables us to make changes without consulting everyone on the project, and also analyze the results the following week. Now I didn’t have to wait 6-8 weeks to see a reaction from the Search Engines in terms of improved (or degraded) ranking and indexing - I knew the result in 5 days.  No more guessing and hoping that in 2 months that the actions I had taken would result in a return to higher rankings - or waste all that time when I saw a result that lowered our ranking.

This is one of the first actions you should try to do when SEOing the site. Setting headers can be done in code, via page directives or by configuring your web server to do it.

Interestingly enough, even though this is an item that is part of the freakin’ Google Webmaster Guidelines, I was suprised to discover that none of our known competitors used it. I was flabbergasted! Why would you ignore a directive that Google puts out?

The reason this is ignored is that Google often tells you what to do, but stop at telling you what the effect will be. They only say it will save “bandwidth and overhead” which I assume to mean that they will not need to send out the same critical mass of Search Engine Spider visits (and the potential for overloading your server), but can send them out less frequently with improved results. What they don’t tell you (and what I had to find out), is that it effectively reduces reindexing time, allowing the bots to go deeper in your site, which increases your content depth on the SE’s. Reindexing also isn’t a heart attack of half your site being dropped from the index, allowing your rankings to both ‘walk up and down the stairs,’ rather than having your site (and possibly you) getting thrown out the window.