How do I rank higher in Yahoo? Use an XML Sitemap

Here’s a secret SEO tip of the week: XML Sitemaps work exceedingly well for Yahoo.  This doesn’t seem to be well known, at least I can’t find it from casual searches. After paying a ton to be a part of Yahoo’s Search Submit Pro Trusted Feed, we decided to cut loose from that program since it was not paying off in terms of conversions.  Subsequently, our rankings went from the first page of results to nonexistent.

This went on for 9 long months. I joined the company in month 8 and asked around if an updated XML sitemap had been submitted to Yahoo. Everyone I talked to seemed to think that the former SEO Analyst had already submitted one. But I couldn’t find any evidence of that, so I submitted an updated one. Results: First page results 93% of the time for the search phrases we target. This occurred over a 4 month period. All from submitting an XML Sitemap and then an updated one 2 months later.

The reason I submitted an updated sitemap was that I discovered that Yahoo was listing web pages from our site that no longer existed. After some digging I realized that the URLs listed in Yahoo had variables that only existed in our old XML sitemap. It was then I had the theory that Yahoo weighed XML sitemap submissions higher. Once I submitted an XML sitemap and saw immediate results I knew my hunch was correct. Yahoo, at least at the time of submission, seemed to almost take the XML sitemap wholehog and apply it to its results.

Your mileage may vary: search engines change their algorithms all the time. What worked in the past 6 months for me may no longer work.  Of the three things that affect search engine results:

1. What you do to your site
2. What your competitors do on their sites
3. What the Search Engines do to their algorithms

- of the three, you only control one: what you do to your site.  Yahoo just went through another algorithm change, so we’ll have to see if this affects how XML Sitemaps affect Yahoo rankings, but this worked very effectively for me (as you can see from the chart above).

Is Google violating copyright law?

SearchEngineWatch reported on a Belgium newspaper suing Google with the headline:

Belgian Newspapers Want $77.5 million in Damages from Google

Interesting.  I’m not a lawyer, but I am familiar with the Berne copyright convention where almost everything created privately and originally after April 1, 1989 is copyrighted and protected whether it has a copyright notice or not. What this means is that anything created is automatically copyrighted.

I mean, what would you think if you found another site copied your website word for word, image for image? Wouldn’t you think that would be a copyright infringement? Well, Google does that all the time through their caching of websites:

Brad Templeton has an interesting post on the subject of copyright.  He makes a point of some of the exceptions, such as making one copy of music does not violate copyright laws, whereas the mass copying through a file sharing software does violate copyright. So, would Google’s caching of sites fall into the latter area of “mass copying?”

This decision on caching would affect other sites such as the Wayback Machine. This site caches old versions of websites, so would be very similar to Google’s caching.

Here’s another thing: it was a newspaper that brought the suit. Newspapers follow a business model of providing the latest news articles everyday, and for internet news sites this means every hour and even every minute. Newspapers also publish corrections and retractions. Having an older version of their news stories for those reasons can be seen as very negative from their point of view.

 Now, some people have the view that Google is doing this newspaper site a service by driving traffic to them. But is caching a site part of offering relevant results from a search? I think not. I think it’s added value that Google offers. But to what end? The web is a dynamic, changing place where webpages get dropped and added to - all the time. Google caches pages in order to always have something to display from it’s results - even if what it displays is days or weeks old. Having dead ends in their search results would not look good from their business perspective. But flagrantly violating copyright without express permission of the owners beats down a path - to what exactly? I think it infringes upon the rights of the owners. Google takes the stance that it is up to the owners of websites to opt-out, either via a robots.txt ‘disallow’ entry or some other method. My view that it is really Google’s responsibility to ask permission before caching websites. I can imagine that their response would be: well, that’s really too hard to do. Well, boo hoo, Big Goo. I guess that’s why they call it work.

Google runs roughshod over the newspaper website’s copyright in the name of ‘added value,’ mandating their right to do so. They profit by offering results, even if those results are considered outdated by the original owners of the sites. For a company that prides itself on ’doing no evil,’ this seems to be on the border. I think that it is well within the rights of the Belgium newspapers to assert their copyright of their original work - now we’ll see how the legal establishment sees it.  

SEO Rap

“Design Coding,” a rap about SEO with good basic Search Engine Optimization advice. 
By m0serious

This guy raps about SEO SEM and Social Media topics. His advice is sound - and entertaining!

SEO is a zero sum game

What do I mean by that? What this means is that when one site moves to a #1 ranking, another site moves to #2. If you move up, then you have eaten your competition’s lunch. Like Daniel Day Lewis’ character in the movie There will be Blood says, “I drink your milk shake! I drink it up!”

Moving your site up in rank directly relates to traffic growth. Google spilled the goods a couple times, and some smart people translated those findings into the value of rank and how it translates into traffic. By moving your site from 6th to #1 you can effectively drive 10x more traffic to your site. 

On a chart comparing our competitors to the site I was working on I saw an interesting phenomenon: as the client’s site moved up a particular competitor moved down in an almost mirror image in the charts. I could see that we were “stealing” rank from the competitor. It was clear to me that this is also what I had to watch out for: If your competitors are doing SEO and you are not, Search Engines will unapologetically weed your site out of existence. 

 And if you were not on the first page of results, you might as well be nonexistent. Scratch that: If you are not in the top 3 positions your site may as well be nonexistent. And that makes sense, doesn’t it? What users place into the search field is a question, and if that question is answered in the #1 site returned why would they click on #2?

Why Google is your homepage

More than 80% of sites are initially discovered through Search Engines.

But don’t believe me, I got that stat from Forrester Research and Georgia Tech’s GVU Center User Surveys.

What does this mean?

What this means is that your home page is really not your home page, Google is.

Have you ever tried getting to deep content using your link structure? Meaning clicking from link to link to get to something 3-4 levels in? If your site is like most sites your usability sucks and it brings your visitors to tears. I know that on most of the corporate and e-commerce sites I am on, when I try to find a particular product or deep page I ignore their convoluted link structure and go straight to Google.

A great tool is to use the “site:” modifier in the search field. This works in Google, live.com (MSN), and Yahoo. What this does is limit your search to a particular domain. Sooo, an example would be:
“site:amazon.com harry potter”. The search engine will then limit the search string to a particular site.

Now, Amazon has a pretty decent search function. The search field is prominent, and has a dropdown for filtering search results. But many sites handicap themselves by having a retarded search function. Check out UCompareHealthcare.com (http://www.ucomparehealthcare.com/). Their search function has 4 search fields. What the hell? And just try to use their link structure: first it makes you pick a type of search: state, Specialty, or name. After that  it took me 5 clicks to get to a particular doctor. Shoot me now. I would much rather go to Google and do a search on: “site:UCompareHealthcare.com dr. lammoglia”. Gee, I immediately get 10 links that I can look through. Having 4 search fields in this era of Google breaks the first rule of usability:

“Don’t make me think.”

Importance of the HTTP Date Stamp in SEO

The Google Webmaster Guidelines urges developers to make use of the If-Modified-Since HTTP header. As the guidelines state:

“This feature allows your web server to tell Google whether your content has changed since we last crawled your site. Supporting this feature saves you bandwidth and overhead.”

What this does is tell the search engine bots the date of the last change, and if the page hasn’t changed, then the bots will not reindex it. The implications of this are many:

• Less bandwidth is used because the bots won’t have to re-explore the page for their index, allowing them to go deeper into the site.
• Only changed pages will be reindexed.

What we experienced on our website was that prior to applying the HTTP If-Modified-Since header we experienced huge reindexing, where we would changed 10% of the site, but when the Googlebot detected that large of a change it would reindex 50% of the site, dropping our pages. As a result it took up to 6-8 weeks for our former rankings to return. With the HTTP If-Modified-Since header the reindexing was restricted to only the pages that have changed, reducing reindexing time from 6-8 weeks now down to 5 days. 

This was huge! Before, I had to gather everyone together: developers, business managers and database folks and make sure we were all on the same page in regards to migrating changes. I knew that with SEO you walked up the stairs, but got thrown out the window (in regards to indexing and rank). But I didn’t want to live my life on that sort of rollercoaster. This enables us to make changes without consulting everyone on the project, and also analyze the results the following week. Now I didn’t have to wait 6-8 weeks to see a reaction from the Search Engines in terms of improved (or degraded) ranking and indexing - I knew the result in 5 days.  No more guessing and hoping that in 2 months that the actions I had taken would result in a return to higher rankings - or waste all that time when I saw a result that lowered our ranking.

This is one of the first actions you should try to do when SEOing the site. Setting headers can be done in code, via page directives or by configuring your web server to do it.

Interestingly enough, even though this is an item that is part of the freakin’ Google Webmaster Guidelines, I was suprised to discover that none of our known competitors used it. I was flabbergasted! Why would you ignore a directive that Google puts out?

The reason this is ignored is that Google often tells you what to do, but stop at telling you what the effect will be. They only say it will save “bandwidth and overhead” which I assume to mean that they will not need to send out the same critical mass of Search Engine Spider visits (and the potential for overloading your server), but can send them out less frequently with improved results. What they don’t tell you (and what I had to find out), is that it effectively reduces reindexing time, allowing the bots to go deeper in your site, which increases your content depth on the SE’s. Reindexing also isn’t a heart attack of half your site being dropped from the index, allowing your rankings to both ‘walk up and down the stairs,’ rather than having your site (and possibly you) getting thrown out the window.

SEOBook no longer available at SEOBook.com

I just discovered by going to seobook.com that I am no longer able to purchase Aaron Wall’s acclaimed SEOBook. This blows. I am a professional SEO Analyst for a company-that-I-will-not-name and have done quite well, doubling our traffic in a 4 month period. Even with that success I thought that perhaps there were things that Mr. Wall could teach a new SEO Analyst. I will not know the answer to that question. The SEOBook, as far as I can tell, is not for sale on the SEOBook site (a tad ironic, no?). I was hoping to steal glean some ideas from reading this tome, but it looks like the folks over at SEOBook are turning to a subscription model that offers training, forums, etcetera for $100 a pop. I’m sure that’s all well and good (for Aaron), but how about the throngs of SEO neophytes without $100/month to burn?

Well, here I am (what, no thunderous applause?).  I am going to reveal the things that work with SEO, some ideas are readily available, and some have come about due to the daily experimenting that I do as part of my job. I don’t think they are actually secrets per se, but are often overlooked by the masses, or else a debunking of commonly held beliefs regarding the science (not art) of SEO. The reason I say ’science’ (and not art) is that search engines are mechanical. They respond to certain things in a proscribed way, and don’t respond to others. Some SEO gurus use the word ‘art’ to couch their ideas in something mysterious (and so they can charge you more), but the mystery part is where the search engines do not reveal all the ways to affect their rankings. It’s not in their best interests to do so. The day Google announced that external links were the core of their search algorithm was the day porn and gambling sites started linkfarms from domains they owned in order to rank high for unrelated but popular terms, like “Britney Spears” (okay, not the best example, since she does have a video out, and I do not mean music video).

Since of the three things that influence search engine rankings are:

1. What you do on your site
2. What the SEs do to their search algorithm
3. What your competitors do on their sites

- You only control one of them: your site (or sites). The mystery is what Google does on their search algorithm, and what  actions your competitors do on their sites. But really: it’s all mechanical and scientific and programmatic. So, if you take a scientific approach to SEO - with analysis, developing a hypothesis, constructing a  test, and then analyzing the results - you will obtain the knowledge necessary to rank well.  That is the approach I take, and so I can inform you, with authority, on what works and what doesn’t work in regards to SEO. And since the Search Engine Algorithms change all the time, this should keep me busy for awhile in naming the best practices, and then updating them based on SE changes.

Stay tuned…