Do you ever search for help on a problem or some piece of information on Google? Do you ever find that for some reason, some sites show high in the list but when you click the link you get welcomed to a "login" page with maybe a short excerpt?

I run into this quite a bit, and I'll list a few sites that I've run into this with. But first, an analysis of how and why sites do this, and how I've gotten around it.

I'm not sure if this behavior is flagged by Google as they are quite secretive about what goes into their Special Sauce, but to me it seems wrong. Google hosts a free-for-the-user service and generates money from advertisers hoping to make money from those users. When someone subterfuges that to try an make a buck it smells bad.

User Agent

Every web browser has an identifier that it sends to a web server every time you request a page. This ID is referred to as a user-agent (UA). ( I know this is quite elementary, and most of you have figured everything out already after this one sentence, but I like to be complete.) The user-agent is used for many different purposes some of which include identifying the browser type, operating system type, and the current version of each of those. The user agent is also often used to communicate to the web server information about certain extensions installed in your web browser. It is quite common for sites to deliver slightly different content to different browsers in order to overcome some certain incompatibility or to chide you about your poor choice of web browsers.

Robot Spiders

The robots that Google and all other search engines also have UAs in order to be good netizens and to allow sites to block certain data from being indexed. Imagine having a section of your site indexed where you used hyperlinks to delete items from a database (this actually happened and would be a great write-up for another day)

The culmination of all of this is the fact that anything accessing somone's site has a UA, and all web indexing bots have uniquely identifiable UAs. The people that noticed this relationship have setup a corollary to the actual purpose of the search engines' internet goodwill. They have in fact decided to give search engines a different version of their site than they deliver to end-users. The site owners do this purely to build their user base and in effect their advertising and subscription revenue.

In Action

Lets see an example: I searched Google for "DateTime sql 1900" to see how 1900-01-01 in sql related to a .NET DateTime object. The very first hit is http://www.sqlservercentral.com/articles/Advanced+Querying/workingwithdatetime/1634/ and if you go to that link you will see that you get only the first paragraph of that article, even though Google has indexed the whole thing and is delivering search results based on the contents of the whole article.

Work Around

There are a couple things that you can do to work around this, the easiest is to just to click on the Google cache link to see the actual content that Google indexed off of. The problem with this approach is that a lot of times this cache is not available; this is the case with our example above. The more effective solution is to change your UA so that the site thinks that you are Google.

Changing your UA can be a tricky proposition involving a registry hackathon with some browsers. Luckily, if you are using Firefox there is an easier way. There is an add-on extension called - of all things - User Agent Switcher that lets you quickly choose a user agent from a pull down menu. It does not come with a list of UAs for search engines, but by searching you should be able to find as many as you need.

Here we have setup a UA called Googlebot with a value of "Googlebot/2.1 (+http://www.google.com/bot.html)" by simply selecting this from the pull down and refreshing the page we get the whole article

I've been meaning to write this down for a while I just procrastinated too long, and I guess I was more bored this time than the times before. Anyways, I may contact Google to see if this is anything that they will do anything about, I would like my browsing experience to be such that when I click on a top link that I am going to get the most relevant data. In the past this has been posting by people with the good of the community at heart but more often now days I'm finding the Google Whammys... And Stop.

List of sites (let me know if you run into any)

• sqlservercentral.com
• techtarget.com
• windowsitpro.com (penton property)
• sqlmag.com (penton property)
• exchangeprovip.com (penton property)
• scriptingprovip.com (penton property)
• securityprovip.com (penton property)

I am obfuscating the name of the site domains so that they don't get any link-back points from search engines