Google, envisioned and originally developed by Page & Brin, first crawled the web in March of 1996. There were several pioneer concepts involved in making Google what it is today, but let’s look at the first two.
The Crawler
University professors for years have learned to publish or perish, and the papers they write are generally authored from extensive research. They, in turn, build on the research before them with their own work. The references in their work act as backlinks to the previous work, and these citations are important to add value and credibility to the current work.
Page began by trying to apply this same principle of backlinking to a search engine in evaluting sites returned on a search. About 10 million pages existed at that time, far, far less than the billions of pages today. Nothing new with this, as Alta Vista had already come online in December of 1995 with their own search engine. The difference came in the way Page envisioned being able to qualify the pages based on the number of backlinks and the value of each as related to the destination page. If you drop back one page on the backlinks, the value of that page, in turn, depends on the number and quality of the links into it. The whole process is complex and recursive, and the number of calculations involve increase much, much more rapidly than the number of pages indexed.
Evaluating the Links
Brin was the math genius here that helped Page with developing the mathematical process for evaluating the quality of the backlinks. This ranking system, now called PageRank, is patented by Google and has become one of the critical components in Google’s claim to fame. For example, a link into your site from a CNN page with high PageRank is probably worth much, much more than a link into your site from a college kid’s site listing his references for his or her term paper. Also, from looking at Google’s patent application, you can learn that Google is (or is planning to) include history information of the links in evaluating their quaility as well; i.e., the growth rate of links to your site and how long they have been there are all important.
The Catch-22
Now here is the problem. Tools are now available to anyone (we have them, we sell them) for evaluating the number and quality of links into your site. You can do your own backlink analysis now. In fact, this analysis is VERY important in the highly competitive web environment if you wish your site and pages positioned well on people’s returned results. Google doesn’t want you to have this information on your backlinks in their site, or you could figure strategies to spam their ranking. So Google cripples their link command you can use to get your backlinks there and your PageRank value displayed on their free toolbar so you can’t see what Google is really doing. The values they do give you are red herrings.
So what happens to my Google listing? My business site (it’s been around for years) positions well, but new sites are sandboxed. Not much value and I can’t even see the current links or PageRank. I go next door to Yahoo, MSN, and do my analysis. The backlink listing they return is highly accurate. I can assume these same backlinks exist in Google. My advertisement goes to Yahoo. It is up and I’m positioned very well in 3 days.
There is a catch-22 there. Google is protecting their system at the cost of driving customers away. I know how Google’s ranking system works. I work carefully and strategically following the rules over time and eventually get positioned well. Meanwhile, Yahoo and MSN run away with the goodies.
Want more on the history of Google? Try John Battelle’s article “The Birth of Google” in Wired’s August issue. Or better yet, check out John’s book The Search: How Google and Its Rivals Rewrote the Rules of Business and Transformed our Culture.