Aug
15
2012

Search Engine History – How Google Came to Dominate

So how did Google get from a market share of less than 1% in 1999 to almost 80% today? What is so special about Google and how has it come to dominate the scene? In this article, the third in my search history series, I look at the role of luck, timing and the mistakes of competitors in the rise of Google. I also evaluate the unique technology and vision at the heart of the Google algorithm.

The Dot-com Shake-out

You might be surprised to hear me moan about the dot-com mania of 1999-2000. After all, for many of us it was an exciting time. However, for the early search engines, it was essentially a death-knell! The new money entering the market was huge but investors wanted to see returns. Search Engines were clearly part of the future, but where was their revenue stream? How could one make money from free results – especially free results powered by very expensive computer infrastructures? This impatience grew following the dot-com crash of March 2000.

Yahoo! seemed to be the only company with any kind of sustainable “pure internet play” (from banner advertising which was at that time well funded by the dot-com dollar). At the same time, AOL were pursuing a moderately successful “walled garden” approach to traffic (i.e. where 70% of searches on the site led to content within AOL properties). The argument at the time was that making it easy for people to “exit” to other sites (the essential purpose of the modern search engine) meant losing the opportunity to sell them anything from your own site and content partners.

Altavista, acquired with DEC by Compaq in 1999, was relaunched at huge cost as a portal competitor to Yahoo! Similarly, Excite (now owned by broadband provider @home) tried to become an AOL clone. Lycos, already the largest “pure portal” in the Spring of 1999, merged with the largest Spanish language ISP, Terra Networks (and then ran itself into the ground).

Disney/ABC and NBC had already invested heavily in Infoseek and Snap respectively but quickly began to realize that the revenue they could derive from the increasingly portal-like services was never going to deliver the short term payback which shareholders typically seek. Steve Bornstein, chairman of Walt Disney Internet Group, said in his press release “the Internet environment has continued to shift and change, and therefore our strategies must also change”. 400 employees at Go Network were laid off and a buyer sought for the Infoseek engine.

In December of 2000 a surprising report from StatMarket.com found that only 6.86% of all the referring traffic to websites originated from search engines (compared to 47.01% from direct navigation or bookmarks and 46.13% from internet links). The quality of this research has since been questioned, as it was based not on server logs but on the global Hitwise stat counter data (the code for which was only placed by many webmasters on the homepage of their site). However, this data came at a time when there was a collective loss of confidence in search and portal technologies. The dot-com crash of March 2000 had left investors with a major hangover.

Is it any surprise, then, that so many search properties disappeared altogether (or closed their internal crawl capability down) in 2001? With the closure of Northern Light (to the public at least) in January 2002, the Dot-Com shakeout had essentially closed or fatally wounded WebCrawler, Magellan, Excite, Infoseek, Go Network, Snap and Lycos. It also critically wounded Altavista, who were to struggle on without any real investment in their search capability until their acquisition by Yahoo! in 2003.

The Emergence of Paid Search

If the rush to convert all search engines to portals was (at the time at least) flawed strategic thinking, bad timing also played a part in the demise of so many early contenders.

Quietly and almost unnoticed during the mania was a relatively minor search player called go-to.com, renamed Overture in 2001. Whilst paid listings had been tried before, Overture (with a 2.76% market share in January 1999) were the first company was make a success of it, with some advertisers paying up to a dollar a click by 1998/9. Go-to.com, in fact, only carried paid results! Personally, I never really liked or used the site myself for that exact reason. However, they persevered with their strategy, determined to find a sustainable revenue model and confident that the research-oriented search engines of the time had little to offer a growing army of web users looking to shop ’til they dropped!

History has proved that Overture were right (to persevere with paid ads) although initially there was limited evidence of this. Perhaps if the dot-com investors had only waited a little longer (or started a little later) the value of sponsored results would have become clear. Either way, the big boys hared off into portal oblivion, whilst the Overture tortoise (and the model it spawned) ultimately won the race.

All the best businesses start in Garages

It really is true. The 12×18 foot garage, in which David Packard and William Hewlett launched Hewlett-Packard in 1939, is considered by many to be the birthplace of Silicon Valley. A short drive away is the Los Altos garage of Steve Jobs’ parents, where Apple Computer Inc. was born. Google Inc, similarly traced its early days to a garage – this one in Menlo Park, California – where the newly formed Google Inc spent its first five months as a proper business in late 1998.

However, the real story of Google began more than two years earlier, as the research project (nicknamed “BackRub”) of two Stanford Ph.D. students: Michigan-born Larry Page and Russian Sergey Brin. The simple obsession which preoccupied both was the links between sites on the exponentially growing web. In his original thesis research, Page hypothesised that any link from one site to another was akin to an academic citation and the description (or anchor text) attached to that link like an annotation. If one could identify the sites most cited for any given annotated subject, one could establish a pecking order of significance for any given subject area.

What particularly fascinated Page was what he saw as a fundamental weakness in Tim Berners-Lee’s original design for the web; namely that it was possible to see easily what any site was linking to but very hard (if not impossible) to see the sites linking back. Backrub sought to catalogue all the links on the web and, through doing so, establish which sites had the most “authority” overall and in any given topic. Page insists that it was never his intention at this point to create a search engine. However, intellectual curiosity drove him forward into the creation of a new kind of crawler, based on a link ranking system (now called PageRank after it’s inventor) which was the first of it’s kind.

It was at this point that Sergey Brin was drawn into the project. At this time. Page estimated the size of the web to be approximately 10 million documents and the number of links to be perhaps 100 million. Altavista had impressed everyone by managing to index all those documents on a single large computer. However, indexing both the documents and the links was, to say the least, a big hairy problem; precisely the sort of challenge that appealed to Brin.

In fact, the problem was much tougher than either probably realized at the time. At this point, the web was growing at an exponential rate and the computing resources required for the task were in fact well beyond the humble resources of a student project. This brings me to the part played by lady luck in the formation of Google.

In late 1996, the BackRub crawler was consuming nearly half of all Stanford’s network bandwidth and, on occasions bringing down Stanford’s internet connection. Page and Brin had to beg and borrow storage capacity and were continually running out of space and lines of credit. Whilst both were hardly mucking about, they were not completely sure about a career in business. As Page puts it now, he was either going to be a professor or run a company that made a difference. Nothing in between would do.

So, believe it or not, in the period from 1996-7, all three of the then market leaders were approached by two students with an idea for licensing a new kind of search engine. In August 2004, at the Search Memories session at SES San Jose, the panel included Doug Cutting (formerly Senior Architect, Excite), Steve Kirsch (formerly founder of Infoseek) and Louis Monier, (formerly CTO at AltaVista). Together they reminisced about the early years of search engines and all admitted, one by one, that they had sent the founders of Google packing. Steve Kirsh was the most colorful; “Go pound sand I told them”. Doug Cutting said “I wasn’t impressed with their demo at all” and Louis Mornier shrugged “I didn’t have the authority to sign a check anyway”.

What if Larry Page and Sergey Brin had been just a little bit better at PowerPoint? What if Louis had had his check book with him? This book might have been called “How to get to the top of Altavista” and Digital might still have been in business, continuing the make the machines that power the internet. Lady luck certainly played her part in the proceedings!

However, despite their efforts to dodge destiny, the Stanford graduates did not find a suitable established partner. Some are born great, others achieve greatness and some have greatness thrust upon them. Perhaps Larry and Sergey fall into that rare latter category. By the end of 1998, the Backrub service (renamed Google in September 1997) was growing fast and serving more than 10,000 daily queries. The guys were introduced through a shared contact to Andy Bechtolsheim, an active early-stage investor. After a short demo, Andy literally thrust a check for $ 100,000 into

Leave a Reply