Google search: Encyclopedia II - Google search - The search engine

Google search - The search engine

Google search - Index size

• ~ 1998: ~ 25,000,000
• August 2000: 1,060,000,000
• January 2002: 2,073,000,000
• February 2003: 3,083,000,000
• September 2004: 4,285,000,000
• November 2004: 8,058,044,651 web pages, 880,000,000 images, 845,000,000 Usenet messages, 4,500 news sources
• June 2005: 8,058,044,651 web pages, 1,187,630,000 images, 1 billion Usenet messages, 6,600 print catalogs, 4,500 news sources

(source: Internet Archive ⁴, GoogleBlog ⁵, Google Groups ⁶, Google Catalogs ⁷)

Google search - Physical structure

Google employs data centers full of low-cost commodity computers running a custom Red Hat Linux in several locations around the world to respond to search requests and to index the web. The server farms in the data centers are built using a shared nothing architecture. The indexing is performed by a program named Googlebot, which periodically requests new copies of web pages it already knows about. The more often a page updates, the more often Googlebot will visit. The links in these pages are examined to discover new pages to be added to its internal database of the web. This index database and web page cache is several terabytes in size. Google has developed its own file system called Google File System for storing all this data.

Please see Google platform regarding the number of Google's servers and their hard- and software.

Google search - Programming technology

Google use their own concept for distributing the task of processing collected data. Chunks from the Google File System of typically 64 MB are processed by the MapReduce framework. This framework makes it possible to apply the map and reduce concepts from functional programming languages across the data stored in the GFS. First a function is mapped across the collected data, then the result is reduced. For example a function extracting the hostname of the URL can be mapped across all pages, it is then sorted and reduced, yielding a figure of how many times a certain hostname has occurred. All mapping and reducing is massively parallelized across the nodes and fault tolerant, so if nodes crash or misbehave during map reduction, work is moved over to another machine.

Google search - PageRank and indexing

Google uses an algorithm called PageRank to rank web pages that match a given search string. The PageRank algorithm computes a recursive figure of merit for web pages, based on the weighted sum of the PageRanks of the pages linking to them. The PageRank thus derives from human-generated links, and correlates well with human concepts of importance. Previous keyword-based methods of ranking search results, used by many search engines that were once more popular than Google, would rank pages by how often the search terms occurred in the page, or how strongly associated the search terms were within each resulting page. In addition to PageRank, Google also uses other secret criteria for determining the ranking of pages on result lists.

Google not only indexes and caches HTML files but also 13 other file types ⁸, which include PDF, Word documents, Excel spreadsheets and plain text files. Except in the case of text files, the cached version is a conversion to HTML, allowing those without the corresponding viewer application to read the file.

Google may have difficulty indexing some websites, in particular those that use frames, links embedded within JavaScript or Java, or complex URLs with more than six variables in the query string. Google offers an explanation why some web pages haven't been included ⁸.

Users can customize the search engine somewhat. They can set a default language, use "SafeSearch" filtering technology (which is on 'moderate' setting by default), and set the number of results shown on each page. Google has been criticized for placing long-term cookies on users' machines to store these preferences, a tactic which also enables them to track a user's search terms over time. For any query (of which only the 32 first keywords are taken into account), up to the first 1000 results can be shown with a maximum of 100 displayed per page.

Despite its immense index, there is also a considerable amount of data in databases, which are accessible from websites by means of queries, but not by links. This so-called deep web is minimally covered by Google and contains, for example, catalogues of libraries, official legislative documents of governments, phone books, etc.

As an April Fool's parody of PageRank, Google introduced an explanation of something called "PigeonRank" ¹⁰

Google search - Google optimization

Since Google is the most popular search engine, many webmasters have become eager to influence their websites' Google rankings. An industry of consultants has arisen to help websites raise their rankings on Google and on other search engines. This field, called search engine optimization, attempts to discern patterns in search engine listings, and then develop a methodology for improving rankings.

One of Google's chief challenges is that as its algorithms and results have gained the trust of web users, the profit to be gained by a commercial web site in subverting those results has increased dramatically. Some search engine optimization firms have attempted to inflate specific Google rankings by various artifices, and thereby draw more searchers to their clients' sites. Google has managed to weaken some of these attempts by reducing the ranking of sites known to use them.

Search engine optimization encompasses both "on page" factors (like body copy, title tags, H1 heading tags and image alt attributes) and "off page" factors (like anchor text and PageRank). The general idea is to affect Google's relevance algorithm by incorporating the keywords being targeted in various places "on page," in particular the title tag and the body copy (note: the higher up in the page, the better its keyword prominence and thus the ranking). Too many occurrences of the keyword, however, cause the page to look suspect to Google's spam checking algorithms.

One "off page" technique that works particularly well is Google bombing in which websites link to another site using a particular phrase in the anchor text, in order to give the site a high ranking when the word is searched for.

Google publishes a set of guidelines for a website's owners who would like to raise their rankings when using legitimate optimization consultants ¹¹. The New Zealand DMA offers a more comprehensive guide to SEO ethics standards ¹².

Adapted from the Wikipedia article "The search engine", under the G.N U Free Docmentation License. Please also see http://en.wikipedia.org/wiki