Search engines like Google and Bing use crawlers, sometimes also called bots or spiders, to gather information about all the content they can find on the internet (look at this now to get more details). It takes a tremendous amount of time for the crawlers to gather this information and to know what information to crawl and which to ignore, especially in densely populated areas.
So to increase the speed of crawling and serving web pages faster, there has been a huge shift towards using content management systems (CMS) that are able to automatically take content from the crawlers and automatically upload the content to the appropriate pages for humans to access. With these systems, content from these crawlers can be stored in XML files, then loaded into a CMS like WordPress or Drupal. When the human browsing the web sees that a particular page is missing, it will automatically prompt the user to search for that page in Google or Bing.
No longer are there 20 feet of text between Google and Bing
In order to speed up the crawl and serving of pages, Google has made a number of additions to the Google page index over the years. These include the quality algorithm, which has been used to reduce the amount of low-quality pages in the index, and in November 2010, the new QuickSort feature, which makes it significantly faster to crawl and index pages, especially in the same website. Now, it is significantly faster to search and index than it was before.
So what about that 20 foot gap between the home page of Google and Bing? Well, it’s not an insurmountable gap, as some reports have indicated that Google has recently made some serious strides in creating algorithms that can quickly index and rank up websites with content in the same location and web pages in different locations. But that’s not the extent of the breakthroughs Google has made in search engine indexing.
The biggest factor is that Google is willing to “adapt” search results in a way that will better serve its users, sometimes in an almost subconscious way. When a search query has high volume, the search results that Google is able to produce are going to be well organized, designed to answer questions in a straightforward manner, and if Google determines that you are on a social networking site, it will recommend relevant search results so that you can stay in the conversation.
While Google has made significant inroads in understanding and controlling this subconscious aspect of search results, Bing has gotten relatively sloppy over the years and has had a reputation for giving users much too little useful information in response to queries that are not obvious or not worded very well.