Both Google and Yahoo process millions of queries a day and still manage to provide you blazing fast results in split seconds, The secret behind these superfast search results is a software named "Hadoop" which is a free Java software framework running on a large computing clusters for data intensive distributed applications, Hadoop enables applications to easily scale out to thousands of nodes and petabytes of data.
Hadoop divides an application into small fragments of work, each of which is then executed on nodes of the cluster, Each portion of data is processed separately and brought back together using a Map/Reduce methodology.

According to Yahoo it uses world's largest Hadoop production application, The Yahoo! Search Webmap is a Hadoop application that runs on a more than 10,000 core Linux cluster and produces data that is now used in every Yahoo! Web search query.