Invisible Web and Knowledge Discovery Tools: A Study

Abstract

Invisible Web which is also known as ‘Deep Web’ or ‘Hidden Web’ refer to information content that is ‘invisible’ to conventional search engines. Public information on the invisible web is currently around 500 times larger than the commonly defined World Wide Web. Nearly 550 billion individual documents are available in the invisible web while the surface web has around only one billion documents. It has been estimated that around 2,00,000 invisible web sites exist on the Information Superhighway at present and is the largest growing category of new information on the Internet. The total quality content of the invisible web is also greater than that of the surface web and highly relevant to every information need, market and domain. But, a larger portion of the invisible web is missing from search engines’ results pages. In this context, this paper attempts to characterize the deep web’s content and relevance to information seekers and profile special tools available to mine the invisible web.