The Deep Web and Darknet Defined

If we conceive of the Web as a data ocean, most of us are interacting with the wavy, transparent, easily navigable Surface Web (see Figure 1). The Surface Web is the portion of the Web that has been crawled and indexed (and thus searchable) by standard search engines such as Google or Bing via a regular web browser. In the darkness below, beneath the electronic thermocline, are the abyssal depths of the Deep Web (also referred to as the Invisible Web or Hidden Web) – the portion of the web that has not been crawled and indexed, and thus is beyond the sonar reach of standard search engines. It is technically impossible to estimate accurately the size of the Deep Web. However, it is telling that Google currently the largest search engine – has only indexed 4-16 percent of the Surface Web. The Deep Web is approximately 400-500 times more massive than the Surface Web (See brightplanet.com). It is estimated that the data stored on just the 60 largest Deep Web sites alone are 40 times larger than the size of the entire Surface Web (See thehiddenwiki.net).

Growing rapidly within the Deep Web is the Darknet (also referred to as the Dark Web, DarkNet, or Dark Internet). Originally, the Darknet referred to any or all network hosts that could not be reached by the Internet. However, once users of these network hosts started sharing files (often anonymously) over a distributed network that was not indexed by standard search engines, the Darknet became a key part of the Deep Web. Unlike the traffic on the Surface Web or most parts of the Deep Web, most Darknet sites can only be accessed anonymously. Preliminary studies have revealed that the Deep Web actually contains the largest expanding reservoir of fresh information on the Internet These websites are usually narrower, but with a much deeper content material, as compared to regular surface sites. Furthermore, because most of the materials are protected content, the overall quality of the content from the Deep Web is typically better and more valuable than that of the Surface Web. It is also estimated that more than 50 percent of the Deep Web content is located in topic-specific directories(www.thehiddenwiki.net), making it even more accessible and relevant to targeted searches. And the Deep Web and Darknet are growing. Multiple technologies, such as ubiquitous computing, distributed/cloud computing, mobile computing, and sensor networks, have all contributed to the expansion of the Deep Web.

Advances in secure/anonymous web hosting services, cryptocurrency/Dark Wallet, and development of crimeware are further contributing to the growth of the Darknet. A variety of cryptocurrencies such as Bitcoin, Darkcoin, or Peercoin (see coinmarketcap.com for a complete listing) have been in use for anonymous business transactions that are conducted within and across most Darknet marketplaces. Hackers for hire and multilingual call centers have also accelerated the growth of Darknet. Of course, there are also plenty of legitimate uses of the Darknet by journalists, political dissidents, whistle-blowers, and human rights advocates. Not surprisingly, Chelsea (formerly Bradley) Manning, Julian Assange, and Edward Snowden all relied heavily on the Darknet for their cause and activities.