The confusion surrounding deep web

When talking about internet indexing, conversations usually revolve around two extremes, the surface web, and the dark web. People often use the terms “dark web” and “deep web” interchangeably.

People often say that the deep web is inaccessible. This statement is wrong. You use the deep web every day. You are what makes up the deep web. To explore where this “inaccessibility” of the deep web comes from, we need to take a look at how search engines work.

A search engine is first fed the addresses of a few initial web pages. The engine downloads these pages and then scans them for hyperlinks. The new addresses are then fed to the engine again. This process is called crawling and by extension, a search engine is called a web crawler.

A website developer can ask a web crawler not to index some parts of a website. These pages include user information, passwords, hidden content, emails, bank accounts, paid content and encrypted pages. These hidden pages make up the deep web.

Search engines can potentially index the dark web. However, to index the dark web, a web crawler needs special software that the developers choose not to include in the search engine. This software is called TOR (short for The Onion Router) and can be quite resource-heavy. Sometimes dark websites do not want to be found. For this reason, they actively block web crawlers on their pages.

In short, search engines don’t index the dark web or the deep web. Both for different reasons.