The story of web search engines begins before the web existed, with a tool that could find a file only if you already knew its name. Over about thirty-five years the job changed from matching filenames, to matching words, to ranking pages by who linked to them, and now to writing a short answer for you. Each stage solved the previous stage’s biggest complaint and created a new one.

This is a tour of that progression, with the dates that can be checked and a plain statement of what is still contested, especially around the newest part.

Where did the idea of search come from?

Before any software, there was a thought experiment. In 1945 the engineer Vannevar Bush described the memex, a desk-sized device that would store a person’s books and records and let them link one item to another along trails of association. Nothing like it was built then, but the idea of navigating information by links, rather than by shelf order, is the ancestor of both the web and search.

The first working tools were far humbler. Directory lookups such as WHOIS in 1982 let you search for people on a network. They did not search documents at all.

What was Archie, and why does it count?

Archie is the usual starting point. Alan Emtage, a student at McGill University in Montreal, released it on 10 September 1990. The internet then was a collection of file servers, and the way to share software was anonymous FTP: a server that anyone could log into and download from. Finding a file meant knowing which server held it.

Archie solved that by collecting directory listings from those servers and building an index, updated about monthly. You typed a filename, and it told you which servers had a file with that name. It indexed titles only, never contents, and it did not understand natural language, so you needed to know roughly what the file was called. Even so, at its peak it accounted for around half of the internet traffic in Montreal, which gives a sense of how useful it was.

The telling detail is that Archie was built by a student solving a practical problem, which is how a good many search tools have started.

Its successors followed the same model for another system. Veronica and Jughead, in the early 1990s, searched the menu titles of Gopher, a text-based browsing system that preceded the web for many sites.

How did search move onto the web?

In June 1993 Matthew Gray’s World Wide Web Wanderer began building an index of web servers, mostly to measure how big the web was. In September of that year W3Catalog appeared, built from hand-maintained lists. ALIWEB relied on webmasters submitting descriptions of their own sites. In December 1993 JumpStation combined crawling, indexing, and searching in one package, though it covered only page titles and headings.

The next step mattered more. WebCrawler, in 1994, was the first to let you search the full text of pages. Lycos, born the same year at Carnegie Mellon University, did similar work at larger scale. The web was growing quickly, and a tool that could find a word inside a page was clearly more useful than one that read titles.

Meanwhile a different philosophy was taking shape. Yahoo began in 1994 as a directory, a human-edited list of sites sorted into categories. The approach could be tidy for popular topics, but no team of editors could cover everything. In 1995 Yahoo added search, but it searched the directory, not the full text of the web.

How does a search engine work?

Almost all of them, then and now, do three jobs. Crawling comes first: a program called a crawler starts from a list of known addresses, downloads the pages, extracts every link it finds, and adds the new addresses to a queue. Then it repeats the process. Because the web is far too large to fetch all at once, the crawler has to prioritize, and it has to avoid hammering any single server. Site owners can use a file called robots.txt to say which parts of a site crawlers should leave alone.

Indexing is second. The engine processes the downloaded pages and builds a structure much like the index of a book, listing for every word the pages where it appears. A 2009 study estimated that large engines covered roughly 40 to 70 percent of the indexable web, and no engine has ever claimed to hold all of it.

Ranking is third, and the hardest. When you search, many thousands of pages might match. Deciding which ten to show first is the part where engines compete and where the details are kept secret. Everything that follows in this story is mostly a story about ranking.

What made AltaVista different?

AltaVista, launched on 15 December 1995 by researchers at Digital Equipment Corporation, was fast, thorough, and supported Boolean queries. It was the first full-text, searchable index of a large part of the web. Its crawler, Scooter, and its backend gave it a speed advantage that people noticed immediately. It reportedly went from about 300,000 visits on its first day to over 80 million a day by 1997.

It was also a lesson in how quickly fortunes turn. Ownership passed from DEC to Compaq, then to CMGI, and eventually to Yahoo, which closed the site in 2013. The reasons for its decline are debated: better rivals arrived, the dot-com crash cancelled a planned stock offering in 1999, and the company changed hands repeatedly. Those are plausible explanations, but no single cause is established.

Why did Google win?

The problem with early full-text engines was relevance. Searching for a common word returned thousands of pages, ordered by how many times the word appeared, which was easy to game by stuffing a page with keywords. A better signal was needed.

The signal was links. A link from one page to another works like a citation, and a page cited by many important pages is probably itself important. Robin Li developed a version of this idea, RankDex, in 1996. Larry Page, Sergey Brin, and Scott Hassan developed PageRank the same year at Stanford, and the patent Page filed in 1998 cites Li’s earlier work. Google, built on PageRank, arrived in the late 1990s, and by the 2000s it had become the dominant engine in most of the world. Li went on to found Baidu in 2000, which is still the leader in China.

Microsoft eventually built its own crawler and launched Bing in June 2009, replacing its earlier Live Search. By May 2025, StatCounter reported Google handling roughly 89 to 90 percent of searches worldwide, with Bing at about 4 percent. Those are measurements of traffic that vary a little by source and by month, so read them as approximate.

PageRank is no longer the whole story, and Google uses many other signals. Its details are not public.

How did search become answers?

For about twenty years the product was a list of ten blue links. The change began with small features, such as direct answers for weather, sports scores, and unit conversions, and then accelerated with large language models.

At its I/O event in May 2023, Google unveiled the Search Generative Experience as a Labs experiment, which produced AI-written summaries above the results. In May 2024 it was renamed AI Overviews and rolled out to US users, reaching the UK, India, Japan, Indonesia, Mexico, and Brazil that August and about 100 more countries on 28 October 2024. In March 2025 Google introduced AI Mode, an experimental view for longer, multi-part questions that accepts text, images, and voice. Other companies, including Microsoft and OpenAI, have built chat-style products with search attached, though this article did not examine them in detail.

How do AI answers get made, and where do they fail?

Most of these systems use a technique called retrieval-augmented generation. The engine first retrieves relevant pieces of text from an index, then hands them to a language model, which writes a response based on that material. The approach lets the model draw on current information without being retrained, and lets the answer point to its sources.

It does not remove the risk of error. The model can still make up details around the retrieved text, take a sentence out of context, blend outdated and current information, or answer with confidence when the material is thin. A citation is also not proof: a linked source shows where the model looked, not that it read it correctly.

Early AI Overviews drew criticism after widely shared mistakes. Google called these “isolated examples,” and about two weeks after launch it limited the feature for some health queries and reduced its use of social media posts. Two other concerns are live debates rather than settled facts. Publishers argue that summaries reduce clicks to their sites. And some analyses suggest AI-assisted queries use much more energy than traditional ones, with a figure of about thirty times attributed to Scientific American. That number is an estimate from a single outlet and should be read that way.

Common questions

What was the first search engine?

Archie, released on 10 September 1990, is generally considered the first. It searched filenames on FTP servers, not web pages, so some histories start with W3Catalog or JumpStation (1993) for the web proper.

What is the difference between a search engine and a web browser?

A browser displays web pages. A search engine is a website or service that helps you find them. Many people use both through the same window, which is why the two get confused.

How does Google decide the order of results?

It uses many signals, including links between pages, and the exact formula is not published. PageRank, which counted links as votes, was the original idea.

Can AI search answers be wrong?

Yes. They can misread sources, merge conflicting information, or state something false with confidence. Checking the cited pages is the practical safeguard.

Do search engines index the entire internet?

No. Studies have put large engines at a fraction of the indexable web, and much content sits behind logins or is blocked from crawlers.

Is there a way to avoid AI Overviews?

Adding “-ai” to a query is one method that has been reported to exclude them, though features and behavior change over time.

What to remember

Search has moved through four stages: finding files by name, finding words in pages, ranking pages by links, and summarizing the results into text. Each stage brought a new weakness: you needed to know the name, keyword stuffing swamped the results, ranking could be gamed, and summaries can be wrong.

The tools change, but one habit stays useful. Treat a result, whether a link or a paragraph, as a lead to check against the original source. The engines are good at finding things and less reliable at telling you whether they are true.

Further reading

Well-known account of how Google and its rivals reshaped search and business. See it on Amazon (affiliate link)

As an Amazon Associate I earn from qualifying purchases. A commission may be earned if you buy through this link, at no extra cost to you.

Sources