This evergreen history article uses authoritative archives and official records. Exact dates are used when documented; gradual inventions and rollouts are described as periods rather than being assigned a misleading single birthday.
Quick facts
- Archie, created at McGill University in Montreal in 1990, indexed file names on FTP servers rather than full web pages.
- Early Web search mixed automated crawlers with human-edited directories.
- WebCrawler, Lycos, AltaVista, Yahoo and others shaped the commercial search market before Google.
- Google’s link analysis improved relevance but did not make it the first search engine.
- Modern search combines crawling, indexing, ranking, structured data, machine learning and increasingly generative AI.
Before the Web: finding files and people
As networked archives grew, users needed ways to locate files. Archie, developed by Alan Emtage and colleagues at McGill University, created a searchable database of file names from public FTP servers. Gopher search tools such as Veronica and Jughead followed.
These systems did not search the modern Web, but they established a basic pattern: collect information from remote servers, build an index and let users submit queries.
The first Web indexes and directories
The Web’s rapid growth in the early 1990s created a new challenge. The World Wide Web Wanderer measured pages and produced the Wandex index. Aliweb allowed site owners to submit descriptions. JumpStation combined crawling, indexing and search in a recognisable Web-search model.
Human-curated directories also became important. Jerry Yang and David Filo began “Jerry and David’s Guide to the World Wide Web” at Stanford in 1994; it became Yahoo. Editors organised sites into categories, which worked while the Web was relatively small but could not scale indefinitely.
Commercial search competition
WebCrawler introduced full-text search across pages in 1994. Lycos built a large index and portal. Excite applied statistical analysis. AltaVista, launched by Digital Equipment Corporation in 1995, was admired for speed, advanced queries and a large index. Ask Jeeves encouraged natural-language questions.
Search companies often became portals filled with news, email, shopping and advertising. The strategy increased time on site but sometimes reduced the simplicity of search itself.
Google changes ranking
Google entered the market in the late 1990s with a sparse interface and a ranking method that examined links between pages. PageRank treated some links as signals of authority, similar to citations in academic work. This helped improve relevance for many queries.
Google combined link analysis with text, anchor words, freshness, location and many later signals. Its advertising system connected commercial messages to user intent, making search highly profitable.
Search engine optimisation emerges
Website owners learned that visibility in search results could determine traffic and revenue. Search engine optimisation developed around technical accessibility, useful content, links and query relevance. Manipulative tactics also appeared, including hidden text, link schemes and automatically generated pages.
Search companies responded with anti-spam systems and quality guidelines. SEO gradually expanded beyond keywords to include mobile usability, speed, structured data, reputation, user needs and content originality.
Bing, vertical search and mobile
Microsoft launched Bing in 2009, replacing earlier Microsoft search brands. Competition also came from specialist services: Amazon for products, YouTube for video, maps for local discovery, app stores for software and social platforms for trends.
Smartphones changed query context. Search engines used location, voice, camera input and personal history. Results pages added direct answers, maps, shopping, news, images, videos and knowledge panels, reducing the need to click a traditional blue link for some searches.
AI search and the answer-engine model
Machine learning has been part of ranking for years, but large language models introduced a new presentation layer: systems can synthesise answers, converse with users and generate summaries. This creates opportunities for complex research and follow-up questions, while also raising concerns about factual errors, source attribution, publisher traffic, bias and copyrighted material.
The central task remains information retrieval: discover reliable material, understand the query and provide useful results. Generative systems do not eliminate the need for indexes, source quality and verification.
What makes a search engine work
A general search engine normally performs several stages. Crawlers discover URLs and follow links. Indexing systems analyse text, media, language and page structure. Ranking systems evaluate relevance and quality. Serving systems assemble results under strict latency limits. Spam and safety systems filter manipulation, malware and harmful material.
No public description reveals every ranking factor because systems change continuously and complete disclosure would invite abuse.
Common misconceptions
- Google was not the first search engine.
- Search engines do not search the live Internet from scratch for every query; they primarily search previously built indexes.
- Ranking is not based on one factor or a fixed list that never changes.
- AI-generated answers still depend on retrieval, training data and sources, and they can be wrong.
Timeline: key years and locations
| Year | Location | Milestone | Why it mattered |
|---|---|---|---|
| 1990 | McGill University, Montreal, Canada | Archie created | Makes distributed FTP file names searchable. |
| 1993 | Europe and United Kingdom | Early Web crawlers and indexes appear | Begins automated discovery of web pages. |
| 1994 | United States | Yahoo directory, WebCrawler and Lycos emerge | Combines human organisation with full-text search. |
| 1995 | Palo Alto, California, USA | AltaVista launches | Demonstrates fast search across a large index. |
| 1996–1998 | Stanford University, California, USA | BackRub evolves into Google | Introduces influential link-based ranking. |
| 2000 | United States | Search advertising scales | Makes query intent a major advertising market. |
| 2009 | Redmond, Washington, USA | Microsoft launches Bing | Creates a long-term global competitor. |
| 2010s | Worldwide | Mobile, voice and knowledge panels expand | Changes search from links alone to contextual answers. |
| 2020s | Worldwide | Generative AI enters search interfaces | Adds conversational synthesis while increasing verification challenges. |