With the advent of LLMs, the oldest frontiers on the world wide web are once again in dispute, with upstarts competing for your browser homepage real estate. Companies like Perplexity, Arc and Exa are using LLMs to answer questions directly, replacing Google’s time-tested 10 blue links - and people in Mountain View are freaking out.
The idea behind LLM-driven search is quite simple. The vast majority of web searches are conducted in order to answer a question. In the not-so-distant past, answering a question required a multi-step process:
Frame question → phrase question → type question into Google → process search results & metadata → click most relevant result → read result → return to search page → click another results → … → discover answer
Voila! You now have an answer to your question.
You’ve likely become quite adept at completing the process above, to the point where it’s almost second nature to you. However, looking at the multitude of steps involved, it’s clear that this is a mentally taxing chore.
Luckily, it’s 2024, and we have LLMs now. People realized pretty early on that these LLMs could, in theory, read web pages for us, and summarize them in a very concise way. It took a while for the technology to get to a point where it could enable this vision, but it has, and today’s LLM-driven search makes the process much faster:
Frame question → type question into NSE → read → know answer
NSEs reduce a lot of the mental load that searching required in the past, while providing better results. The difference is particularly stark with more complicated searches, those that require research (or clicking on several search result links). AI is just better at parsing a whole lot of content and extracting the most relevant bits. Search will become LLM-driven, changing the internet as we know it.
For the past 20 years, Google has been the irrefutable steward of the internet. Not a benevolent ruler, but certainly a gardener, in the sense that they imposed order in a way which benefited many of the internet’s denizens. If you had a website, you could submit it to Google’s crawlers, who in return would direct search traffic your way. The relationship was mutually beneficial: you gained traffic for your site (which you then presumably monetized through ads or product sales), and Google got better search results, which attracted more users, which Google could monetize via ads.
In essence, Google had the traffic and websites had the content; each needed the other to thrive. This status quo lasted for more than two decades, and while Google received a lot of backlash for a wide range of downright rude business practices, they ultimately enabled growth for the web at large.
What happens when you stop sending traffic to websites?
Look at Perplexity, for example. Here’s what a search result looks like on February 7th, 2024:
I searched for “short story sci-fi writers”. Perplexity parsed this statement and extracted a question: “who are some good short story authors, specializing in sci-fi, that I should read?” It then set about answer that question. To do so, it crawled the web, found several sources relevant to my query (likely based on keyword search), fed them to an LLM and wrote an answer. If I need any follow-up questions answered, I can just type them into the search field and get the answer amended.
These results are so good that Perplexity keeps a separate tab on its site for you to see other people’s searches.
In the results, the sources Perplexity used to come up with the answer are provided as a curtesy - but I think you’ll agree with the following assumption:
99% of NSE searches will end on the search result page, without directing any traffic to the model’s sources.
This breaks the fundamental reciprocal relationship between search and websites I’ve outlined above. NSEs enjoy the content produced by websites, but do not provide traffic in return.