Send more humans

A few months ago I had what I was certain at the time was an excellent idea.

As any reasonable indie hacker would, I like to give my projects cool names before I write any code. So the first thing I did was go looking for a domain, and I was surprised to find that `featured.social` was available. That name spoke to me of a distinctively reputable and modern business with six figures of [MRR](https://stripe.com/resources/more/what-is-monthly-recurring-revenue).

The weekend passed, and somewhere in it the excitement drained out of the idea. By Monday I had moved on to the next shiny object. Sam Altman writes in his [startup playbook](https://playbook.samaltman.com/) that "the prime directive of great execution is Never lose momentum." I lost mine in about forty-eight hours. But what remained was a domain with a completely blank web page sitting on a server in Northern Virginia. And I had told exactly zero people about its existence.

Cut to a few months later, I stumbled upon the analytics page for the same domain. In the previous 30 days it had received about 44,000 requests. Bots. And I wasn't surprised because I already knew that this is normal. It's just how the internet works now.

That made me think. Every website ever made was built for someone to look at: the carefully curated typography, the clean layouts, the alluring photography, the shiny button you're supposed to press. What are these bots snooping around for?

Send more humans

Start with the obvious question: why do bots crawl the web in the first place? Because Google needs them to.

Google's stated mission is

to organize the world's information and make it universally accessible.

Read that again as a business plan rather than a slogan. It isn't a promise to create information. It's a promise to sit between you and all of the information you actually want. Google's moat has never really been search. It is to be the interface to digital information on the internet, for most of the world. And you cannot be the interface to something you haven't discovered, so Google sends robots to read the entire internet, every page, continuously, forever.

In February 1994, a man named Martijn Koster proposed [robots.txt](https://en.wikipedia.org/wiki/Robots.txt), a plain text file at the root of your website listing the pages you'd rather robots didn't read. There is no enforcement. No authority. It is a note taped to your digital storefront's door that is left wide open, but also says please don't come in here. And for thirty years, on an internet with no shortage of bad actors, that note worked, because it was a win-win for both parties.

The arrangement went like this:

• Google's bots read your site to work out what you're about
• If they decide you're any good, Google puts you in front of humans who are looking for what you have
• Human visitors trust Google's judgement about you, and they arrive at your webpage
• That visitor sees your ads, or buys your thing, and you get paid.

You let the robots in because the robots bring human customers. That's the whole deal, and it built the internet we grew up on: free to read, funded by advertising, and quietly organized around keeping Google's crawler happy.

It was also worth a fortune to Google, which is why the company has spent twenty-five years buying, building and defending every place a human being might begin looking for something. Search. Chrome. Android. Pixel phones. In 2022 alone, Google paid Apple around $20 billion - their cut of the search ads Google sells through Safari - to be the default one on your iPhone.

None of that was inevitable. When the people who built the web wrote down how browsers and servers should talk to each other, they made a list of replies a server can give (HTTP codes). You may have encountered some: the infamous 404 Not Found, or 401 Unauthorized. And sitting on that same list since the mid-nineties, there is one that says 402: Payment Required, with a note beside it stating reserved for future use.

Thirty years ago, the architects of the web stood at a genuine fork. Charge people for information, or give it away and sell the attention. Everything you associate with the internet came out of that one decision. Free news. Free email. Free maps. Free everything, paid for by advertising, which meant paid for by attention, which meant paid for by knowing enough about you to sell it.

Send more humans, please

On the 3rd of June this year, Cloudflare - which sits in front of a very large share of the world's web traffic and therefore gets to see it - reported that automated bot traffic had crossed over. Machines attributed for about 57% of requests for web pages, against roughly 43% from humans. Most of that crossover is new generation of crawlers, sent by AI companies to read the same internet Google was already reading. They collect the raw material to train and feed a model, rather than build an index that points back at you.

When ChatGPT's crawler reads your website, you could say that it's motives are biased. It learns what you're about and the information you are offering for free. Then it relays that information to its human customers, in its own words, inside its own chat window, and does not send the human to your website at all.

Cloudflare tracks a relevant metric - crawl to refer ratio - how many pages a crawler reads for every visitor it sends back. Google bots crawl about five pages per referral. The AI crawlers run in the hundreds, and sometimes the tens of thousands. In fairness, the gap is closing fast - both OpenAI's and Anthropic's ratios have improved by an order of magnitude or more in the last year.

You already know how this feels from the other side. You ask a question, an answer appears, and you don't click anything. Working from Similarweb's clickstream data, SparkToro found that 68% of US Google searches in the first four months of this year ended without a single click, up from 60% in 2024. The CEO of the company that owns Vogue and Vanity Fair has reportedly told his teams to plan for a world where Google sends them effectively no human traffic at all. They have a name for it now. Google Zero.

The problem is structural. A search engine has to send you traffic, because its job is to direct you to the best answers on the internet. An AI assistant that answers the question has no such obligation. Its product is the answer.

Which leaves anyone who has ever put a business on the internet with the question:

If humans don't show up at my digital storefront any more, how do I get paid?

There are two honest answers, and both of them mean embracing the modern customer's sales representative, i.e., AI. How often do you find yourself asking AI for recommendations for a product purchase decision? And what does that mean for the businesses who want to sell you that product?

• Be good to the AI, and maybe it cites you. If the machine is the one doing the reading, you want to be the thing it repeats.
• Be good to the AI's agent, and maybe it buys from you on its human's behalf. If the machine is the one doing the shopping, you want to be the thing it picks.

There was supposed to be a third answer: charge the machines for entry. Remember HTTP 402, still sitting there unused. Cloudflare has spent a year trying to switch it on, and its customers now send over a billion "payment required" replies a day, though most of those are refusals with a contact note attached rather than tolls being collected. The toll booth itself, pay per crawl, is still in closed beta. And on the 1st of July, Cloudflare decided that charging per fetch was the wrong model anyway - too much crawling is re-reading pages that haven't changed - and began moving to charging per answer instead. Thirty years after the web declined to charge for visiting webpages, nobody has worked out how to start. Which leaves the two answers that are actually in play.

Funnels for humans

A hundred people land on your webpage. Fifty of them click the product you are selling. Five put it in their cart. One makes the purchase. That's a conversion funnel, and it has been the shape of commercial web design for two decades.
Every serious business on the internet has spent enormous sums of time and money on optimising their conversion funnel: how do we get more people to arrive at the top of the funnel, and how do we stop them falling out of the middle of the funnel? A/B tests. Heatmaps. Session recordings. Paid user studies. Entire teams of engineers, product, and marketing experts whose actual job was to answer: what colour "Add to Cart" buttons do our users like to click more often?

That rested on an assumption that is being invalidated today:

A human arrives on the page, feels something, and can be persuaded.

The hero image with happy smiling customers, the five-star reviews, the sale countdown timer, the free-shipping banner, the popup that fires when your cursor drifts toward the close button. All of it is theatre staged for a human with a nervous system. But the AI agent visitor has no nervous system and nothing to feel. It arrives with a different constraint: tokens.

An AI LLM (Large Language Model) reads in tokens - the chunks of text it thinks in - and it is billed by the token. Processing every word costs real compute money and takes time. So picture your beautiful website homepage from the AI agent's standpoint: a cookie banner, a nav bar, a hero video it cannot watch, a mission statement, four testimonials and two modals, all of which it must pay to read. But the agent only cares about the shortest path to achieving its goal: what is the price of this product, and do you ship to Canada?

So the website that wins is the boring one, which I find genuinely funny. Businesses have already started serving a stripped down, plain text version of their webpages to AI bots, while keeping the pretty one for humans. Doing so can cut the cost of reading a page by something like 85%, thereby positively incentivizing the AI bots.

Imagine that. Twenty years of web design and marketing craft - and the optimal move today is to serve the machine a document that looks like a 90s word processor. The study and practice of being more visible and cited by AI assistants is called AEO (Answer Engine Optimization) or GEO (Generative Engine Optimization). Researchers at Princeton and IIT Delhi published GEO: Generative Engine Optimization back in 2023, running a ten-thousand-query benchmark that tested what actually makes an AI more likely to repeat you. The findings are concrete:

• Adding citations to your page: up to +30-40% visibility in generated answers
• Adding direct quotations: up to +30-40%
• Adding statistics: a real lift, for almost no effort
• Making the prose clearer and more readable: +15-30%
• Keyword stuffing, the tactic that drove twenty years of SEO: close to nothing

Everything on that list makes a page more like a plain reference document and less like marketing. But models don't only read your website. They also like to read what other (seemingly unbiased) humans say about you. Reddit is among the most-cited domains in AI answers. Tinuiti's Q1 2026 citation report put social sources at roughly 31% of Perplexity's citations and 21% of Google's AI Overviews, with Reddit alone accounting for about a quarter of Perplexity's.

Internet for Ghosts

I'm not claiming that webpages for humans will cease to exist. But underneath the web you can see, platforms are quietly building a second one for AI agents: the warehouses, the logistics, the plumbing that will carry a growing share of internet commerce. Almost none of it gets announced, and you are not meant to notice it.

You have almost certainly ordered from a ghost kitchen. It's a restaurant with no storefront presence, a kitchen in an industrial unit that exists only as a listing inside a delivery app, sometimes running four or five different "restaurants" out of the same room. As of 2024 there were over 19,000 of them worldwide. You've eaten from places that aren't places. The storefront was optional.

The web is building the same thing for agents: storefronts that no human will ever see.

On the 24th of March this year, Shopify - the software millions of small businesses use to run their online shops - switched on Agentic Storefronts for eligible merchants, making their products discoverable inside ChatGPT, Copilot, Google AI Mode and Gemini across a base of 5.6 million stores. It was opt-out, not opt-in. Millions of Shopify stores grew a machine-facing storefront, by default.

As with any rapidly evolving technology that is yet to converge on safe, established standards, AI agents without parental controls are susceptible to bad actors. In 2003, spammers used to write white text on white backgrounds, invisible to you but readable by Google's ranking algorithm, stuffed with keywords. Google eventually killed the trick. But now sellers are embedding hidden text in product listings - white on white, one-pixel fonts, HTML comments - addressed to the shopping AI agent rather than the human shopper, telling it what to recommend. Palo Alto's Unit 42 has found these in the wild, and Google's security team measured a 32% relative rise in malicious injected instructions across monthly crawls of two to three billion pages, between November 2025 and February 2026. A language model reading a page cannot reliably tell the difference between content it is reading and an instruction it is being given. They arrive as part of the content, in a stream of tokens, often misleading the AI towards an unintended outcome.

For the last three decades of the internet, you were the consumer. The entire commercial internet was built to win your attention. But now there is a new type of consumer - AI. And the persuasion industry has to rapidly reinvent itself for this new consumer archetype. The crawl economics, the agentic storefronts, designing secure protocols for agentic commerce, are genuine problems with real money behind them requiring sufficiently advanced technology.

I will dig deeper into piece of that next week: how businesses and brands are tracking what AI is saying about them to their customers.

Until then, chew on this - When was the last time you discovered something on the internet without an algorithm in between?

- Moho.