Cloudflare and OpenAI say an experiment that feeds signals from Cloudflare’s network to OpenAI’s search crawler has helped OpenAI discover and index “hundreds of millions of new web pages.” The two companies describe these as early results from a research pilot, and the figures are their own; no outside measurement has been published.
The results were set out on 15 October 2026 in a post on Cloudflare’s blog, written by Cloudflare’s Matthew Conroy and Jack Galilee together with Henry Aspegren and James Balfour of OpenAI.
What the pilot is
Cloudflare runs a network that sits between many websites and their visitors. The companies say they launched a research pilot in July 2026 to find out “whether signals from Cloudflare’s network could help ChatGPT users discover new content more efficiently.”
In practice, Cloudflare’s network provides signals that help OpenAI’s search crawler find pages it did not know about. OpenAI “then independently evaluates and ranks how those pages appear based on how relevant and helpful they are,” the post says. A crawler is the automated program a search service uses to fetch and read web pages.
The pilot “used a participating cohort of ~1% of Cloudflare’s total network,” according to the companies, and in that time “has enabled OpenAI to discover and index hundreds of millions of new web pages.” They describe the answer to their research question so far as “a resounding yes.”
The post does not say which sites were in that 1% cohort, how they were selected, or how many of the new pages have since appeared in ChatGPT answers.
Why it matters for websites
The companies argue that discoverability is changing as people turn to AI tools for answers. Useful pages, such as an appliance manual, current passport rules or a restaurant’s latest menu, “can be valuable even if few other sites link to them,” they write.
ChatGPT Search links back to its sources, which the companies say gives people a way to reach the original sites and publishers. The post also ties the work to OpenAI’s wider push, saying that with GPT-6 and Intelligent UI, OpenAI is making ChatGPT a better place “to shop, learn, and find answers.”
Cutting wasted crawling
A big part of the pilot is about avoiding pointless visits. The companies say they learned “how easily automated traffic can make a nonexistent page look interesting”: bots repeatedly request files that do not exist, some sites redirect almost any address to their homepage, and other addresses lead to copies of the same content. Spotting those patterns lets the crawler skip pages before fetching them.
They are also using “change signals” to tell the crawler when a page is worth another visit and when it is not. “For a website owner, a crawler that repeatedly downloads an unchanged page adds load without bringing any benefit,” the post says.
The companies acknowledge a trade-off: “Finding new pages can initially increase crawling as those pages are evaluated and indexed.” They have not published figures on how much crawling has gone up or down for participating sites.
Flagging harmful pages
Cloudflare and OpenAI say they are also testing signals that flag potentially harmful pages. Their example is a legitimate site with a single compromised page, such as a fake login form planted by an attacker. Flagging only that page, rather than the whole domain, “keeps people safe without penalizing the site owner,” they write. Early results suggest the signals can help identify pages that should be kept out of an index, according to the post.
Control and training data
The post stresses that site owners stay in charge. “A signal about a page doesn’t grant permission to access it,” the companies write. “This research uses metadata about the web for search and discovery, rather than sharing page content for model training.” Site owners can use Cloudflare’s AI Crawl Control to see and manage how AI systems access their sites.
What’s still unknown
Beyond the headline number, the companies have not released data on how many discovered pages were useful, whether participating sites saw more visitors, or how the pilot affected server load. All of the results so far are company claims.
What to watch
The companies say the next phase will improve the selection of pages, test “faster and more reliable change signals,” and measure what happens after content is discovered, including whether better discovery “can translate into real business outcomes.” Watch for whether the pilot expands beyond 1% of Cloudflare’s network and whether site owners get an explicit opt-in or opt-out.
What this means
- Early, self-reported results. The “hundreds of millions” figure comes from the two companies.
- Metadata, not content. The companies say page content is not shared for training.
- Site owners keep control. Access is still governed by owners’ settings, including AI Crawl Control.
Sources
- Matthew Conroy, Jack Galilee, Henry Aspegren (OpenAI) and James Balfour (OpenAI), “Cloudflare and OpenAI: making AI discovery better for everyone”, Cloudflare blog, 15 October 2026. https://blog.cloudflare.com/cloudflare-openai-discovery-agentic-web/
Related stories
- Profound Launches Dynamic Bot Rendering, Which Serves AI Crawlers a Fully Rendered Copy of Web Pages
- AI Agents Are Learning to Spend Money: The Payments Infrastructure Race
- The Claude Code Leak Revealed Coinbase x402 Payment Protocol And Nobody Noticed
Follow @tsnmediahub on X for breaking AI news as it happens.
Follow on X
