IndexPagesBot

01 What it is

IndexPagesBot is the automated client of the Index Pages indexing pipeline. A customer submits a URL, the pipeline goes to work on that address across search engines and the retrieval layers that answer from them, and part of that work involves requesting the page itself. Those requests carry the user agent in section 05, which is how they end up in your access log.

It is not a search engine crawler. There is no Index Pages index, no Index Pages results page, and nothing here that your site could be ranked in. Every address it touches was handed to us on purpose by someone who wanted that address indexed – it does not discover sites, it does not follow links across the open web, and it does not expand one page into a sweep of your domain.

02 Why the method is closed

What the bot does on a page – in what order, at what cadence, and alongside what else – is a trade secret, and it stays one. It is the product of years of proprietary work, it is the reason the service exists rather than being a shell script anyone could write, and setting it out in public would hand it to every competitor at once. Everyone who touches the pipeline – staff, contractors and infrastructure partners alike – works under a non-disclosure agreement, and the Privacy Policy is deliberately unspecific in exactly one place for the same reason: describing our operational logging in detail would describe the method.

What we can describe is the direction of it, because that is the part that concerns you. Every request the bot makes points at a single outcome: that the submitted page is discovered, indexed, and retrievable – by Google, Bing, Yandex and the rest, and through those indexes by the assistants that answer from them. Everything it does is aimed at a page ranking and being cited.

There is no second agenda underneath that. The bot does not harvest your content for a training set or a dataset, does not price-check you, does not clone you, and does not feed any product that competes with you. If a page of yours is in our queue, someone paid for that page to do better in search.

03 Why there is no IP list

We publish no IP ranges, no ASN, and no reverse-DNS verification scheme, and we do not confirm individual addresses on request. That is a decision we made deliberately, and not a gap we have failed to get round to.

A client that always arrives from the same announced block of data-centre addresses is trivially recognised, and traffic that is trivially recognised is trivially discounted. Work performed from a fixed, published set of data-centre IPs is measurably weaker at reinforcing indexation than work whose origins are diverse. That diversity is part of how the result is produced. Publishing the list would also hand a map of our infrastructure to anyone who asked for one.

There is a consequence for you in that, and you should hear it from us rather than work it out later: because there is no published list, you have no way to verify that a request claiming to be IndexPagesBot is actually ours. A user agent is a string, and any client on the internet can send any string it likes. Treat it as a declaration of intent from us – a courtesy – and not as an authenticated identity.

04 robots.txt

robots.txt is a discovery protocol. It tells an autonomous crawler which parts of a site it may explore and index on its own initiative, and it exists because a search engine, left to itself, would otherwise walk the whole thing.

Our bot does not explore. It is given one specific address by the person who asked for that address to be worked on, and it goes to that address and nowhere else. There is no crawl to budget, no directory tree to be steered away from, and no discovery to permit or forbid – so the file's directives do not describe the thing it is doing, and it does not read them as instructions. We put that here so you do not have to infer it from a log.

In practice the difference is small: a handful of requests to a page whose owner submitted it, not a sweep of your site. And submitting a page you have no right to submit is a violation of the Acceptable Use Policy that we act on – so if the bot is visiting a page you own and nobody on your side asked for it, that is a matter for section 07. A Disallow line will not settle it.

05 Finding it in your logs

Every request the bot makes sends this user agent, unchanged:

Grep your access log for IndexPagesBot and you have every visit. Match on the product token, not the whole string: the name before the slash is stable, while the version after it moves whenever the client changes. A rule pinned to the whole string will stop matching one day and will not tell you.

The address in parentheses is this page. It is there so that whoever finds the string in a log has somewhere to go and someone to write to, which is presumably how you got here.

06 How to block the bot

You cannot block it reliably, and saying so beats publishing a recipe that does not work. There is no address range to deny, the requests do not all originate from one place, and section 04 explains why a Disallow line is not the lever it appears to be. Anything broad enough to stop us dependably would be broad enough to stop a large share of your ordinary human visitors as well, which is a cure considerably worse than the disease.

What you can do, if you would rather not see it at all, is refuse the user agent. Match IndexPagesBot in nginx, Apache, Cloudflare, or whatever sits in front of your origin, and return a 403. That works on every request carrying the string, which is to say on every request that told you the truth about itself. It is a partial measure by construction: it removes the visits that identified themselves, and leaves everything else exactly as it was.

If what you actually want is for a domain to be left alone entirely, a firewall rule is the wrong instrument. Write to abuse@index-pages.com from an address on that domain, name it, and we will exclude it at our end. That covers what a user-agent rule cannot, and it applies across the whole pipeline instead of one edge. It costs you a single email.

07 Contact

Questions about the bot, complaints, and exclusion requests go to abuse@index-pages.com. Security findings go to security@index-pages.com and are welcome there. The remaining desks are on the contact page.

What the service does for the people who pay for it is on the home page; the rules that govern it are in the Terms of Use.