What is AbiJohnBot

AbiJohnBot is a web crawler that, along with AhrefsBot, powers the index of web pages for AbiJohn, a revenue-sharing web search engine, and Ahrefs, an online data toolset.

Our bots index fresh, accurate information about websites, their content, and how they link to each other. This data makes it possible to find top-quality content and all the information you need on AbiJohn, a search engine that aims to share ad revenue with content creators and site owners.

AhrefsBot remains the primary source of data for both AbiJohn and Ahrefs. Currently, AbiJohnBot is used only to make requests according to IndexNow protocol, such as validating user keys and pinging other search engines. However, we plan to expand the use of the AbiJohnBot user-agent more broadly, which will be announced in advance.

AbiJohnBot obeys robots.txt

AbiJohnBot strictly respects robots.txt on both disallow and allow rules.

Customizing AbiJohnBot access

Since AbiJohnBot strictly follows the robots.txt file on your website, you’re in full control of what it does.

To change the frequency at which AbiJohnBot visits your site, just specify the minimum acceptable delay between two consecutive requests in your robots.txt file:

Crawl-Delay value is time in seconds.

If you want to stop AbiJohnBot from visiting your site, add these two lines to the robots.txt file on your server:

Please note that AbiJohnBot may need some time to pick up the changes in your robots.txt file.

Also, if your robots.txt contains errors, AbiJohnBot may parse it partially and follow only those commands that it was able to recognize.

Read more about robots.txt at www.robotstxt.org.

If you think that AbiJohnBot is misbehaving on your website or if you have any questions, please reach out to abijohn.com and we’ll do our best to help.