How website checks work

When a LeadHunter user turns on website checks, LeadHunter visits the public website of a business to read contact details that the business publishes itself, such as a contact email, a phone number and links to its social media profiles.

What it does

• It identifies itself with a user agent containing LeadHunterBot.
• It reads robots.txt first and does not fetch pages that robots.txt disallows. It honours Crawl-delay up to 10 seconds.
• It reads at most the homepage and one contact page per website, with at least one second between requests to the same site.
• It never logs in, submits forms, solves CAPTCHAs or bypasses any access control.
• It only follows links on standard web ports and never connects to private network addresses.

How to opt out

Add these lines to the robots.txt file at the root of your website:

User-agent: LeadHunterBot
Disallow: /

LeadHunter re-reads robots.txt at least once an hour, so the change takes effect quickly.