RehlProspectBot

REHL is software for real estate brokerages and agents, operated by Rehl, Inc., a Delaware C-Corporation with operations in Florida. RehlProspectBot is our crawler. It visits the public websites of real estate brokerages we are considering contacting about our product.

It is not a search engine crawler. We build no index, we cache nothing for the public, and we republish no part of your site anywhere.

RehlProspectBot/1.0 (+https://rehl.us/bot; sebastian@rehl.us)

The token before the slash, RehlProspectBot, is the name to use in robots.txt. Both passes described below send this exact string on every request, we never send a browser user agent, and we never retry under a different one to get past something that refused us.

If your logs show something else from us, we want to know. Write to sebastian@rehl.us with the line from your log. This page is meant to account for every request we make, and a request it does not account for is a bug on our side, not a detail.

Two passes, both small and both bounded:

• A site check. Your homepage, plus at most four pages your homepage links to. It is looking for one thing: whether your site has a working property search. That is the subject of what we would write to you about, and we would rather check than guess.
• A contact-details read. At most six pages in total, normally the homepage and a contact or about page. It reads business email addresses your site publishes, and your privacy and terms pages to check whether you have asked people not to do that.

Neither pass logs in, submits a form, solves a challenge, or touches anything that is not linked from your public pages. Neither one revisits a site inside six hours of its own last attempt, so the most either can do in a day is four visits, and in practice a given site sees far fewer than that. There is no continuous crawl and nothing runs in a loop against your server.

• The url we ended up at, and the HTTP status you returned.
• Whether a property-search vendor’s script appeared in your markup, and whether your navigation points at listings.
• Business email addresses your own site publishes, such as an office or contact address.
• What your robots.txt said, and whether we were allowed to proceed.

We do not keep copies of your pages, we do not collect personal information about your site’s visitors, and we set no cookies on your site because we run no browser.

One group in your robots.txt at https://yoursite.com/robots.txt. We read it before we request any page, and we treat it as binding:

User-agent: RehlProspectBot
Disallow: /

That is the whole thing. You do not have to write to us, and we do not ask you to verify anything. We also honour a Disallow aimed at *, even though most crawlers would read a rule for everyone as not applying to them once you have named a group for someone else.

A few more specifics, because “we honour robots.txt” is a sentence anybody can write:

• Crawl-delay is honoured before every request, up to 30 seconds. Above 30 seconds we skip your site rather than sit on your server.
• nofollow, in a robots meta tag or an X-Robots-Tag header, stops us following any link on that page.
• If we cannot read your robots.txt (it errors, times out, or comes back as something that is not a plain-text file), we treat that as a no and skip your site on that pass. Not having a robots.txt at all is fine and means what the standard says it means.
• Blocking us at your firewall works too. We do not route around it.

Write to sebastian@rehl.us and say which domain. We will delete what we hold about that firm and stop visiting, and you will get a reply from a person, not a ticket robot. No form, no account.

Same address for anything else: a rule you think we got wrong, a request that we slow down, or a complaint. If our crawler is behaving in a way this page does not describe, that is a bug and we want to hear about it.

Blocking the crawler stops the crawler. If you also want no email from us, say so at sebastian@rehl.us and that is permanent, whatever your website does or does not say. You can also reply to any message you have had from us. See our privacy policy for the rest.