RehlProspectBot

REHL is software for real estate brokerages and agents, operated by Rehl, Inc., a Delaware C-Corporation with operations in Florida. RehlProspectBot is our crawler. It visits the public websites of real estate brokerages we are considering contacting about our product.

It is not a search engine crawler. We build no index and we cache nothing for the public. The one thing we ever copy is your logo, and only when we are building a preview for you; that is described in full below.

RehlProspectBot/1.0 (+https://rehl.us/bot; sebastian@rehl.us)

The token before the slash, RehlProspectBot, is the name to use in robots.txt. We send this exact string on every request, we never send a browser user agent, and we never retry under a different one to get past something that refused us.

If your logs show something else from us, we want to know. Write to sebastian@rehl.us with the line from your log. This page is meant to account for every request we make, and a request it does not account for is a bug on our side, not a detail.

Three passes, all small and all bounded:

• A site check. Your homepage, plus at most four pages your homepage links to. It is looking for one thing: whether your site has a working property search. That is the subject of what we would write to you about, and we would rather check than guess.
• A contact-details read. At most six pages in total, normally the homepage and a contact or about page. It reads business email addresses your site publishes, and your privacy and terms pages to check whether you have asked people not to do that.
• A logo read. Your homepage, and then one image file it points at, if we are putting together a preview of what your own branded site would look like on REHL. This is the only thing we fetch that we keep a copy of, and it is covered in What we keep below. The image is very often on a different host from your homepage, so we read that host’s robots.txt too and take its answer for that file.

No pass logs in, submits a form, solves a challenge, or touches anything that is not linked from your public pages. The first two do not revisit a site inside six hours of their own last attempt, so the most either can do in a day is four visits, and in practice a given site sees far fewer than that. There is no continuous crawl and nothing runs in a loop against your server.

• The url we ended up at, and the HTTP status you returned.
• Whether a property-search vendor’s script appeared in your markup, and whether your navigation points at listings.
• Business email addresses your own site publishes, such as an office or contact address.
• What your robots.txt said, and whether we were allowed to proceed.
• A copy of one image file: your logo, and only when we have built a preview site for your firm. It is stored on our file host and shown on that preview. It is the only part of your site we hold a copy of. If you would rather we did not, write to us at the address below and we will remove it and use a placeholder.

We do not keep copies of your pages, we do not collect personal information about your site’s visitors, and we set no cookies on your site because we run no browser.

One group in your robots.txt at https://yoursite.com/robots.txt. We read it before we request any page, and we treat it as binding:

User-agent: RehlProspectBot
Disallow: /

That is the whole thing. You do not have to write to us, and we do not ask you to verify anything. We also honor a Disallow aimed at *, even though most crawlers would read a rule for everyone as not applying to them once you have named a group for someone else.

A few more specifics, because “we honor robots.txt” is a sentence anybody can write:

• Crawl-delay is honored before every request, up to 30 seconds. Above 30 seconds we skip your site rather than sit on your server.
• nofollow, in a robots meta tag or an X-Robots-Tag header, stops us following any link on that page.
• If we cannot read your robots.txt (it errors, times out, or comes back as something that is not a plain-text file), we treat that as a no and skip your site on that pass. Not having a robots.txt at all is fine and means what the standard says it means.
• Blocking us at your firewall works too. We do not route around it.

Write to sebastian@rehl.us and say which domain. We will delete what we hold about that firm and stop visiting, and you will get a reply from a person, not a ticket robot. No form, no account. That includes taking down a copy of your logo and the preview it appears on, if we made one.

Same address for anything else: a rule you think we got wrong, a request that we slow down, or a complaint. If our crawler is behaving in a way this page does not describe, that is a bug and we want to hear about it.

Blocking the crawler stops the crawler. If you also want no email from us, say so at sebastian@rehl.us and that is permanent, whatever your website does or does not say. You can also reply to any message you have had from us. See our privacy policy for the rest.