Skip to content
Typestream
Try it free

For publishers

TypestreamBot

Typestream’s crawler reads the news articles publishers list in their feeds, so the people who use Typestream can follow how the news covers the organisations, people and topics they care about.

How to recognise it

Every request carries this user agent:

Mozilla/5.0 (compatible; TypestreamBot/1.0; +https://dev.typestream.nz/crawler)

robots.txt rules for TypestreamBot apply to it, and so do rules for every crawler (User-agent: *) when there are none for it by name.

What it fetches, and why

  • Article pages listed in your RSS or Atom feeds and your Google News sitemap, and article links to your site in GDELT’s public news feed. It doesn’t crawl the rest of your site.
  • Each page once, when it first appears. It fetches the HTML only: no images, scripts or stylesheets, nothing larger than 5 MB, and it never signs in or gets around a paywall.
  • From each article, Typestream keeps its headline, byline, dates and text, to answer its users’ questions about coverage. People using Typestream see a headline, a short snippet and a link back to you, never the full text. If a page asks for no snippets (nosnippet), they see none.

robots.txt and pace

  • It reads your robots.txt before fetching anything, keeps it for up to 24 hours, and never fetches a path it disallows.
  • It waits at least 2 seconds between requests to one site, and longer if your robots.txt sets a Crawl-delay.
  • It backs off and tries later when your site answers 429 or a server error, and gives up on other errors.

How to opt out

To stop it fetching anything from your site, add this to your robots.txt:

User-agent: TypestreamBot
Disallow: /

To keep it out of part of your site, or slow it down:

User-agent: TypestreamBot
Disallow: /members/
Crawl-delay: 10

It picks up changes the next time it reads your robots.txt, within 24 hours. Articles it already has stay in Typestream until you ask us to remove them.

Have your articles removed

Ask here and a person reviews it. Once approved, the site or page stops being fetched and shown in Typestream, and the text we stored is deleted.

What to remove

The domain, such as example.co.nz

So we can tell you when it’s done. Use an address at the domain if you can.

Contact

Write to crawler@typestream.nz about the crawler: to report a problem or ask us to slow down. Include your domain, and a request’s time and URL if you’re reporting one.