About our crawler · hello@papershelter.com
PaperShelter is a personal newspaper. Our crawler, PaperShelterBot/1.0, visits the public RSS/Atom feeds that our users choose to follow so we can build each reader a personalised digest.
User-Agent: PaperShelterBot/1.0 (+https://papershelter.com/bot)
It reads feed entries (headline, link, short description) and, for ranking and writing AI summaries, briefly fetches article pages. We respect robots.txt and crawl-delay, skip sites that disallow crawling, and back off when a site asks us to slow down.
We keep headlines, links, source names, dates, AI-written summaries and content hashes. Full extracted article text is used only internally to write summaries and is deleted within 7 days. We never republish a publisher's full article text to our readers — we always link back to the original source.
Publishers who would like to be excluded, or who have any concern, can email us at hello@papershelter.com. We handle removal and exclusion requests promptly.