feed-survey(1)

NAME

feed-survey - study how the sites RSS Parrot follows name their feeds

SYNOPSIS

feed-survey list db
feed-survey sample db [-n n]
feed-survey fetch db [-hosts n]
feed-survey feeds db [-hosts n]
feed-survey analyze db

DESCRIPTION

feed-survey is a research tool, not part of rss-kakapo(1), and is not built by package.nix. It gathers evidence for the rules by which rss-kakapo(1) reads a web page: which of the feeds a page names it follows, where it finds them, and what it names the feed's account after. Those rules were once set without data; they should be changed with it.

Each step reads and writes the SQLite database db, and can be run again: it carries on where it stopped, or, for analyze, starts over. The steps run in order:

list

Reads RSS Parrot's public listing, https://rss-parrot.net/web/feeds, page by page, a second apart, into the table listing: every feed's handle, feed URL, site URL and title.

sample [-n n]

Picks n sites, 1000 by default, into sample. A few hosts (YouTube, Bluesky, nitter instances) make up much of the listing and all name their feeds alike, so the sites are picked so that their URLs differ the most: farthest-point sampling on SimHash fingerprints of the URLs. The same listing gives the same sample. A sample is drawn once; delete its rows to draw another.

fetch [-hosts n]

Fetches each sampled site's page into fetch: its status, final URL after redirects, headers and first MiB, as much as rss-kakapo(1) reads of a page. A failed request is stored with its error and not retried; delete its row to try again.

feeds [-hosts n]

Fetches, into feed_fetch, the feed that rss-kakapo(1) follows for each page fetched with status 200: the one logic.ReadSiteMeta() picks, resolved against the page as served and with its query trimmed, as rss-kakapo(1) does. The table has the columns of fetch; url is the feed's. Pages that name no feed are skipped, and failures are kept as by fetch.

analyze

Reads every page fetched with status 200 and records, in page, page_link and page_anchor, every '<link>' of the page in order, in its head or after it, the '<a>' elements that look like feeds, and the feed each of four rules picks: rss-kakapo(1)'s own (logic.ReadSiteMeta()), upstream's, the first feed link, and the first on the page's host.

In meta it records, for every such page, the title and description the page gives in its head and those its feed gives, if feeds fetched one, with white space collapsed alike. page_is_item says whether the page's title holds the title of one of the feed's items, of at least 6 characters and other than the feed's own: a page named after a post, as a blog's front page often is after its newest one.

The rest of the study is SQL over these tables.

Run it from the repository root, so that the database lands in tmp/, which git ignores:

go run ./cmd/feed-survey list tmp/feed-survey.db

POLITENESS

Every request carries a 'User-Agent' that names the tool and where its source is, so that a site's owner can find out what asked. fetch and feeds ask each host one request at a time, two seconds apart, and up to n hosts at once, 8 by default.

FILES

tmp/feed-survey.db

Where the database is kept by convention.
The data stays out of the repository: it holds third parties' pages, and
can be fetched again.

EXIT STATUS

0 on success, 1 if a step failed, 2 if it was called wrongly.

SEE ALSO

rss-kakapo(1)

CAVEATS

The pages are the sites as they were when fetched; a site that has changed since is studied as it was.

The listing is of the sites RSS Parrot's users asked for, not of the web, and its site URLs are what upstream recorded: one ending in '/' was taken from a feed rather than given by a user (page.from_feed).