1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151

.Dd October 30, 2026
.Dt FEED-SURVEY 1
.Os
.Sh NAME
.Nm feed-survey
.Nd study how the sites RSS Parrot follows name their feeds
.Sh SYNOPSIS
.Nm
.Cm list
.Ar db
.Nm
.Cm sample
.Ar db
.Op Fl n Ar n
.Nm
.Cm fetch
.Ar db
.Op Fl hosts Ar n
.Nm
.Cm feeds
.Ar db
.Op Fl hosts Ar n
.Nm
.Cm analyze
.Ar db
.Sh DESCRIPTION
.Nm
is a research tool, not part of
.Xr rss-kakapo 1 ,
and is not built by
.Pa package.nix .
It gathers evidence for the rules by which
.Xr rss-kakapo 1
reads a web page: which of the feeds a page names it follows, where it
finds them, and what it names the feed's account after.
Those rules were once set without data; they should be changed with it.
.Pp
Each step reads and writes the SQLite database
.Ar db ,
and can be run again: it carries on where it stopped, or, for
.Cm analyze ,
starts over.
The steps run in order:
.Bl -tag -width Ds
.It Cm list
Reads RSS Parrot's public listing,
.Lk https://rss-parrot.net/web/feeds ,
page by page, a second apart, into the table
.Sy listing :
every feed's handle, feed URL, site URL and title.
.It Cm sample Op Fl n Ar n
Picks
.Ar n
sites, 1000 by default, into
.Sy sample .
A few hosts (YouTube, Bluesky, nitter instances) make up much of the
listing and all name their feeds alike, so the sites are picked so that
their URLs differ the most: farthest-point sampling on SimHash
fingerprints of the URLs.
The same listing gives the same sample.
A sample is drawn once; delete its rows to draw another.
.It Cm fetch Op Fl hosts Ar n
Fetches each sampled site's page into
.Sy fetch :
its status, final URL after redirects, headers and first MiB, as much as
.Xr rss-kakapo 1
reads of a page.
A failed request is stored with its error and not retried; delete its row
to try again.
.It Cm feeds Op Fl hosts Ar n
Fetches, into
.Sy feed_fetch ,
the feed that
.Xr rss-kakapo 1
follows for each page fetched with status 200: the one
.Fn logic.ReadSiteMeta
picks, resolved against the page as served and with its query trimmed, as
.Xr rss-kakapo 1
does.
The table has the columns of
.Sy fetch ;
.Sy url
is the feed's.
Pages that name no feed are skipped, and failures are kept as by
.Cm fetch .
.It Cm analyze
Reads every page fetched with status 200 and records, in
.Sy page ,
.Sy page_link
and
.Sy page_anchor ,
every
.Ql <link>
of the page in order, in its head or after it, the
.Ql <a>
elements that look like feeds, and the feed each of four rules picks:
.Xr rss-kakapo 1 Ns 's own
.Pq Fn logic.ReadSiteMeta ,
upstream's, the first feed link, and the first on the page's host.
.Pp
In
.Sy meta
it records, for every such page, the title and description the page
gives in its head and those its feed gives, if
.Cm feeds
fetched one, with white space collapsed alike.
.Sy page_is_item
says whether the page's title holds the title of one of the feed's items,
of at least 6 characters and other than the feed's own: a page named
after a post, as a blog's front page often is after its newest one.
.Pp
The rest of the study is SQL over these tables.
.El
.Pp
Run it from the repository root, so that the database lands in
.Pa tmp/ ,
which git ignores:
.Bd -literal -offset indent
go run ./cmd/feed-survey list tmp/feed-survey.db
.Ed
.Sh POLITENESS
Every request carries a
.Ql User-Agent
that names the tool and where its source is, so that a site's owner can
find out what asked.
.Cm fetch
and
.Cm feeds
ask each host one request at a time, two seconds apart, and up to
.Ar n
hosts at once, 8 by default.
.Sh FILES
.Bl -tag -width Ds
.It Pa tmp/feed-survey.db
Where the database is kept by convention.
The data stays out of the repository: it holds third parties' pages, and
can be fetched again.
.El
.Sh EXIT STATUS
0 on success, 1 if a step failed, 2 if it was called wrongly.
.Sh SEE ALSO
.Xr rss-kakapo 1
.Sh CAVEATS
The pages are the sites as they were when fetched; a site that has
changed since is studied as it was.
.Pp
The listing is of the sites RSS Parrot's users asked for, not of the web,
and its site URLs are what upstream recorded: one ending in
.Ql /
was taken from a feed rather than given by a user
.Pq Sy page.from_feed .