1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509

.Dd September 30, 2026
.Dt MEDIATHEK 7
.Os
.Sh NAME
.Nm mediathek
.Nd why the MediathekView-to-PeerTube bridge works the way it does
.Sh DESCRIPTION
.Xr mediathek 1
is the reference.
This page gives the reasons behind it, so that the next change does not
undo one of them by accident.
Read the section of the behaviour you are about to change first.
.Sh LIGHT ON THE SERVER
It runs on a small machine (2 cores, 4 GB, a nearly full 19 GB disk), so
nothing is done per request that could be done once a day, and nothing is
done at all that the broadcasters' CDNs already do:
no remuxing, no transcoding, no proxying of video bytes, and no caching of
anything fetched at request time.
Media endpoints are redirects or small generated playlists; the only
upstream fetch at request time is a subtitle file.
.Sh THE FILM LIST
The film list is one JSON object on a single ~500 MB line, with the same
key
.Pq Ql \&"X"
repeated for every row.
It is read with
.Sy encoding/json Ns 's
token decoder, which streams (memory stays at a few MB), accepts repeated
keys, and reports whether the closing brace arrived.
That last point matters: a download cut short would otherwise parse as a
smaller, valid-looking list.
.Pp
The list is fetched as
.Pa .bz2
and decompressed with the standard library, straight into the parser: no
temporary file, no xz dependency.
Only the full list is used, never MediathekView's diffs; a full import a
day is simple and makes every generation self-contained.
.Pp
Empty sender and topic columns mean
.Dq same as the row before ;
the small and HD URL columns are relative to the main URL, as
.Ar n Ns | Ns Ar suffix .
.Sh GENERATIONS
Every data table has a
.Ql gen
column that references
.Ql generation(id) ON DELETE CASCADE ,
so deleting a generation row deletes the whole generation, and nothing can
be left behind by forgetting a table.
Foreign keys are enabled in the connection string, not by a
.Ql PRAGMA
after opening: pragmas are per connection, and
.Sy database/sql
opens several.
.Pp
The full-text indexes are virtual tables, which a cascade does not reach.
They are contentless
.Pq Ql content=''
with
.Ql contentless_delete=1 ,
so a row can be removed by its rowid alone, and an
.Ql AFTER DELETE
trigger on the indexed table does that; the cascade fires the trigger.
Rows are indexed only after deduplication, and deleting a rowid that was
never indexed is a no-op, so the trigger is safe for the rows dedup drops.
.Pp
The flip itself only changes the states in the generation table, in one
short transaction, so the server always sees one complete generation.
Deleting the old previous generation (about 700\~000 rows through the
cascade) happens after the flip, in its own transaction, so it does not
hold the write lock during the flip.
If that delete fails, the generation stays in state
.Ql building ,
where the next import cleans it up.
.Pp
The verification thresholds guard against the failures that have actually
been seen with the upstream list: truncated downloads (closing brace), a
broken crawler producing a fraction of the list (80% of the live row count,
100\~000 absolute), and a crawler losing whole broadcasters (25 senders).
.Pp
Exactly one previous generation is kept, for
.Cm generations rollback .
With about 700 MB per generation and a third one being built during an
import, the file levels off at about 2 GB; freed pages are reused, the file
does not shrink.
.Sh STABLE IDENTIFIERS
Links and NewPipe subscriptions must survive every re-import, and also any
later change of our own interpretation of the list: how broadcasters are
grouped, how show names are normalized, how URLs are rewritten.
.Pp
A video's uuid is therefore derived only from what the film list says, never
from what we make of it: a name-based (version 5 shaped) hash of the sender,
the show name and the main URL, all exactly as in the list.
The show name is part of it because KiKA files the same clip under up to
seventeen shows; sender and URL alone collide 170 times, with the show name
not once.
.Pp
Episode deduplication keeps one video per (channel, normalized title, day)
\(em the one with the best quality \(em and stores the uuids of the others as
aliases of it, so every uuid ever handed out keeps resolving while its entry
is in the list, whichever duplicate wins today.
.Pp
Channel names are
.Ar family Ns - Ns Ar show ,
e.g.
.Ql ard-maischberger .
The family slugs are fixed labels and must never be renamed.
If a family is ever split, one part keeps the old slug, the other gets a new
one, and the old slug goes into
.Sy familySuccessors ,
through which channel and account lookups fall back, so subscriptions to
the old names keep working.
Numeric ids are hashes of the names, cut to 53 bits so they survive a
JavaScript number.
.Sh FAMILIES
ARD and its regional stations (WDR, NDR, SWR, BR, HR, MDR, RBB, SR, Radio
Bremen, ONE, ARD-alpha, tagesschau24) are one family, and ZDF with ZDFneo,
ZDFinfo and ZDF-tivi another: they share shows, and about 15\~000 episodes
appear under more than one of them.
Every other broadcaster is its own family, including 3sat, PHOENIX and KiKA,
which ARD and ZDF run jointly.
Merging across families would be wrong:
.Dq Tagesschau
is a different show at SRF than at ARD.
.Sh WHAT NEWPIPE READS
NewPipe's PeerTube support (NewPipeExtractor,
.Pa services/peertube )
decides most of the API's shape:
.Bl -bullet
.It
Link handlers match on the path only
.Pq Pa /w/ , Pa /videos/watch/ , Pa /c/ , Pa /video-channels/ , Pa /a/ , Pa /accounts/
and take the instance from the URL, so a pasted link works without adding
the instance first.
.It
Its JSON helper throws on a missing key, so these must always be present:
a video's
.Ql name ,
.Ql publishedAt ,
.Ql uuid ,
.Ql account.{name,host,displayName} ,
.Ql channel.{url,displayName} ,
.Ql privacy ,
.Ql category.label ,
.Ql licence.label ,
.Ql nsfw ;
a file's
.Ql resolution.{id,label} ,
.Ql fileUrl ,
.Ql torrentUrl
.Pq may be empty ;
a channel's
.Ql displayName
and
.Ql ownerAccount.{name,url} .
.It
In lists it tells item types apart by keys: an object with
.Ql followersCount
is a channel.
So channels always carry it and videos (and the accounts embedded in
videos) never do.
.It
It builds uploader URLs as
.Pa accounts/ Ns Ar name Ns @ Ns Ar host ;
the
.Ar @host
is stripped.
.It
A stream's format comes only from its URL's extension.
A URL with a query string, or without
.Pa .mp4
at the end, would get no format, which crashes the app in the download
dialog; such files go through
.Pa /f/ Ns Ar uuid Ns / Ns Ar height Ns Pa .mp4 .
.It
Quality labels are sorted by their digits, so they are always
.Ar height Ns Ql p .
Height means the smaller side (PeerTube's convention), so vertical and
square video get sensible labels.
.El
.Sh HLS
For every file of a streaming playlist NewPipe adds two streams: a
progressive one from
.Ql fileUrl ,
and an HLS one at
.Ql playlistUrl
with
.Ql master
replaced by the resolution id.
For playback the app prefers the HLS one; downloads offer the progressive
one only.
.Pp
So an HLS-only video has
.Ql playlistUrl
.Pa /hls/ Ns Ar uuid Ns Pa /master.m3u8 ,
per-height
.Pa /hls/ Ns Ar uuid Ns / Ns Ar height Ns Pa .m3u8
redirects to the broadcaster's variants, and
.Ql fileUrl
points at
.Pa /remux/ Ns Ar uuid Ns / Ns Ar height Ns Pa .mp4 ,
which answers 404.
The stub exists only to give NewPipe a
.Pa .mp4
extension to recognise; making it work would mean remuxing on the server.
The master is written from the database, not fetched: the variant URLs and
heights are known, and the bandwidths only need to order the variants.
.Pp
A
.Ql playlistUrl
ending in
.Pa -master.m3u8
makes NewPipe derive variant URLs differently (from
.Ql fileUrl ) ,
so that shape is avoided.
.Sh QUALITY LABELS
The film list has up to three URLs per entry (small, normal, HD) but no
resolution.
The height is read from the URL by a table of CDN naming schemes, each
checked with ffprobe or against an upstream HLS master on real URLs from the
list (ZDF
.Ql _p Ns Ar NN ,
ARD
.Ql . Ns Ar size Ns .mp4 ,
NDR, BR, SR letter codes, ARTE bitrates, Funk file prefixes and bitrate
ladders, SRF and ORF HLS variants, Akamai csmil lists, and any
.Ar W Ns x Ns Ar H
in the path).
Where a file name says nothing (WDR, MDR, KiKA use opaque ids), the label
is the majority height of that host's column, measured on samples; about 1%
of files fall back to a blind guess by column.
.Pp
Unanchored regular expressions over every URL were most of the import's
CPU, so most rules only look at the file name, and the expensive ones are
gated by a cheap string test.
.Sh URLS
Plain-HTTP URLs are rewritten to HTTPS, since Android refuses cleartext by
default: most hosts serve the same path over HTTPS, some only under another
host name (e.g.
.Pa nrodl.zdf.de
\(->
.Pa nrodlzdf-a.akamaihd.net ) ,
each checked on samples.
Two hosts have no HTTPS at all and are kept as HTTP.
.Pp
Live streams in the list (topic
.Dq Livestream ,
and URLs under
.Pa /hls/live/
or on ARD's live CDNs) are dropped: they are not videos, and a PeerTube
video that never ends confuses NewPipe's player.
Entries dated in the future are stored, and hidden by the server until
they are due, so they appear without a re-import.
.Sh SUBTITLES
The broadcasters serve TTML/EBU-TT-D (ARD, ZDF, KiKA; ORF plain TTML with
line breaks as escaped text) or WebVTT (SRF).
NewPipe needs a format it knows from the file extension, so
.Pa /sub/...vtt
fetches the original and converts it on every request.
Nothing is cached on the server, per the rule above; the response allows
clients and a proxy to cache it for a day.
The converter drops styles and layout and keeps only timing and text.
.Sh THUMBNAILS
The film list has no pictures, and the video files carry none either: of
100 files from the 50 biggest CDN hosts, none had cover art or even a title
tag.
Grabbing a frame from the video works but costs 2\(en4 seconds and several
megabytes per video, which is exactly the video traffic this server
refuses to carry.
.Pp
What is left is the broadcaster's own page for the video, the list's
website column: ARD's JSON API (its pages are a JavaScript app), PHOENIX's
JSON API (its HTML carries an unrendered template), and
.Ql og:image
from the page head for the others.
This is a deliberate exception to
.Sx LIGHT ON THE SERVER ,
kept small:
.Bl -bullet
.It
Only the image URL is stored, never the image; the thumbnail endpoint is a
redirect to the broadcaster's image server, and the phone loads the picture
from there.
Storing the pictures would be about 15 KB each, some 10 GB for the list,
which the disk does not have.
.It
The cache is a table outside the generations, keyed by the video uuid, which
survives re-imports, so each video is looked up once, not once a day.
The importer prunes rows of videos no longer listed.
.It
A lookup is started by the first request, never in bulk: nobody scrolls
through 700\~000 videos, and a crawl of all of them would be a daily burst of
requests against the broadcasters for pictures nobody sees.
.It
At most four lookups run at once and at most 200 wait; beyond that a
request gets a 404 and nothing is queued, so a burst of list views cannot
pile up work.
Concurrent requests for one video share one lookup.
.It
A request waits at most 2 seconds, which is what a list can afford; the
lookup itself may take 15 seconds and fills the cache when it finishes, so
the picture is there on the next view.
Because that first answer is only
.Dq not yet ,
it is sent with
.Ql no-store ;
.Dq found
and
.Dq none
are cacheable for a day.
.It
Only hosts on a fixed list are fetched, and a redirect off them is refused:
the website URLs come from an outside list, and the server must not be
usable to request arbitrary addresses.
.It
Site-wide sharing pictures and
.Dq not found
images (ARD's sharing icon, ZDF's logo and question-mark Mainzelmann) are
treated as no picture.
.El
.Pp
The broadcasters' image services are asked for a thumbnail-sized image
where they take a size: 640 px for ARD, 640\(mu360 for ARTE, 768\(mu432 for ZDF,
and 480 for SRF,
whose service answers 400 for most widths (240, 320, 480 and 960 work).
.Sh ARTE
The film list carries one entry per ARTE language version, each with MP4
files of that version: French audio for ARTE.FR, German for ARTE.DE, and so
on.
ARTE's player streams one HLS master per programme instead, with every
audio language (and a
.Dq clear speech
variant of each) and every subtitle as renditions; the master of each page
language only differs in which audio is marked
.Ql DEFAULT .
.Pp
The master is looked up like a thumbnail (see
.Sx THUMBNAILS
for the reasons): on the first request, from
.Ql api.arte.tv/api/player/v2/config/ Ns Ar lang Ns / Ns Ar programme ,
on a fixed host list, and stored outside the generations.
It is keyed by programme id and language, not by video, because the plain,
subtitled and audio-described entries of one programme share it.
Its text is stored, not its URL: NewPipe does not play a master as given.
For every quality it builds an HLS stream from
.Ql playlistUrl
with
.Ql master
replaced by the height, and plays that one URL.
ARTE's per-quality playlists are video only, so each of these URLs must
itself be a master, of that one quality plus all audio and subtitle
renditions; it is written from the stored text, so playing costs ARTE
nothing but the media.
Only the H.264 variants are offered: not every phone decodes the HEVC ones,
and ExoPlayer only ever sees the one variant NewPipe picked.
.Pp
ARTE's masters sometimes list a subtitle file a second time as
.Dq VO (Forced) ,
labelled with the language of the original audio rather than of the text:
Studio 54
.Pq 132050-000
has an
.Dq en
track that is the French file, and 3 of 40 programmes checked had such a
copy (labelled Portuguese, Arabic).
A rendition whose file an earlier one already lists is dropped when the
master is served, so the stored text stays ARTE's.
.Pp
The language choice is ExoPlayer's.
NewPipe fills its audio-track menu from the streams its extractor reports,
and the PeerTube extractor names none, so the menu stays hidden; nothing in
PeerTube's API can change that.
ExoPlayer plays the rendition marked
.Ql DEFAULT
(it ranks that above the phone's language), so the served master marks the
video's own one.
All versions of a programme in one page language share a master, whose
default is the dubbed audio, but each film list entry is one version, and
its MP4 name says which:
.Ql 132050-000-B_SQ_0_VO-STF_...
is the original audio
.Pq Ql VO
with French subtitles
.Pq Ql STF ;
the audio renditions carry the same code
.Pq Ql tv.arte.audio.version.vo-eng .
Serving a master moves
.Ql DEFAULT=YES
to the rendition with the entry's audio code (not its
.Dq clear speech
twin), and to the subtitles that go with the version where they matter:
translations of an original version
(for
.Ql VO-STF
the track
.Ql vo-fra )
and subtitles for the hard of hearing (for
.Ql VA-STMA
the track
.Ql va-mal ) ;
a dubbed version's same-language subtitles are the forced ones, shown
anyway.
On 155 videos checked every audio code had its rendition.
Like the duplicate filter this is a rewrite of the stored text when
serving, not of what is stored, so one lookup serves every version.
Its caption menu, on the other hand, is built from ExoPlayer's text tracks,
so ARTE's subtitles appear there.
It shows a track's language, not its label, though (ExoPlayer puts
.Ql NAME
into the label), and selects by language: ARTE's German hearing-impaired,
automatic and plain subtitles all read
.Dq de ,
and a tap picks any of them.
So the served master writes each subtitle's name into its
.Ql LANGUAGE ,
behind the code:
.Ql de-Deutsch (Für Gehörlose und Schwerhörige) ,
made unique.
The code stays in front because ExoPlayer compares languages on the part
before the first
.Ql - ,
so forced subtitles still follow the audio, and a caption language chosen
once roughly carries over.
ExoPlayer lowercases the value, and it is not a valid language tag, which
only NewPipe, the one reader of these playlists, has to tolerate.
The proper fix is NewPipe's: label the menu from the track's label and
select with a track override.
.Pp
Because the subtitles live in the master,
.Pa /captions
stays empty for ARTE rather than listing them twice.
A proper language switch belongs in NewPipe, like the
.Ql newpipe://
scheme (see
.Sx SHARED LINKS ) :
an audio menu built from ExoPlayer's tracks when the extractor names none.
.Pp
Programme ids come from the website column
.Pq Pa www.arte.tv/ Ns Ar lang Ns Pa /videos/ Ns Ar id Ns Pa / ;
they also link the versions of a programme to each other.
Collection pages
.Pq Pa RC-...
have none.
.Sh LOGOS
Accounts and channels show the broadcaster's logo, from the Kodi add-on
plugin.video.mediathekview (MIT), trimmed and set on a dark square so its
white artwork shows on both of NewPipe's themes.
They are part of the binary; channels use their main sender's logo, the
merged ARD and ZDF families the ARD and ZDF logos.
.Sh SHARED LINKS
The HTML pages exist for links shared outside NewPipe.
They carry Open Graph tags, so a messenger shows a preview.
Their
.Sq og:image
is the picture from the thumbnail cache if there is one, otherwise
.Pa /thumb/ Ns Ar uuid Ns Pa .jpg ,
whose first fetch (usually the messenger's crawler) starts the lookup:
rendering a page never waits for a broadcaster, and the preview has a
picture from the second share on.
All URLs in them are absolute, built from
.Fl -base-url ,
as elsewhere.
.Pp
The lists on these pages are paged by
.Dq older than
a time
.Pq Pa ?vor= Ns Ar unixtime ,
not by page number.
The daily import adds videos at the top; with numbered pages every page
would shift, and a reader going through them would see some videos twice
and miss others.
A page cut inside one second would lose the rest of that second, so a page
is extended to the end of its last second; it can be a little longer than
the nominal size, but every entry is on exactly one page.
.Pp
The obvious next step, a button that opens the page in NewPipe, does not
work with NewPipe as it is (checked against its manifest and
NewPipeExtractor, September 2026):
.Bl -bullet
.It
Its PeerTube intent filter names a fixed list of hosts, and this one is
not on it; an
.Ql intent://
link with
.Ql package=org.schabi.newpipe
only restricts which app may match, it still needs a filter that accepts
the URL, and browsers only follow links to filters marked BROWSABLE, which
its share filter is not.
.It
Its
.Ql vnd.youtube\&:
filter does accept any host and is BROWSABLE, but such a URL is offered to
the services in turn: YouTube's parser takes the first eleven characters
after
.Ql //
as a video id (here
.Ql mediathek-n ,
a valid one), and PeerTube's rejects the scheme outright
.Pq Ql new URL() .
.El
.Pp
So the pages show their link with a copy button instead, to paste into
NewPipe's search, which does accept any PeerTube URL.
The real fix belongs in NewPipe: a
.Ql newpipe://
scheme that hands its URL to the normal link handling.
.Sh SEE ALSO
.Xr mediathek 1