Profpatsch/users/Profpatsch/observations

View full contents ↓

observations(1)

observations - publish audio readings of other people's texts

observations add [--episodes dir] [--audio dir] [--out dir] --source-title title --source-url url --source-author author --source-published date [--source-site site] [--source-note note] [--title title] [--recorded date] [--notes text] [--number n] [--ffprobe path] [--ffmpeg path] recording
observations render [--episodes dir] --out dir
observations list [--episodes dir]
observations transcript [--format what] [--ffprobe path] [--ffmpeg path] recording
observations listen [--episodes dir] [--audio dir] [--gemini-key-file path] [--pad seconds] [--no-gemini-call] [--ffmpeg path] episode from to
observations convert [--episodes dir] [--force]

observations ingests a recording of someone reading a text out loud and renders a static website and podcast feed from it. For what the format is and why the pipeline is shaped this way, see observations(7).

Ingests recording, which must be the file as it came off the recorder.

The recording is probed for its duration and byte size (both needed by the feed), the transcript track is extracted and decoded if there is one, the audio is copied to the audio directory as NNN.m4a, and the metadata is written to episodes/NNN.json.

add records one source, which is the common case. An episode that reads several texts gets the rest by appending them to the 'sources' array in episodes/NNN.json and running render again; give it a --title at ingest, since such an episode is not named after any one of its texts. The flags are deliberately not repeatable: zipping parallel flag lists by position misaligns silently as soon as one source has a --source-site and another does not, and quietly attributing a text to the wrong author is a worse failure than typing six lines of JSON.

--source-title title

Title of the text being read.
Required.

--source-url url

URL of the text being read. Must be http or https. Required.

--source-author author

Who wrote the text. Required.

--source-published date

When the text was published, as YYYY-MM-DD. Required.

--source-site site

The publication the text appeared in, when it differs from the author.

--source-note note

What this text is doing in the episode – "intro only", "the main piece". Only useful for an episode that reads more than one.

--title title

Title of the episode. Defaults to the title of its source, which is what a single-source episode wants. Set this for an episode built around a subject rather than one text.

--recorded date

Date of the recording, as YYYY-MM-DD. This is also the publication date of the episode. Defaults to the recording's modification time.

--notes text

Prose shown on the episode page and included in the feed entry.

--number n

Episode number. Defaults to one past the highest existing episode.

--ffprobe path, --ffmpeg path

Paths to the ffmpeg tools. Default to ffprobe and ffmpeg on PATH.

Writes the site into the --out directory: an index.html, a NNN/index.html per episode, and feed.xml. An episode page carries the player, the sources, the submission form and the episode's transcript, when it has one; the feed carries the summary but not the transcript. What the page does with the transcript, and why, is in observations(7).

Rendering is a pure function of the episodes directory and can be re-run at any time. --out is required.

Prints one line per episode: number, recording date, running time, transcript word count and source title.

Prints the transcript embedded in recording, without ingesting it. Reads the file and writes nothing, so it is safe to point at anything, including an episode already published.

--format what

transcript

The stored format, on standard output. The default.

text

The transcript as prose, for reading or grepping.

streams

What the container holds: duration, audio codec, and every data track with the transcript marked. This is what answers "why is the transcript missing".

--ffprobe path, --ffmpeg path

Paths to the ffmpeg tools, as for add.

Exits non-zero when the recording carries no transcript, so it can gate a script that is about to overwrite the original.

Prints what a speech model hears in one passage of an episode, beside what the source says and what the transcript currently says. It is for the passages a correction pass cannot settle from the text – see observations-transcripts(7), which is where the judgement rules live.

from and to may be written as HH:MM:SS.mmm as the transcript stores them, MM:SS as a player shows them, or as bare seconds. episode may be written 001 or 1.

Three views are printed, none of them authoritative:

GEMINI

What the model heard. It is a second recogniser, wrong in different places than the first.

SOURCE

The source's own wording, when the span falls inside a .quote. Where there is one this settles the passage and the model is not needed.

TRANSCRIPT

What the file says now, printed last and labelled, because it is the thing being checked.

It never reports a timestamp. The model answers to the millisecond and is late by 250 to 450 ms, which is worse than interpolating within a line; a timestamp for splitting a line comes from the per-word data in git history (observations-transcripts(7)).

Reads the episode and its audio and writes nothing. This is the only subcommand that uses the network or wants a credential; add, render and list never reach it.

--gemini-key-file path

File holding a Gemini API key. Without it the key is taken from GEMINI_API_KEY, and failing that from pass(1). The password store is tried last because it is the one source that can block on a gpg-agent(1) prompt. Consent is not at issue here, unlike observations-inbox(1): the audio is a published episode, not someone else's voice.

--pad seconds

Audio to include either side of the span, default 2. A word at the very edge of a clip is the one most likely to be misheard. The padding is stated in the output, so the extra words are known to be context rather than part of the passage.

--no-gemini-call

Print the source and transcript views only. Needs no key, and answers "what does the article actually say here" on its own.

--ffmpeg path

Path to ffmpeg, used to cut the clip.

Migrates episodes/NNN.transcript.json, the shape the recogniser's protobuf decoded into, to episodes/NNN.transcript, the shape people edit.

This is a one-off: it exists for the episodes ingested before the stored format changed, and is kept because it is the only route back should a conversion ever have to be redone from the JSON in git history. It refuses to overwrite an existing transcript unless given --force, since that transcript has probably been corrected by hand and the JSON has not.

--episodes dir

Episode metadata, which lives in git. Default episodes.

--audio dir

Audio files, which do not live in git. Default audio.

--out dir

Where to render the site. For add this is optional and renders after ingesting.

episodes/NNN.json

One episode: its optional title, the 'sources' array, and the audio's size and duration. A couple of dozen lines, meant to be read and hand-edited; this is the place to add a second source. Committed to git.

episodes/NNN.transcript

The transcript for that episode: one timed line of speech per subtitle-shaped
line, with paragraph breaks and quotations on their own.
In its own file because it is hundreds of lines and would otherwise bury the
metadata above in every diff.
Written by
**add**,
then corrected by hand - raw recogniser output has a mishearing in every
proper noun.
See
observations-transcripts(5)
for the format and
observations-transcripts(7)
for how to correct one.
Committed to git.

audio/NNN.m4a

The recording, byte for byte as it came off the recorder.
Never committed; see
observations(7).

users/Profpatsch/web/observations/

Rendered site, committed and published with the rest of the website.
**Generated - do not edit**.
Change the episode JSON or the transcript for content, or the templates and
the
*styleCSS*,
*replyCSS*
and
*transcriptCSS*
stylesheets in
*render.go*
for layout, then re-run
**render**.
Every generated file says so in a comment at the top.
The page embeds the transcript, so a correction to
*episodes/NNN.transcript*
does not reach the site until it is rendered again.

/var/lib/observations/audio/

Where the audio is served from on the server.

Check that a recording still carries its transcript, before anything is done to it:

observations transcript --format streams ~/Downloads/recording.m4a

Ingest a recording and render the site:

observations add \
    --source-title "Why We're Dropping Basecamp" \
    --source-url https://blogs.library.duke.edu/blog/2023/11/30/why-were-dropping-basecamp/ \
    --source-author "Will Sexton" \
    --source-site "Duke University Libraries Blog" \
    --source-published 2023-11-30 \
    --out ../web/observations \
    ~/Downloads/recording.m4a

An episode built around a subject, reading two texts. Ingest names the first and gives the episode its own title:

observations add \
    --title "Commoditizing Your Complement" \
    --source-title "Laws of Tech: Commoditize Your Complement" \
    --source-url https://gwern.net/complement \
    --source-author "Gwern Branwen" \
    --source-site gwern.net \
    --source-published 2018-03-17 \
    --source-note "intro only" \
    ~/Downloads/recording.m4a

then the second is appended to 'sources' in episodes/002.json:

{
  "title": "Strategy Letter V: The Economics of Open Source",
  "url": "https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/",
  "author": "Joel Spolsky",
  "published": "2002-06-12",
  "site": "Joel on Software",
  "note": "the main piece"
}

Publish it. The two halves go by two different routes, because the audio is not in git:

git add users/Profpatsch/observations users/Profpatsch/web/observations
git commit -m "observations: episode 001"
git push legosi-web canon
rsync -av users/Profpatsch/observations/audio/ \
    root@legosi:/var/lib/observations/audio/

ffmpeg(1), pass(1), observations-transcripts(5), observations(7), observations-transcripts(7)

Profpatsch

The audio directory is deliberately outside git, which means it is not backed up by pushing. The recordings are the one part of an episode that cannot be regenerated; keep them somewhere durable.

The feed carries no itunes:image, because there is no cover art yet. The feed is valid and works in podcast clients, but Apple Podcasts will not accept a directory submission without artwork.

Transcripts are recogniser output as it came out of the recording, and stay that way until someone corrects them by hand. An uncorrected one mishears every proper noun in the episode and punctuates by guessing at intonation; a corrected one has been checked against the sources it quotes. render publishes whichever of the two it finds, with nothing on the page marking which it is, so an episode whose transcript has not been through a correction pass publishes its mishearings under the names of the people it quotes. See observations-transcripts(7) for what correcting one involves and what must not be done to it.

Episode numbers and the URLs derived from them are permanent: they appear in feed GUIDs, and podcast clients remember them. Renumbering a published episode makes clients treat it as a new one.

observations(7)

observations - an audio format for reading someone else's text out loud, and thinking about it

This page is the reasoning behind observations(1), which is the reference: the flags, the files and the caveats are there, and nothing here is needed to publish an episode. What is here is why the format and the publishing pipeline are shaped the way they are – the constraints that would otherwise be rediscovered by experiment, and the one data-loss trap that is easy to walk into.

An observation is a recording of one person reading a short text – an essay, a blog post, a piece of documentation – and thinking out loud about it while doing so. It is deliberately not a podcast in the interview or panel sense: there is no guest, no segment structure, and no attempt at production value.

The form is fixed by three properties, each of which has consequences for the software:

The text is someone else's

An episode is always about one or more specific sources, each identified by a URL, a title, an author and a publication date. The sources are the anchor: the episode page links to them and the feed names them. This is why a source has four required fields and not a free-form description, and why the episode title normally carries the source's own title rather than an invented one – the reader should be able to find the thing being read.

Some episodes are built around a subject rather than a single text, and work through several in one sitting: a short one to introduce the idea, then the piece that treats it properly. Such an episode has a title of its own, because neither source's title is an honest name for it, and each source carries a note saying what it is doing there. The single-source case stays the default, and writes no title at all.

The commentary is unscripted

The value of the form is that reactions happen while reading, not afterwards in an edit. That means recordings are long, single-take, and not worth editing. A 30-minute episode is normal, and the recording is the artefact rather than raw material for one.

One voice, recorded on a phone

There is no studio and no multitrack session. The input to the pipeline is whatever the phone produced.

observations serves the file that came off the recording device, unmodified. It does not transcode, normalise, trim, or re-tag it.

This is not laziness, it is the consequence of the input. The recorder produces HE-AAC mono at 48 kbit/s – already speech-tuned and already small, on the order of 11 MB for half an hour. Any re-encode of that is lossy-to-lossy: it would cost quality to buy compatibility that AAC in an .m4a container does not actually lack, since every podcast client and every browser plays it. Transcoding to MP3 in particular would produce a larger file that sounds worse.

The second reason is the one that matters more, and it is the subject of the next section.

The recordings this format is built around are made by Google Recorder, which performs speech recognition on the device as it records. It stores the result inside the .m4a, as an additional mett data track with the content type application/transcription_2, alongside two further tracks (audio_tags_2, waveform_1) that observations ignores.

That track is a word-level transcript: for the first episode, 308 timed segments containing 4051 words, each with a start and end offset in milliseconds. It is obtained with

ffmpeg -i rec.m4a -map 0:1 -c copy -f data -

and a small protobuf decoder – no speech recognition, no API, no alignment pass. It is, in effect, free.

It is also unrecoverable. Nothing outside the original file has it. Every ordinary audio operation – transcoding, remuxing, "cleaning up the metadata", running the file through an editor – drops unknown data tracks silently and succeeds. The audio is unharmed, the file plays fine, and the transcript is simply gone, with no error to notice. Re-deriving it later means running a speech recogniser over the archive and accepting a different, worse result.

The rule that follows is the least negotiable thing in this program: add extracts the transcript at ingest time and stores it in the repository, whether or not anything currently displays it. For the first two episodes nothing did: the data was captured anyway, because the moment to capture it is the only moment it exists. The episode pages render it now, which is the argument paying off rather than changing – had it been captured only once there was a use for it, there would have been nothing to render.

add does not refuse a file with no transcription track; it ingests it and says the track is missing. Refusing was tried, on the argument that a missing track usually means the file has already been through a tool that stripped it. The argument is right about the cause and wrong about the remedy: add cannot tell a stripped file from one that never had a transcript, so refusing turns a fact worth knowing into an obstacle in front of every recording that was made some other way – and by the time add runs, whatever damage there was has already been done. Ingesting loses nothing that was not lost before, and an episode without a transcript is still an episode.

The moment the check is worth anything is before the destructive step, not at ingest, and it does not need a recording to be an episode:

observations transcript --format streams recording.m4a

prints the container's data tracks and marks the transcript, and exits non-zero when there is none. That is the check to run before moving, converting or tidying up a recording, which is the moment the track is usually lost.

Storing it in git rather than leaving it in the container is deliberate: it is small, it diffs, it greps, and it survives the audio file being moved, re-hosted or lost. It also makes a transcript view, cross-episode search, or click-to-seek a later rendering decision rather than a re-recording problem. The first and third of those are now on the episode pages, and neither required going back to the audio – which is the argument holding: the seeking was built years after the recording, from a file that was captured before anyone knew what it would be for.

Being in git is also what makes the file safe to edit. What the recogniser produced is unrepeatable only up to the moment it is committed; after that the history holds it, a correction is an ordinary diff, and a bad one is git checkout. The rule that nothing may touch the transcript belongs to the window between the recorder and add, not to the file afterwards – which is just as well, because raw recogniser output is wrong often enough to be worth correcting. See observations-transcripts(7).

It lives in episodes/NNN.transcript, beside the episode rather than inside it. The two were one file at first, and the result was that a thirty-line record of what an episode is sat underneath tens of thousands of lines of timed words: every diff, every git log -p and every attempt to hand-edit a source had to be done through that. The split costs one derived path – the transcript's name comes from the episode's, so there is nothing to keep in sync and no way to point at another episode's transcript – and buys back a metadata file that can be read at a glance.

What comes out of the container is a flat list of words, each with its own start and end. That is the right shape for a protobuf and the wrong shape for a file someone edits: encoded as JSON it cost 117 bytes and five lines per word, so a half-hour episode was twenty-two thousand lines, and correcting a single word meant finding it in that and then computing an index into a flattened array the nesting hid. Every error in the first correction pass came from that arithmetic – deletions shifting the indices of later edits, paragraph breaks riding on deleted words, one word replacing three.

So the stored format is line-oriented and subtitle-shaped instead: about six words to a line, timestamps in a fixed-width margin, structure on its own lines. Correcting a word is editing a line. The cost is that per-word timings are not kept – interpolating within a line is accurate to about a tenth of a second, which is inside what seeking audio needs, and the exact values remain in git history. See observations-transcripts(5).

The page is the recording plus the text of it, and every decision on it follows from one fact: leaving the page stops the audio, and no browser comes back to a position half an hour into a recording.

So every link off an episode page – the sources, and the → at the end of each quotation – opens in a new tab, which the rest of the site does not do. The arrow's link carries a text fragment taken from the quotation's opening words, so a browser that supports them scrolls to the sentence and highlights it and one that does not opens the article. Those words come from the transcript, which is the second place a correction pass pays off: a misheard quotation matches nothing. Where the speaker reworded the run-up, a .skip moves the fragment past it (observations-transcripts(5)).

Where scripting is available the transcript is also an index into the audio: the line being spoken is highlighted, and clicking a line seeks to its stored start, which is exact rather than interpolated. Seeking is offered only while the recording plays, and the text is styled as clickable only then – a click that started playback would mean a page being read in quiet can be made to speak by accident. Selecting text does not seek, so a passage can be copied out while it plays, and nothing scrolls on its own. Without scripting the text is all still there and each paragraph timestamp is an ordinary link, so a #t‌ms fragment names a passage in a way that can be shared.

The transcript is not in the feed. An entry carries the episode's summary; a corrected mishearing is not news, and should not re-notify every subscriber.

The site is published by pushing a git repository, and it would be simpler for the audio to ride along in the same push. It does not, for two reasons.

First, this repository is published to https://sources.profpatsch.de/Profpatsch/, so anything committed here appears in a public source browser. Audio files are not source.

Second, and less reversibly: git keeps every blob forever, and compressed audio does not delta-compress. At roughly 11 MB an episode, a weekly cadence is about half a gigabyte a year added permanently to a repository that is cloned often and never shrinks. The cost is not the disk, it is that the mistake cannot be undone later without rewriting history.

So the two halves are published by two different mechanisms, and this is the one piece of friction the design accepts deliberately:

Pages and feed

Rendered into users/Profpatsch/web/observations/ and pushed with the rest of the website.

Audio

Copied to /var/lib/observations/audio/ on the server and served from a separate nginx location.

Keeping the audio outside the Nix store is the same argument in a different key: were the site a store path containing the episodes, every new episode would require a system rebuild and a closure copy to publish one HTML file and one audio file. The site is static output, not a system configuration, so it is deployed like content.

The obvious way to be present on the Fediverse is to be an actor, and the machinery to do that already exists in this repository (the activitypub-go library). observations does not use it.

An actor is a mailbox: once it can be followed, it can be replied to, and replies have to be received, moderated and answered. That is a permanent obligation, and it is a social one rather than a technical one – it cannot be discharged by writing better code. It is not obviously the right one for a format whose whole point is unhurried reading.

A podcast feed has none of that and loses very little. It is subscribable in every podcast client, and it is the standard input to feed-to-Fediverse bridges, so the option of an actor remains open and costs nothing to keep open. The bet is that publishing should not create an inbox until there is a reason to want one.

The section above argued against having an inbox, and then the site grew one: a drop zone where a listener can send a recording of their own, served by observations-inbox(1). That is a reversal, and it is worth being precise about what was actually being argued, because most of it still holds.

The objection was never to receiving things. It was to the obligation that a federated actor creates: something followable is something repliable, and a reply in a conversation expects an answer. That expectation cannot be bounded by code, it does not stop when attention runs out, and withdrawing from it after other servers have followed is not something the protocol makes graceful.

A submission slot is a different shape, and the differences are the whole argument:

So the standing bet is unchanged: no actor, no federated inbox, and conversation still happens on Mastodon under an account that already exists and is already moderated. What changed is the recognition that "send me a recording" does not need any of that machinery, and that asking listeners to record observations of their own while providing nowhere to put them was the weaker position.

An endpoint on the public internet that accepts uploads is found by scanners within days, so the slot is not open: a passphrase spoken at the start of each episode is what unlocks it. Reaching for something only listeners have is deliberate. It cannot be scraped from the page, it costs nothing to change, and the set of people who have heard it is exactly the set of people the invitation is meant for.

Submissions are held, not forwarded. The notification carries an unguessable link rather than the audio, because a recording attached to a mail outlives every deletion, and being able to delete one is the point. There is no listing route for the same reason a token is not a filename: one link exposes one recording, and knowing it says nothing about any other.

Transcription is offered because reading is faster than listening, and it is opt-in because the recording is someone else's voice and sending it to Google is not a decision this program gets to make for them. Unticked means the audio never leaves the machine it arrived on.

A submission is the one place where the rule above – that the artefact is the file that arrived – is bent, and it is worth saying why the exception does not undermine it. A recording made in the page comes from MediaRecorder, which cannot write a duration into a container it is still writing, so the file arrives declaring no length and no player can offer a scrubber for it until it has been played through once. Such a file is remuxed at ingest: the encoded audio is copied into a new container, not decoded, so the samples are the samples that arrived. The reason this is safe here and not for an episode is that the thing an episode would lose to a remux – the transcript track, which exists nowhere else – is a property of Google Recorder files, and a submission has no such track to lose. See observations-inbox(1).

The slot takes a dropped file, and it also records one: the episode page has a button that captures from the microphone and hands the result to the same file input a dropped file lands in. This is the difference between replying being a thing you do now and a thing you do later with a phone, a file manager and a laptop – which is to say, mostly not at all.

It is strictly an addition. The button ships hidden and is revealed only where MediaRecorder exists, so a browser without it, or without scripting at all, is left with a plain file input that works everywhere. The recorded blob goes through the same checks as any other upload, and the server sniffs the bytes rather than believing what the browser called them.

Episodes are numbered sequentially and live at /NNN/. Feed GUIDs are derived from that number.

Numbers rather than title slugs because a URL that encodes the source's title cannot be corrected later without breaking every subscriber and every archive that recorded it, and because the source's title is not observations's to fix. Numbers rather than dates because two recordings can share a day, and because the publication date of the episode and the date of the text being read are different things that a date-shaped URL invites confusing.

The title displayed – 001 -- Why We're Dropping Basecamp – carries both: the number that is stable and the source title that is searchable. Retitling an episode is therefore always safe.

ffmpeg(1), observations(1), observations-transcripts(5), observations-transcripts(7)

Profpatsch

observations-transcripts(5)

observations-transcripts - transcript file format for observations episodes

An episode's transcript lives in episodes/NNN.transcript, one file per episode, beside the episodes/NNN.json that describes it. It is a line-oriented text file: a line whose first character is '.' is a control line, and every other line is a timed line of speech.

Nothing in the format needs quoting or escaping except a source title. No word a speech recogniser produces contains a space, a quote or a backslash, so a speech line can be split on whitespace and read directly.

HH:MM:SS.mmm HH:MM:SS.mmm  text of the line

Two timestamps – when the line starts and when it ends – then two spaces, then the words. Timestamps are offsets into the recording, always twelve characters wide, always with hours and milliseconds even when they are zero. The fixed width is the point: the text of every line begins at the same column, so the file can be read as prose by looking past a margin, and a timestamp can be found by eye without counting.

A line covers exactly the audio its words do: the start is the first word's, the end is the last word's. Lines do not overlap and are in ascending order. Gaps between them are silence.

.\" text

A comment, in the roff convention.

.episode NNN

The episode this transcript belongs to.

.recorded YYYY-MM-DD

The day of the recording.

.source id "author "title""

Declares a text the episode reads, and gives it a short id for .quote to refer to. Author and title are Go-quoted strings, because titles contain apostrophes, colons and occasionally quotation marks.

.url id url

The URL of a declared source. It is a line of its own because URLs are long and would otherwise force the .source line past a readable width.

.p

A paragraph break, as the recogniser marked it. It carries no timestamp; the line after it does.

.quote id

Opens a passage read aloud from the named source.

.skip n [-m]

Trims the words a quotation offers as its link into the source text. Written directly after the .quote it belongs to, and only where the default is wrong.

.endquote

Closes it.

A quotation is rendered with a link to the passage it was read from, built as a text fragment from the quotation's own opening words (observations(1)). That works whenever the speaker began reading at the start of a sentence. When they reworded the run-up it does not: the article says "the group from whom the DEI movement drew", the recording says "from which the DEI movement drew", and the opening words match nothing. .skip n drops the first n words before the fragment is taken, and an optional negative second count drops words from the end:

.quote sexton .skip 2 00:34:11.100 00:34:13.500 from which the DEI movement drew its .endquote

Four of episode 001's thirty quotations need one.

It counts words rather than giving replacement text on purpose. Nothing can check either against the article, because rendering does no networking, so a line that could state words the recording does not contain would be unverifiable in both directions; a skip can only ever select words that are already in the file. A skip that runs past the end of the quotation leaves the link pointing at the article, which is what a fragment that matches nothing does anyway.

The id is conventionally the author's surname in lower case, so that marking up a passage by hand requires nothing to be looked up.

.\" observations transcript, episode 001. .episode 001 .recorded 2026-08-28 .source sexton "Will Sexton" "Why We're Dropping Basecamp" .url sexton https://blogs.library.duke.edu/blog/2023/11/30/\ why-were-dropping-basecamp/

.p 00:00:00.150 00:00:02.670 Hey, and welcome to observations. 00:00:04.230 00:00:06.570 Today, I would like to talk. I would 00:00:06.570 00:00:08.490 like to read and then maybe talk a

.p 00:00:41.490 00:00:43.530 And the article starts like this. .quote sexton 00:00:45.510 00:00:47.730 We at Duke University libraries have 00:00:47.730 00:00:49.650 decided to stop using the project .endquote 00:00:52.110 00:00:54.150 I'm not yet done with the intro.

Text aims for 37 characters and may reach 42, which are the subtitle widths the broadcast style guides converge on: long enough not to shred a sentence into scraps, short enough to take in at a glance.

The width is not only cosmetic. Because a line is about six words, and a line's timestamps are exact, the position of a word inside it can be interpolated to about a tenth of a second – which is what makes it safe not to store per-word timings at all. Longer lines would degrade that: at one line per utterance the same interpolation is out by up to three seconds.

The recogniser produces a start and end for every word, and this format keeps only the line's.

This is a deliberate loss, taken once. Interpolating within a line is accurate to roughly 120 ms in the median and 420 ms at the ninetieth percentile, which is inside the tolerance of seeking audio by clicking a word – the use the timings exist for.

In the event nothing needs the interpolation. An episode page seeks by the line, to its stored start, which is exact and is where the phrase a reader clicked begins; a listener following the text does not want to land in the middle of a sentence. The estimate above is therefore the argument that a line is a fine enough grain to store, rather than something the site computes.

The exact per-word data remains in git history, in the NNN.transcript.json files this format replaced, so the decision is reversible for as long as the history is.

The recogniser groups words into silence-delimited segments. The format does not name them, because it does not have to: a line never spans a pause, so every one of them survives as a gap between consecutive lines.

The recogniser emits each word twice, once bare and once punctuated: 'Hey' alongside 'Hey,'. Only the second is kept. The two never differ except in punctuation and the case of the first letter, so nothing is lost that could not be recovered by stripping.

observations(1), observations(7), observations-transcripts(7)

Profpatsch

The format has no way to say that a passage is unintelligible. A correction pass leaves such passages exactly as the recogniser heard them, which means a transcript can contain words nobody said and nothing marks them as suspect. See observations-transcripts(7).

observations-transcripts(7)

observations-transcripts - correcting the transcript of an episode

Every episode ships with the transcript Google Recorder produced while the recording was being made (observations(7)). It is free, it is word-level, and it is wrong: it mishears every proper noun in the episode, it punctuates by guessing at intonation, and it writes down every "um".

This page is the procedure for correcting one. For the file it produces, see observations-transcripts(5).

It is not automated and is not a subcommand of observations(1). The judgements involved are the whole of the work, and a program that made them automatically would be a program that invented sentences. What follows is therefore a strategy, meant to be carried out with whatever tools are at hand.

observations(7) is emphatic that the transcript exists nowhere but inside the original .m4a, and that ordinary audio tools destroy it silently. That is true, and it is why add extracts it and commits it.

It stops being true the moment the file is in git. Then the history holds it, a correction is a diff that can be read line by line, and a bad correction is git checkout. Edit episodes/NNN.transcript in place.

Two things do have to hold:

An episode reads someone else's text aloud, so a third of it is quotation whose correct wording is a fetch away. Aligning the transcript against its sources is the first thing to do, before reading a word of it: it marks the quoted passages and finds most of the mishearings in one step.

The method, which needs no special tooling:

  1. Fetch each URL in the 'sources' array of episodes/NNN.json and strip it to text.
  2. Normalise both sides to lower-case alphanumerics and split into words.
  3. Align the two word lists. Python's 'difflib.SequenceMatcher', with 'autojunk=False', does this well; the flag matters, because without it common words are discarded as noise and the match falls apart.
  4. Matching runs of six words or more are quotations. Merge runs separated by a small gap: a stumble mid-sentence splits one quotation into two.
  5. The short mismatches inside those runs are the mishearings, with the correct wording sitting beside them in the source.
    On episode 001 this matched 62 runs, which merge into 30 quoted passages, 38% of the episode, with boundaries landing exactly where the reading starts

Five classes, in the order they are worth fixing. The counts are from episode 001, 4051 words of a single half-hour recording, and are typical.
Proper nouns
The largest class and the most damaging, because a misheard name is unsearchable and looks like ignorance on the reader's part. The recogniser has no idea who is being discussed, so the same name comes out differently every time it is said: "David heinemann, Maya Hansen", "David heinemeyer, Hansen", "David Henry anime Hansen", "Henson" and "Hence" were all one person.
Misheard quotations
Found by the alignment above, along with the correct wording.
Sentences broken at a breath
The recogniser punctuates by intonation, not by syntax, so it ends a sentence wherever the speaker paused and capitalises the next word to match. A sentence that took two breaths arrives as two, and the seam is invisible in the file because every line is short anyway. The opening of episode 001 came out as four sentences, three of which cannot stand alone:
about an article published. In 2023, November 30th. On the Duke University libraries blog. By a person named Will Sexton.
It is the largest class by count and the easiest to miss: 345 sentences in the episode, median 9 words against 18 to 33 in the article being read, and 36% of them opening with a conjunction or a preposition. Inside a .quote it is mechanical, because the source has the right boundaries - 119 words across 91 lines were restyled from it in one pass, and no word changed. Outside one it is a judgement call, and the test is whether the second fragment can stand as a sentence: "And so a lot of people left" can, and stays; "By a person named Will Sexton" cannot, and is joined to what it belongs to. Speech is genuinely more fragmentary than prose, so the answer is not to join everything.
It also survives a pass that thinks it has done it. Episode 001 needed a second round of 34 joins after the first, because the two rounds see different things: reading the file finds the stop at the end of a line, where the dangling word is right there in the margin ("DHH was already always very."), while reading the page finds the stop in the middle of a running paragraph, where nothing marks it but the sentence not making sense ("a higher degree of. I'm thinking about societal issues"). Do both, in that order, and treat the page as the one that decides.
Two mechanical guards are worth having, because the judgement is easy and the bookkeeping is not. A join may not cross a .p: five candidates in 001 looked like broken sentences and were a paragraph break and a three-second pause apart, and joining those would have said the speaker ran on when he had stopped. And lowercasing the following word is only right when the recogniser capitalised it - "Python Ruby Dynamic language hype" opened a joined sentence and is a name, not a sentence opening, so a blind pass over 30 joins produced exactly one word that had to be put back.
Question marks the recogniser guessed
67 words in episode 001 ended in a question mark. Most were not questions - "Thinking about?", "coming from?" - but some were, and the difference is not mechanical. See QUESTION MARKS.
Fillers
49 "um" and "uh". A closed list, and it should stay closed: "like" and "so" occur 66 and 46 times and are mostly doing grammatical work.

  1. Commit anything outstanding, so the correction is its own diff.
  2. Align against the sources, as above. Mark the quotations with .quote and fix the wording the alignment disagrees with.
  3. Read the rest as prose.
  4. Correct by editing lines. A word is on a line with its timestamps in front of it; change the word.
  5. Flag, do not guess. See WHAT NOT TO CORRECT.
  6. Tidy the line breaks - see REFLOWING LINES.
  7. Verify, see VERIFYING.

Lines are filled greedily by convert, which is fast, mechanical and slightly stupid: it will strand a single word on a line of its own where a human would have balanced the two lines. Episode 001 has 33 such lines out of 751.
No cleverer algorithm is wanted here. Judging where a line should break is exactly what a careful reader - or a language model - does well and a line-filler does badly, so it is left as a manual pass with the following things to look for:

A guessed question mark is wrong, but it is doing a job: it ends a sentence. Deleting it outright runs two sentences together - "for Team Management And apparently recently"

The failure mode of this work is not leaving a mistake in. It is producing a fluent sentence the speaker never said, which nothing downstream can detect and no future reader will question.
Three things are therefore left exactly as recognised:
Anything unintelligible
If the source text does not settle it and the surrounding words do not either, it stays as it is and goes on a list with its timestamp, for someone who can play the audio - or for observations listen, see LISTENING TO A PASSAGE.
This is not a formality, and the list is not short: episode 001 produced sixteen such passages, and the guesses that could have been made instead would have been wrong in the way that matters. "Let base cam subscription Labs" is "let Basecamp subscription lapse"; "Complicated JavaScript fondant" is "no complicated JavaScript framework", where dropping the "no" would have the sentence say the opposite of what was said; "wolf sheep and shipsies" is the title of the post being described, which no amount of context supplies - it is "Wolves, sheep, and gypsies", and what settled it was not the audio but the blog the episode is about, which still has the post on its front page. When a passage names something, the thing named is a source too. A plausible substitute - "moron ethical" for "moral/ethical" - reads as fluent prose while saying something the speaker did not say.
The worst of them were worse than that. Several passages came out of the recogniser as harmless nonsense while the words actually spoken named a notorious book and a fascist leader - the episode is about racism in the software industry, and it says so plainly. A pass that had guessed at those would have replaced the point of the episode with something inoffensive and false.
Spoken self-corrections, outside a quotation
He stumbled and fixed it aloud, and the stumble is what happened.
Inside a .quote this does not apply, because there the transcript is reproducing someone else's text and the text is what it is: "the merger of murder of George Floyd" and "inclusive workspaces workplaces" are both misreadings of the article, and both are written as the article has them. The distinction is the whole point of marking the quotations - what the speaker said and what he was reading from are now different things in the file, and only the first is a record of a performance.
Jokes that depend on the error
Episode 001 contains "continued use of big band Basecamp. Sorry, I said bandcamp, which is a" "completely different thing". Correcting "bandcamp" would delete the joke and leave an apology for nothing. This is why the proper-noun pass is hand-checked rather than a blind find-and-replace.

The list of doubtful passages needs ears. observations listen supplies a substitute for them: it cuts the passage out of the recording, asks a speech model what it hears, and prints that beside the source's wording and the transcript's:
observations listen 001 00:24:49 00:25:02
It takes the API key from the password store itself; a key from anywhere else goes in GEMINI_API_KEY or --gemini-key-file.
Timestamps may be written as the transcript stores them, as MM:SS, or as bare seconds; the flags are in observations(1). Nothing is written, so it is safe to run against a published episode.
The model is a second recogniser, not an ear. It is wrong in different places than the one that produced the transcript, which is exactly what makes it useful - on episode 001 it settled a dozen passages that two readings had failed to: "Ly." was the tail of "calmly", "stickers" was "shtick", "The stage is" was "the state is". It also hears stumbles the first recogniser tidied away, which the format keeps.
Three rules, each of which was paid for:

The mechanical checks are worth running because they catch precisely the damage that reading the prose does not:

The state of episode 001 after one pass, as a sense of scale:
words 4051 -> 3976 (49 fillers, plus merged words) paragraphs 84 -> 83 (one was only "Um.") audio span unchanged
That pass took two rounds: everything derivable from the sources first, then a second round applying sixteen passages that had to be listened to. Both rounds are the same operation on the same file, and the second is the one that fixed the worst errors in the episode.

observations(1), observations-transcripts(5), observations(7)

Profpatsch

A correction pass is only as good as the corrector's access to the audio. The alignment above fixes everything that is quotation, but the commentary between the quotations is unscripted and has no source to check against.
There is nothing in the file that records which passages are still doubtful: the flagged list lives in whatever conversation produced it and is lost afterwards. In episode 001 the passages that needed ears included the two most serious errors in the transcript: in both, the words spoken were an explicit comparison to fascism, and in both the recogniser produced harmless nonsense instead. Neither was recoverable from the source text, and neither looked wrong on the page.
A pass done without audio therefore produces a transcript that reads cleanly, is right about every proper noun in the sources, and is still wrong in exactly the places where being wrong is worst.
That transcript is now published: the episode page renders whatever is in the file, and nothing on it distinguishes a corrected transcript from raw recogniser output, or a checked passage from one nobody could make out. A reader has no way to tell, which is the argument for finishing a pass rather than shipping it half done.
A speech model closes some of that gap - see LISTENING TO A PASSAGE for observations listen and the rules for trusting it. It is a second recogniser, though, and not an ear: it is wrong in different places rather than in none, and where it and the first one disagree there is still nobody to settle which is right. The passages it cannot settle are the same ones as before, and they are the ones worth asking a human about.