Semantic search over fediverse posts. Point it at some actors, let it walk their outboxes, embed everything (text and images), then search by meaning rather than by keyword.

Picking this up fresh? Read "What Phase 0 established" for the constraints that are already settled, then go to Phase 1. Everything before Phase 1 is finished and committed; don't relitigate it without a reason.

This document records why things are the way they are, and the traps found by measurement rather than guesswork. Mechanics of the shipped CLI belong in --help and the manpage; the non-obvious constraints belong here.

Phase 0 exists as spike.go. It downloads all-MiniLM-L6-v2, runs it through GoMLX's pure-Go backend, and asserts the output matches hugot's published reference vector for the string "robert smith". It is a probe, not a library.

Four things were settled, each by measurement:

hugot is not needed. github.com/knights-analytics/hugot wraps exactly this stack, but costs 37 extra packages: viant/afs (18 packages of cloud-storage abstraction that fedisearch never touches) and x/crypto (9, via its model downloader). Calling onnx-gomlx, gomlx and hftokenizer directly costs about 400 lines of tensor glue — which is what spike.go is — and reproduces hugot's reference vector to within 1e-4. Closure went from 148 packages to 111.

buildGo can build the closure. The risk was google.golang.org/protobuf, unavoidable because the .onnx format is protobuf, and containing a //go:embed that buildGo's external analyser does not collect. It works anyway: buildGo.external passes source paths inside the unpacked tree, so buildGo.package's dirOf (head srcs) lands in the directory that also holds the .binpb, where its go list invocation finds it. No change to nix/buildGo was needed and none should be made without re-checking this.

The result is a 21 MB statically linked closure whose only runtime references are tzdata, mailcap and iana-etc. No Go compiler, no ONNX Runtime shared object, no rust tokenizer archive.

Inference costs ~171 ms per string, single-threaded, batch of one. That is ~30 minutes for 10,000 posts. Fine for a one-off backfill, worth re-measuring batched before assuming it scales.

_ "github.com/gomlx/compute/gobackend" is load-bearing. The blank import registers the backend under the name "go" via init(). compute.NewWithConfig panics rather than returning an error when no backend is registered, so deleting the import fails at runtime, not at compile time.

The tokenizer does not truncate at all, and >512 tokens is a hard crash. This corrects an earlier belief — recorded here previously — that the pipeline truncates at 256. It does not truncate anywhere. hftokenizer accepts api.EncodeOptions{MaxLen: n} and ignores it:

MaxLen=8    ->  402 tokens   IGNORED
MaxLen=128  ->  402 tokens   IGNORED
MaxLen=256  ->  402 tokens   IGNORED

Not a configuration subtlety: With() stores the options struct (hftokenizer.go:272) and only AddSpecialTokens, IncludeSpans and IncludeSpecialTokensMask are ever read. The tokenizer.json shipped with the model declares its own truncation.max_length: 128, which is parsed into a json.RawMessage (types.go:12) and never used. Truncation is implemented for the rust tokenizer (hugot/backends/tokenizer_rust.go:71) but has no Go equivalent, so the pure-Go path silently has none.

Proving this needs care, because token counts alone cannot answer "was anything cut". The method: build two texts sharing an N-token prefix and diverging afterwards. If the tail were discarded, both would embed identically.

prefix   tokens   cosine     verdict
64       77       0.749367   tail seen
128      141      0.824462   tail seen
256      269      0.979436   tail seen
400      413      0.991603   tail seen

No cosine of 1.000000 at any length: nothing is ever cut. (The drift toward 1.0 is mean-pooling dilution over a growing shared prefix, not truncation — a useful check that the method measures what it claims.)

Past the model's 512 position embeddings it does not degrade, it panics:

 512 tokens -> ok, norm=7.9167
 522 tokens -> ERROR: dimension of axis #1 doesn't match ...
               got (Float32)[1, 513, 384] and (Float32)[1, 512, 384]

So fedisearch must truncate itself, at 510 (+ [CLS] + [SEP] = 512). This is a correctness fix, not tuning: without it one long post aborts the indexer. Truncate inside the special-token wrapper — keep ids[0] ([CLS]=101), take ids[1:511], re-append [SEP]=102. Slicing the tail off the finished array drops the [SEP] and feeds the model a malformed sequence.

Why 510 rather than the 256 the model was trained at: 256 is a soft limit where quality gains flatten, and sentence-transformers' max_seq_length: 256 reflects that. But cutting there would discard real content from long posts, whereas 510 only prevents the crash. Neither limit binds today — measured against 125 real posts the median is ~26 tokens, the 90th percentile ~79, a maxed-out 500-character Mastodon post ~104, and the longest observed post plus alt text ~250. Roughly one token per 4–5 characters of English prose. Cut at the hard limit, record the token counts, and let the telemetry say whether 256 ever mattered. It bites earlier for instances with raised character limits (Akkoma and Hometown commonly allow 5,000, or ~1,000 tokens) and for non-Latin scripts, which fragment far more aggressively.

Do not chunk long posts without overlap. Splitting a text and embedding the halves separately is not equivalent to embedding the whole, because the transformer only mixes tokens within a single forward pass. A fragment is embedded as though the rest of the text never existed. chunkdemo.go demonstrates this with measured cosine similarities:

"I went shopping for some movie that | talked about splattering his brains out"

vs query "a film recommendation":       vs query "graphic violence and gore":
  full sentence      0.3735               full sentence     0.1915
  tail chunk only    0.1342               tail chunk only   0.1997

The tail loses most of its association with films, and scores higher than the full sentence on violence — the chunk is not merely degraded but actively misleading, since without "movie" it reads as a real threat rather than a plot description. Averaging the two chunk vectors recovers only 0.8857 cosine against the true whole-sentence embedding.

Repeating a few words across the boundary largely fixes it (tail similarity to the film query rises 0.1342 → 0.3770 with four words of overlap), so if chunking is ever needed, overlap is not optional. Truncation is the gentler default for posts that state their topic early: keeping only the first sentence of a sample post scored 0.8415 against its topic versus 0.8020 for the whole thing.

You cannot inject a chunk's vector as a "prior" into the next chunk. The weights are fixed and identical for every input; changing them is training, not inference. What is actually available is either arithmetic on finished vectors, or prepending context as text and paying for another forward pass. Comparing them on whether the representation ranks the correct query above the wrong one (absolute cosines are not comparable across queries, but margins within one representation are):

                            film    violence   margin
tail alone                  0.1342   0.1997    -0.0655   ranks violence higher
mean(head, tail)            0.4049   0.2419    +0.1629
"About a movie." + tail     0.4278   0.2625    +0.1653
full sentence               0.3735   0.1915    +0.1820

The bare fragment has a negative margin: it prefers the wrong query. Both repairs restore the sign, and the cheap one is close to the expensive one, so vector averaging is a defensible fallback for ranking even though nothing is recomputed. What averaging cannot do is fix a fragment whose tokens are wrong, because per-token conditioning only happens inside a forward pass — the same token id gets measurably different vectors depending on its neighbours (0.9048 cosine for "bank" in "bank robbery" versus "bank holiday").

meanPool currently ignores the attention mask. That is only safe at batch size one. See Phase 2 — this is the single easiest way to produce embeddings that look plausible and retrieve badly.

Model choice mangles non-Latin scripts. Verified against the real Go tokenizer, not assumed:

"今天天气很好"  →  [CLS] [UNK] 天 天 [UNK] [UNK] [UNK] [SEP]

Four of six characters are destroyed. all-MiniLM-L6-v2 inherits bert-base-uncased's vocabulary, which contains 244 CJK characters out of the ~20,000 in use. Chinese, Japanese, Korean, Arabic, Hebrew, Greek, Cyrillic and Thai posts will embed to near-noise. Emoji become [UNK] too. Latin-script European languages degrade gracefully ("Grüße" → gr ##u ##ße, umlauts stripped, meaning mostly intact).

The decision is to ship MiniLM anyway and record the [UNK] ratio per embedding, so the limitation is visible in the data instead of silent. The schema keys embeddings by (model, space) precisely so a multilingual model can be added later without migration. Candidates when that time comes: paraphrase-multilingual-MiniLM-L12-v2 or multilingual-e5-small — both 384 dims, both ~470 MB instead of 90 MB, both SentencePiece rather than WordPiece (check go-huggingface's tokenizers/sentencepiece package supports what we need before committing).

Goal: fedisearch add <handle> then fedisearch ingest puts real toots in SQLite. No embeddings, no search. Ends with a stats command reporting the actual shape of the corpus — language distribution, image counts, boost counts, text length percentiles — which is the data needed to sanity-check the Phase 2 model choice.

SQLite via modernc.org/sqlite (pure Go, no cgo, already in the shared go-deps.nix:442). Database at $XDG_DATA_HOME/fedisearch/index.db.

Follow inventory/main.go's migration pattern: a []struct{version int; sql string} applied in order, tracked in a schema_version table, never edited once shipped — add a new migration instead.

Follow notenlesen/main.go:141 for opening the database: pragmas travel in the DSN, not a post-open db.Exec, because database/sql pools connections and a post-open pragma configures only whichever connection happened to run it.

sql.Open("sqlite", "file:"+path+"?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL)")


CREATE TABLE actor (
  id                INTEGER PRIMARY KEY AUTOINCREMENT,
  handle            TEXT NOT NULL UNIQUE,   -- @user@host, as typed
  actor_url         TEXT NOT NULL UNIQUE,   -- resolved AP id
  outbox_url        TEXT NOT NULL,
  display_name      TEXT NOT NULL DEFAULT '',
  icon_url          TEXT NOT NULL DEFAULT '',
  added_at          INTEGER NOT NULL,
  last_ingest_at    INTEGER,
  newest_ap_id      TEXT,        -- for incremental walks downward from the top
  oldest_ap_id      TEXT,        -- resume point for an interrupted backfill
  backfill_complete INTEGER NOT NULL DEFAULT 0
);

CREATE TABLE post (
  id            INTEGER PRIMARY KEY AUTOINCREMENT,
  actor_id      INTEGER NOT NULL REFERENCES actor(id) ON DELETE CASCADE,
  ap_id         TEXT NOT NULL UNIQUE,   -- the Note's id; idempotency key
  url           TEXT NOT NULL DEFAULT '',
  published     INTEGER NOT NULL,       -- unix seconds
  kind          TEXT NOT NULL,          -- 'create' | 'announce'
  content_html  TEXT NOT NULL DEFAULT '',
  content_text  TEXT NOT NULL DEFAULT '',
  summary       TEXT NOT NULL DEFAULT '',  -- content warning
  sensitive     INTEGER NOT NULL DEFAULT 0,
  language      TEXT NOT NULL DEFAULT '',  -- from contentMap key
  in_reply_to   TEXT NOT NULL DEFAULT '',
  conversation  TEXT NOT NULL DEFAULT '',
  audience      TEXT NOT NULL DEFAULT '',  -- 'public' | 'unlisted'; see ingest
  announce_url  TEXT NOT NULL DEFAULT '',  -- Announce target, unresolved
  resolved_at   INTEGER
);
CREATE INDEX idx_post_actor     ON post(actor_id);
CREATE INDEX idx_post_published ON post(published DESC);
CREATE INDEX idx_post_kind      ON post(kind);

CREATE TABLE attachment (
  id         INTEGER PRIMARY KEY AUTOINCREMENT,
  post_id    INTEGER NOT NULL REFERENCES post(id) ON DELETE CASCADE,
  url        TEXT NOT NULL,
  media_type TEXT NOT NULL DEFAULT '',
  alt_text   TEXT NOT NULL DEFAULT '',   -- the AP `name` field
  blurhash   TEXT NOT NULL DEFAULT '',
  width      INTEGER,
  height     INTEGER,
  local_path TEXT NOT NULL DEFAULT '',   -- filled in Phase 2
  fetched_at INTEGER,
  UNIQUE(post_id, url)
);
CREATE INDEX idx_attachment_post ON attachment(post_id);

-- One row per embedding model actually used. `space` is the shared vector
-- space: two models may write into the same space (rare) or their own.
CREATE TABLE model (
  id         INTEGER PRIMARY KEY AUTOINCREMENT,
  name       TEXT NOT NULL,           -- e.g. KnightsAnalytics/all-MiniLM-L6-v2
  revision   TEXT NOT NULL DEFAULT 'main',
  kind       TEXT NOT NULL,           -- 'text' | 'image'
  space      TEXT NOT NULL,           -- e.g. 'minilm-l6-v2' | 'clip-vit-b32'
  dims       INTEGER NOT NULL,
  max_tokens INTEGER NOT NULL DEFAULT 0,
  sha256     TEXT NOT NULL DEFAULT '',
  local_dir  TEXT NOT NULL,
  added_at   INTEGER NOT NULL,
  UNIQUE(name, revision, kind)
);

-- Vectors are little-endian float32, L2-normalized (see Phase 2).
-- target_kind lets one table serve posts, attachment alt texts and images.
CREATE TABLE embedding (
  target_kind TEXT NOT NULL,      -- 'post' | 'alt_text' | 'image'
  target_id   INTEGER NOT NULL,   -- post.id or attachment.id
  model_id    INTEGER NOT NULL REFERENCES model(id) ON DELETE CASCADE,
  space       TEXT NOT NULL,      -- denormalized from model, to query without a join
  dims        INTEGER NOT NULL,
  vec         BLOB NOT NULL,
  unk_ratio   REAL NOT NULL DEFAULT 0,  -- fraction of [UNK]; flags degraded rows
  -- Truncation telemetry. Both counts are stored, not just a flag, so the
  -- question "how much was lost" is answerable and not merely "was anything".
  input_tokens INTEGER NOT NULL DEFAULT 0,  -- tokens the text produced
  used_tokens  INTEGER NOT NULL DEFAULT 0,  -- tokens actually fed to the model
  truncated    INTEGER NOT NULL DEFAULT 0,  -- used < input
  input_chars  INTEGER NOT NULL DEFAULT 0,
  created_at  INTEGER NOT NULL,
  PRIMARY KEY (target_kind, target_id, model_id)
);
CREATE INDEX idx_embedding_space ON embedding(space, target_kind);

Four deliberate choices worth keeping:

Both content_html and content_text are stored. HTML extraction will need fixing, and keeping the original means re-extraction never requires re-fetching 10k posts from someone else's server.

embedding is keyed by (target, model). One post can carry many embeddings from different models in different spaces. Adding a multilingual model, or CLIP, needs no migration. Vectors from different spaces must never be compared.

unk_ratio is stored per embedding. A post at 0.7 has not been meaningfully indexed. Without this the tokenizer limitation is invisible.

Both input_tokens and used_tokens are stored, rather than a lone truncated flag, so the corpus can answer how much text is actually being lost and whether the soft 256 limit would have mattered:

SELECT count(*), max(input_tokens) FROM embedding WHERE input_tokens > 256;

Since we truncate at 510 rather than 256, this telemetry is what would justify revisiting that choice — or a chunking strategy, given what chunkdemo.go showed about fragments.

booster-bot/main.go:657 strips tags with regexp.MustCompile("<[^>]*>"). That is fine for finding a 4-digit PIN, and wrong here. Mastodon splits URLs across spans with class="invisible":

input:   <p>Burna Boy – Giza</p><p><a href="…"><span class="invisible">https://www.</span>
         <span class="ellipsis">youtube.com/watch?v=bWxyVF1LJA</span><span class="invisible">o&amp;t=23</span></a></p>

regex, tags→" ":  "Burna Boy – Giza https://www. youtube.com/watch?v=bWxyVF1LJA o&t=23"
regex, tags→"":   "Burna Boy – Gizahttps://www.youtube.com/watch?v=…"   ← words glued
correct:          "Burna Boy – Giza youtube.com/watch?v=bWxyVF1LJA"

Both regex variants produce junk tokens that waste the token budget and pollute the embedding. Use golang.org/x/net/html (already vendored, go-deps.nix:545), and:

Verified: mentions and hashtags come out clean (<a …>@<span>bob</span></a> → @bob, #nix likewise), so no special handling is needed for them.

fedisearch add @Profpatsch@mastodon.xyz
fedisearch list
fedisearch ingest [--actor HANDLE] [--delay 500ms] [--max-pages N]
fedisearch stats

add resolves the handle by WebFinger (/.well-known/webfinger?resource=acct:user@host), fetches the actor document with Accept: application/activity+json, and stores outbox_url.

ingest walks the outbox as a paginated OrderedCollection. Confirmed working unauthenticated against mastodon.xyz: 10,683 items, 20 per page, next carrying a max_id cursor and prev a min_id one. Cursors are stable, so:

Filter on Public ∈ to ∪ cc, never to alone. Measured over 120 real activities:

audience addressing count
unlisted to: [followers], cc: [Public] 119
public to: [Public] 1

The obvious predicate — "index items addressed to Public" — captures one post in 120, and presents as a broken ingest rather than a wrong condition. Both forms are world-readable; unlisted merely means "keep out of public timelines", which is Mastodon's default for most posting. Store which one it was in an audience column ('public' | 'unlisted') so the distinction survives.

Two activity types matter:

Idempotent on ap_id via INSERT … ON CONFLICT(ap_id) DO UPDATE, so re-running is safe and an interrupted run resumes from the stored cursor.

What the corpus looks like, measured over 65 notes, since it shapes Phase 2:
median post length 96 chars
90th percentile 460 chars
under 60 chars 22 / 65
empty after HTML extraction 3 / 65 (all image-only)
replies (of 18 sampled) 11
attachments carrying alt text 24 / 24
alt text length, short posts 277–947 chars

Two consequences. Short text is not a problem in itself — MiniLM was trained on sentence pairs and embeds a 20-character post fine. But empty text must not be embedded (see Phase 2), and reply fragments like "@RosaCtrl Ah, not bad!" are not thin so much as dependent: the meaning is in the parent, and no model recovers information that is not in the input. in_reply_to is stored from the start so thread context stays possible without re-ingesting.

One attachment in the sample was video/mp4. CLIP cannot embed video, so Phase 2 must check media_type rather than assume images.

Politeness: 500 ms between pages by default (≈4.5 minutes for a 10k outbox), honour 429 and Retry-After, send a real User-Agent identifying the tool.

main.go (CLI), db.go (schema + migrations), activitypub.go (WebFinger, outbox walking, HTML extraction), ingest.go. spike.go stays untouched and still builds. Nix: add modernc-sqlite and golang-x-net.html to default.nix from the shared deps.

fedisearch stats reports real numbers over your own outbox: post count, boost count, attachment count, how many attachments have alt text, language distribution, and text length percentiles.

Goal: every post, alt text and image has a vector in the database.

spike.go becomes embed.go, keeping the model loading and tensor plumbing but fixing what batch-of-one let it dodge.

Fix meanPool first. It currently averages over every position. With batching, short sentences get padded to the batch's longest, and padding positions produce garbage vectors that drag the mean toward zero. Divide by the count of real tokens, using the attention mask:

sum over t where mask[t]==1, then divide by count(mask==1)

This is the single most likely source of "embeddings that look fine and retrieve badly". Guard it with a test: embedding one string alone and the same string inside a padded batch must produce the same vector.

Then normalize. sentence-transformers' own modules.json lists Transformer → Pooling → Normalize, and 1_Pooling/config.json confirms pooling_mode_mean_tokens: true. The spike deliberately stops before normalizing so it can compare against hugot's unnormalized reference vector; production should store L2-normalized vectors, because then cosine similarity is a plain dot product (see Phase 3).

Keep spike.go's assertion working as a regression test — against the unnormalized pooled output — since it is the only ground truth we have.

Truncate, because nothing else will. See the tokenizer trap above: no layer in this stack enforces a length, and exceeding 512 positions panics inside gomlx rather than degrading. After encoding with special tokens, if len(ids) > 512, keep ids[0], take ids[1:511] and re-append [SEP]. Assert len(ids) <= 512 before building tensors, so a future model with different limits fails loudly at the boundary rather than deep inside a graph compilation error.

Batch. Measure before choosing a size; 171 ms/post single-threaded means ~30 min for 10k posts, and batching should improve throughput substantially. Note the Go backend does no per-shape JIT, so shape bucketing is pointless here (hugot disables it for this backend too).

Record per embedding: unk_ratio (token IDs equal to the [UNK] id, 100 for this vocabulary, over the token count), plus input_tokens, used_tokens, truncated and input_chars.

Skip empty text. Measured on 65 real posts, three strip to "" — they are image-only. Embedding one yields the [CLS]/[SEP] vector alone, which is near-identical for every empty post, so they cluster together and surface for any query. Store the post (Phase 2's image embedding reaches it); write no text embedding.

Three targets, in two spaces:

target_kind source space
post content_text, prefixed with summary if set minilm-l6-v2
alt_text attachment.alt_text minilm-l6-v2
image the downloaded image clip-vit-b32

Alt text is embedded as its own unit, not appended to the post body. Two reasons: a 300-character generated alt text would otherwise dominate a twenty-word toot, and keeping them separate means a result can say the image matched, not just the post matched. (Your alt texts are unusually rich because mastodon-alt-text writes them, so this is a real signal.)

Content warnings are indexed, not hidden: summary is prepended to the post text so CW'd posts stay findable, and shown in results so they are not jarring.

Model: Xenova/clip-vit-base-patch32, onnx/vision_model_quantized.onnx (89 MB) plus preprocessor_config.json. 512 dims.

Preprocessing, per CLIP's preprocessor_config.json (do not invent these numbers):

  1. resize shortest edge to 224, preserving aspect
  2. center-crop 224×224
  3. rescale by 1/255
  4. normalize with mean [0.48145466, 0.4578275, 0.40821073] and std [0.26862954, 0.26130258, 0.27577711]
  5. layout NCHW

Use a proper resampler. hugot's imageutil.resizeImage is nearest-neighbour, but CLIP's config specifies "resample": 3 (bicubic). Nearest-neighbour measurably degrades image embeddings. Use golang.org/x/image/draw with CatmullRom.

Mastodon serves WebP heavily, so import golang.org/x/image/webp for decoding (decode-only; there is no encoder, which matters if thumbnails are ever cached).

Images are downloaded to $XDG_CACHE_HOME/fedisearch/media/, with local_path recorded on the attachment row. Rate-limit per host, same as ingest.

Models download on first use to $XDG_CACHE_HOME/fedisearch/models/<name>/, and a row lands in model recording name, revision, dims, sha256 and local dir. Do not use go-huggingface's hub package: it exists in the closure only because hftokenizer.New takes a *hub.Repo, and fedisearch calls NewFromContent instead. Downloading is three HTTP GETs; own that path, since checksums and revisions need recording anyway.

Keep downloadFile's write-to-.partial-then-rename: ensureModel decides "already present?" by existence and non-zero size, so an interrupted download would otherwise masquerade as complete forever.

fedisearch embed [--model NAME] [--batch N] [--kind text|image] [--limit N]

Idempotent: skip targets that already have a row for that model. Re-runnable after ingesting more posts.

Goal: type a query, get ranked posts and images.

Brute-force cosine over a flat []float32 arena loaded at startup. At this scale this is the right answer and an ANN index would be pure overhead: 10k posts × 384 dims × 4 bytes ≈ 15 MB, and a full scan is well under a millisecond.

Because vectors are stored normalized, cosine similarity is the dot product — no square roots, no division, and a contiguous float32 loop the compiler vectorizes well. (On the unit sphere, distance² = 2 − 2·cosine, so ranking by largest dot product and by smallest euclidean distance give identical orderings. Verified.)

Query flow: embed the query text once per space — MiniLM for text and alt-text targets, CLIP text encoder for the image space — then rank each space separately and merge. Never compare vectors across spaces; they are unrelated coordinate systems.

Note this means image search needs CLIP's text encoder (onnx/text_model_quantized.onnx, 64 MB) in addition to its vision encoder, because that is what puts a text query into the same 512-dim space as the images. CLIP's text side truncates at 77 tokens, which is fine for queries.

Web only, localhost. net/http + html/template, no JavaScript framework. A search box; results as cards showing text, author, date, a link to the original, and image thumbnails. Show the content warning where present. Show the similarity score — it makes bad results diagnosable rather than mysterious.

Consider surfacing unk_ratio on a result when high; it explains directly why a non-English post ranked oddly.

fedisearch serve [--addr localhost:PORT]

buildGo.program as established in Phase 0. Add modernc-sqlite, golang-x-net.html, golang-x-image.draw and golang-x-image.webp to default.nix — all already in the shared go-deps.nix.

A fedisearch.service user unit running fedisearch serve bound to localhost (follow mastodon-alt-text.service for style), a fedisearch.1 manpage, and packages.fedisearch in flake.nix alongside the existing fedisearch-spike.

Installation follows the repo convention: nix profile install .#fedisearch, copy the unit to ~/.config/systemd/user/, systemctl --user enable --now.