Profpatsch/users/Profpatsch/whatcd-resolver
- resources/ 6.2 KiB · 2 files
- services/ 698 B · 2 dirs, 4 files
- src/ 199.7 KiB · 9 files
- static/ 5.5 KiB · 2 files
- templates/page.html 3.1 KiB
- .NOTES 7.9 KiB
- .gitignore 174 B
- Main.hs 93 B
- README.md 417 B
- assets.go 8.0 KiB
- assets_test.go 2.4 KiB
- bencode.go 8.5 KiB
- bencode_test.go 9.0 KiB
- build.ninja 447 B
- coverart.go 8.0 KiB
- db.go 7.8 KiB
- db_test.go 6.0 KiB
- default.nix 5.3 KiB
- go-deps.nix 7.5 KiB
- go.mod 817 B
- go.sum 4.5 KiB
- handlers.go 23.4 KiB
- main.go 8.0 KiB
- notes.org 1.7 KiB
- otel.go 12.5 KiB
- otel_test.go 5.9 KiB
- queries.go 17.0 KiB
- queries_test.go 9.1 KiB
- redacted.go 31.4 KiB
- redacted_test.go 8.3 KiB
- releasetype.go 4.2 KiB
- server-notes.org 26 B
- transmission.go 8.1 KiB
- whatcd-resolver.1 7.1 KiB
- whatcd-resolver.cabal 3.0 KiB
whatcd-resolver
To run:
ninja run-services
in one terminal (starts the background tasks)
ninja run
to start the server. It runs on 9092.
You need to be in the nix-shell in ./...
You need to set the pass key internet/redacted/api-keys/whatcd-resolver to an API key for RED.
You need to have a transmission-rpc-daemon listening on port 9091 (no auth, try ssh port forwarding lol).
whatcd-resolver(1)
NAME
whatcd-resolver - browse, rank and download music from a Gazelle tracker
SYNOPSIS
whatcd-resolver-go
DESCRIPTION
whatcd-resolver is a web UI over the Redacted API. It mirrors search results into PostgreSQL, ranks the torrents of each release group so that one of them can be called "the best", hands that one to a local Transmission daemon, and then streams the downloaded audio files and cover art back out over HTTP.
There is no command line interface: all configuration is through the environment, and everything else happens in the browser.
Two implementations
This directory contains two implementations of the same program. The original is in Haskell (Main.hs, src/*.hs), built as the whatcd-resolver attribute; the Go port is in the top-level *.go files, built as whatcd-resolver-go. They share no code, only static/. Both builds list their sources explicitly, so neither sees the other's files.
This page documents the Go port. The most visible differences from the Haskell version are that it does not start its own PostgreSQL (see DATABASE) and that it does not export traces over the OTLP protobuf encoding (see TRACING). Everything the user sees is intended to be identical, so the two can be run against the same database and compared.
ENVIRONMENT
WHATCD_RESOLVER_DATABASE_URL
PostgreSQL connection URL, e.g. postgres://user@localhost:5433/whatcd_resolver. Required; the process exits immediately without it.
WHATCD_RESOLVER_REDACTED_API_KEY
API key for Redacted. If unset, pass(1) is consulted for internet/redacted/api-keys/whatcd-resolver, as in the Haskell version.
WHATCD_RESOLVER_LISTEN
Address to bind, default 127.0.0.1:9094. The Haskell version listens on 9093; the default differs on purpose so both can run at once.
WHATCD_RESOLVER_TRANSMISSION_HOST, WHATCD_RESOLVER_TRANSMISSION_PORT
Transmission RPC endpoint, default localhost and 9091.
WHATCD_RESOLVER_TRANSMISSION_DOWNLOAD_DIRECTORY
Where Transmission puts its downloads. When unset or not a directory, file streaming and cover art are disabled and the rest of the UI still works.
WHATCD_RESOLVER_TOOLS
Directory of helper binaries; only exiftool is looked up there. Falls back to
PATH.
OTEL_EXPORTER_OTLP_ENDPOINT
Trace collector, default http://localhost:4318.
OTEL_SERVICE_NAME
Service name in traces, default whatcd-resolver.
WHATCD_RESOLVER_DISABLE_TRACING
When set to any value, no traces are exported and no collector is contacted.
DATABASE
The Haskell version starts a private PostgreSQL 14 through
tmp-postgres,
under
$XDG_DATA_HOME/whatcd-resolver/database
on port 5431.
The Go port does not: it connects to a server that is already running, named by
WHATCD_RESOLVER_DATABASE_URL.
The schema is applied at every startup and is idempotent. It is ported verbatim, which matters more than it might appear: seeding_weight and artist_ids are STORED generated columns computed by a plpgsql(7) function from the API JSON. Changing that function, or the JSON that feeds it, silently re-ranks every torrent in the database rather than failing. For the same reason the JSON normalisation in the API client (renaming "snatched to "snatches, and the per-endpoint torrent id key to "torrentId""") is load-bearing and must not be tidied away.
Development database
The authoritative data lives on haku. To work against a copy of it locally, dump it read-only and restore it into a local cluster:
ssh haku 'sudo -u whatcd-resolver pg_dump \
-h /var/lib/whatcd-resolver/.local/share/whatcd-resolver/database-socket \
-p 5431 -d postgres -Fc -Z6 -f /tmp/whatcd-dump.pgc'
scp haku:/tmp/whatcd-dump.pgc ./tmp/
pg_restore -d whatcd_resolver --no-owner --no-privileges ./tmp/whatcd-dump.pgc
The deployed database contains two structures that no code refers to: redacted.torrent_artists, a denormalised artist-to-torrent join table that is roughly half stale, and redacted.artist_filter_cache, a materialised view of the artist filter subqueries. Both are abandoned optimisation experiments and should be dropped from any restored copy.
The tests skip themselves unless
WHATCD_RESOLVER_DATABASE_URL
is set, so
go test ./...
works on a fresh checkout.
With it set, they run against the restored copy, checking among other things
that the migration does not disturb existing data and that every stored
.torrent
still decodes.
TRACING
Spans are exported to an OTLP collector, in practice the jaeger-all-in-one that runs beside the service. Attribute names carry a "_." prefix, as in the Haskell version, so existing saved queries keep matching.
The OTLP payload is encoded as JSON by otel.go rather than using the official exporter. That exporter imports go.opentelemetry.io/proto/otlp, which pulls in grpc, genproto and grpc-gateway (about 45 packages) even when only the HTTP transport is used, and every one of those would have to be pinned by hand in go-deps.nix, since buildGo(7) does no module resolution. The JSON encoding of OTLP is part of the specification and Jaeger accepts it on the same endpoint, so the dependency buys nothing here.
SEE ALSO
transmission-daemon(1), buildGo(7)
The Redacted API is documented on the tracker's own wiki, which requires an account.
AUTHORS
Profpatsch
CAVEATS
-
htmx and howler are fetched from the network at startup and their SHA-384 computed, so the process needs network access to start and will fail without it. A failed prefetch is fatal, which is not hypothetical: the third asset, stylize.css, had to be vendored into resources/ because its upstream repository was deleted from GitHub, at which point neither implementation could start any more. The deployed instance survived only because it had been running since before the deletion and still held the file in memory, which is where the vendored copy was recovered from. See resources/README.md.
Note what the integrity attribute does and does not do for the two assets that are still fetched: the hash is computed from whatever was just downloaded, so it guarantees the browser sees the same bytes the server did, but it does not pin the upstream content. Pinning would mean writing the expected hash next to the URL and refusing to start on a mismatch.
-
Recommendations are effectively untested against real data: redacted.similar_artists is empty in the deployed database, so the feature has never run there. The query is ported verbatim and is known to execute, but its output has not been compared against the Haskell version's.
-
Torrents removed from Transmission are treated as never downloaded: both the hash and the stored .torrent are cleared, and the UI offers to fetch the file again.
-
Fetching recommendations makes one API call per favourite artist, serially, and can take minutes. It is deliberately synchronous, as in the Haskell version.