Status of the three-phase plan to turn hosty into a self-hosted app platform: drop a .hosty file on a machine, get a sandboxed service on its own subdomain.

Cross-cutting: the environment database (hosty.db) replaced the per-app runtime.hosty, fixing a case where apps owned their own secret store. See "The environment database" below.

Known limitation, orthogonal to the phases: system-mode apps do not survive a reboot (the attach and the companion unit live in tmpfs). See the section after Phase 1c.

Picking this up fresh? Jump to "Next session" at the end of Phase 2; it says what state things are in, what to do next, and which traps are already known.

This document records why things are the way they are. Mechanics of the shipped CLI belong in --help; the non-obvious constraints we hit belong here.

hosty start has two modes:

user mode (default) system mode (-system)
runs as --user unit, your uid root portable service, DynamicUser
isolation none full portable-service profile
state $XDG_STATE_HOME/hosty /var/lib/hosty
purpose quick local dev running arbitrary apps

User mode is the fast iteration loop and is deliberately unsandboxed. System mode is the real deployment target: hosty is meant to run arbitrary third-party apps, so containment is not optional. That is why the portable-service default profile is used and not trusted — default brings DynamicUser, ProtectSystem=strict, PrivateDevices, a SystemCallFilter, and friends. -profile exists as a per-app escape hatch, but the default must stay sandboxed.

A .hosty file already contained an OS-ish tree (usr/, etc/, an os-release carrying PORTABLE_PREFIXES=), so portablectl was the mechanism that matched the artifact we already had. portablectl attach copies the unit onto the host and layers a RootDirectory= drop-in, giving us image isolation without hosty implementing containers itself.

We shell out to portablectl/systemctl rather than using the org.freedesktop.portable1 D-Bus API. AttachImage() only installs units — it neither starts nor enables them — so D-Bus would still require a second connection to the service manager. The CLI does both, and hosty already shells out to systemctl elsewhere.

Revisited after the system-mode bugs below, since "the CLI is a lossy wrapper" was the obvious suspect. It is not, and the measurements are worth keeping so this is not re-litigated:

Varlink is not available. portabled exposes no varlink interface; only io.systemd.{Hostname,Login,ManagedOOM,Manager} exist. D-Bus is the only API.

D-Bus would not have prevented any of the three bugs. The worst of them — is-attached reporting another image's state — is a defect in systemd's GetImageState, not in portablectl:

$ busctl call … GetImageState s "does-not-exist-xyz"   → s "running-runtime"
$ portablectl is-attached does-not-exist-xyz           → running-runtime

Both transports agree, because the CLI is a faithful wrapper. Notably GetImageOSRelease and GetImage do fail correctly (No image … found), so the looseness is specific to that one method; ListImages is authoritative either way, which is what hosty parses now.

ReattachImage() is a trap. It documents exactly what hosty wants — reattach while units are running, no stop/detach/attach dance. Against a live app it failed twice: with our drop-in present, Directory not empty; with the drop-in removed, No unit files associated … Image not attached? — and it detached the image and deleted its unit while the service kept running, leaving the same broken half-attached state that took a full stop/start to recover from. It fails destructively and non-atomically, so it is worse here than the CLI path.

What D-Bus would buy is the structured changes array returned by Attach/Detach (copy/symlink/write/mkdir triplets): hosty would know exactly which paths portabled created instead of hardcoding them. That is a real improvement and the only argument that still stands — worth taking up in Phase 3, which wants progress signals anyway. godbus/dbus/v5 is already vendored in the depot, so the dependency is not the obstacle.

Each of these cost real debugging time; they are the reason the code looks the way it does.

The unit is synthesized at deploy time, not packed. hosty pack no longer embeds a .service file. Two reasons. First, everything the unit needs is already in _hosty_meta (ExecStart, profile, env aliases), so packing it duplicated a source of truth and user mode ignored the packed copy anyway. Second and decisively: NixOS drops the FHS unit search paths, so systemd-portabled never scans <image>/usr/lib/systemd/system. A unit packed there is invisible and attach fails with "Couldn't find any matching unit files". hosty now writes the unit into <image>/etc/systemd/system (scanned on NixOS) and usr/lib/systemd/system (for FHS hosts), just before attach.

/nix/store used to be bound into the namespace. Nix-built binaries are dynamically linked against their closure — interpreter and libs live in /nix/store, which RootDirectory= hides — so the drop-in carried BindReadOnlyPaths=-/nix/store or the app could not exec at all. This was reversed in Phase 1b; see below for why the reasoning was wrong.

Writable data uses StateDirectory=. The profile sets ProtectSystem=strict, so the app cannot write into its own image. The systemd-native answer for "immutable image, local data" is StateDirectory=, which is bound in writable while everything else stays read-only. With DynamicUser the directory really lives at /var/lib/private/hosty/<name> and is exposed at the normal path via symlink plus bind mount; hosty runs as root and so still reaches it. An earlier attempt to hand-manage this with a shared unix group was removed — it was solving a misdiagnosed problem (see below).

The FUSE mountpoint lives under /run, not in the state dir. _hosty_fs is mounted by a companion <name>-fs.service, a separate unit from the app. If the mountpoint sat inside the app's StateDirectory, the two units would disagree about the path the moment DynamicUser relocation kicked in (one sees /var/lib/hosty/..., the other /var/lib/private/hosty/...), and unmount fails. RuntimeDirectory= is never relocated, so the mountpoint is /run/hosty/<name>/fs and both units agree.

FUSE needs allow_other. The mount is created by root (fs-serve) but read by the app's transient DynamicUser. FUSE denies foreign uids by default, so system mode passes -allow-other; this requires user_allow_other in /etc/fuse.conf (programs.fuse.userAllowOther = true on NixOS). User mode does not need or use it.

Drop-in ordering matters. portablectl writes 10-profile.conf (the security profile) and 20-portable.conf (RootDirectory=). Later drop-ins win, so hosty's must sort last — hence 50-hosty.conf. It was originally 10-hosty.conf, where the profile silently overrode our settings.

Unit lifecycle coupling. The app BindsTo= + After= its -fs unit, so it never runs without its filesystem and starts after it. The -fs unit is PartOf= the app, so stopping or restarting the app tears the mount down too. Together the pair lives and dies as a unit in both directions.

Detach ordering. portablectl detach fails with "Directory not empty" if anything is still live. Cleanup therefore stops the app, stops the fs unit, waits for the mount to go, removes hosty's own drop-in, and only then detaches, with a short retry for the asynchronous teardown.

Read-only .hosty files. A pre-existing bug surfaced by the test: openReadOnlyDB requested journal_mode(WAL), but switching a database to WAL must create a -shm sidecar next to it — impossible in the Nix store. Any .hosty on read-only media failed to open. Fixed with mode=ro&immutable=1.

Debugging lesson. The read-only failure surfaced as unable to open database file (14) right after some nobody:nogroup ownership appeared, which looked exactly like a permission problem and led to ~50 lines of unnecessary group-management code. The ownership was a red herring. When two suspicious things co-occur, confirm which one actually fails before building on the theory.

system-test.nix is a NixOS VM test asserting the properties that matter: attach succeeds, the app runs as a non-root DynamicUser, it serves HTTP, HOSTY_FS is writable from inside the sandbox, data survives stop/start, and teardown leaves no attached units or stale mounts. It also guards the read-only .hosty regression and that user mode still synthesizes a working unit.

nix-build users/Profpatsch/hosty/system-test.nix

A second node (cgo) covers rss-parrot — see Phase 1c. It runs two apps at once, which is what a host doing its job looks like and what several bugs needed to become visible at all. That node configures itself through nixos-module.nix rather than repeating its settings, so the module is under test too: if it stops providing hosty, allow_other or the state dir, the node fails the same way legosi would.

LoadCredential is still uncovered — no app in the tree both declares sensitive config and builds. pocket-id (ENCRYPTION_KEY_FILE) is the natural target. vaultwarden cannot serve as one: vaultwarden-hosty does not build, because default.nix passes -setup-exec and hosty pack has never accepted such a flag (git log -S confirms it was never implemented). The first-run setup wizard it implies is a feature to build, not a typo to fix.

Phase 1 shipped a sandbox with a hole in it. The .hosty promise is "drop the file on a machine, get a service", but with BindReadOnlyPaths=-/nix/store an image only ran where the right store paths happened to already exist. Nothing in the file recorded which paths those were, so a garbage collection on the host could break an installed app with no diagnostic. And since the point of system mode is running arbitrary third-party apps, handing each of them a read-only view of the entire host store also handed them the machine's full inventory — every package and version, plus any secret that had leaked into a store path.

The Phase 1 note argued that because vaultwarden and rss-parrot are CGO SQLite and "irreducibly dynamic", the bind was unavoidable — "no amount of making hosty itself static would avoid it". The premise is true and the conclusion does not follow. Dynamic linkage requires the libraries to be present at their build-time absolute paths; it does not require them to come from the host. Store paths are content-addressed, so the same closure placed at the same paths inside the image satisfies the loader identically.

closureMap (in default.nix) does exactly that: closureInfo yields the transitive closure, which becomes -map entries placing each store path at its own path in the image. No new hosty machinery was needed — -map @file and absolute destinations already worked.

CA certificates, and they fail silently. Nix's openssl ships an empty $out/etc/ssl/certs and falls back to the compiled-in /etc/ssl/certs/ca-certificates.crt — a path NixOS supplies and no package does, so it appears in no closure. The dangerous part is the failure mode: SSL_CTX_set_default_verify_paths() returns success whether or not it found anything. Measured in a chroot containing only vaultwarden's closure:

with the bundle:     set_default_verify_paths=1  loaded_certs=167
without the bundle:  set_default_verify_paths=1  loaded_certs=0

Every outbound HTTPS request would fail verification at runtime with nothing to point at. caBundleMap packs the bundle for apps that make outbound calls (vaultwarden fetches site icons, rss-parrot federates); asciinema-server only serves and needs none.

Go runtime data. tzdata, mailcap and iana-etc are read opportunistically by the stdlib for timezones, MIME types and /etc/services. closureInfo catches them automatically where a hand-maintained list would not.

Chasing the closure exposed a comment in default.nix claiming hosty was "static, pure-Go, no CGO". It was none of those: importing net made the stdlib select the cgo resolver, which drags in the dynamic loader, so the binary needed glibc from the host. The claim was true of the source — the SQLite driver is modernc.org/sqlite, a Go translation, and go-fuse talks to /dev/fuse directly — and false of the binary. Building buildGo's stdlib with CGO_ENABLED=0 (644ad561) made it true, and hosty's closure fell from 49MB to 17.7MB.

So only rss-parrot packs a closure today; it is genuinely CGO (buildGoModule with CGO_ENABLED=1). hosty-hello and asciinema-server are statically linked buildGo binaries, and pocket-id is an upstream static release build — none of them pack anything.

While there, 1951d4ed fixed three bugs that had made every Go binary in the tree retain its own sources and the 227MB Go distribution as runtime references: go tool compile -trimpath takes a semicolon-separated list rather than repeated flags (so the existing -trimpath=$PWD -trimpath=${go} silently discarded the first), sources were compiled at their store paths so $PWD never matched anyway, and GOROOT_FINAL has done nothing since Go 1.20. source-forge's closure went from 584MB to 22MB. Debug information is unaffected — symbol tables are complete and panics still report file and line, now relative.

system-test.nix asserts the drop-in never mentions /nix/store and that the running app cannot see the host store, read through /proc/<pid>/root rather than by entering the namespace (a self-contained image has no shell to enter with — an earlier attempt with nsenter -- sh failed for exactly that reason). It reports 0 store entries against the host's 659.

The negative assertion is the load-bearing one. Without it the section would pass just as happily against a host store that happened to contain the right paths, i.e. for entirely the wrong reason. It was verified by reintroducing the bind and confirming the test fails.

nixos-module.nix provides four things and deliberately no service of its own: hosty is a CLI, not a daemon, and apps are installed by running it as root. programs.fuse.userAllowOther = true is the load-bearing option — FUSE allow_other fails without user_allow_other in /etc/fuse.conf, and legosi had it commented out. systemd-portabled indeed needs no wiring: NixOS ships it in its default units whenever systemd is built withPortabled, which was confirmed by eval rather than assumed.

There is no declarative app management, on purpose. baseDomain exists as an option from the start even though nothing reads it yet, because the production label is undecided and hosty-test is meant to be thrown away.

rss-parrot was added to system-test.nix as a second VM node before legosi was touched. It is the counterpart to hosty-hello in the one way that matters: buildGoModule with CGO_ENABLED=1, so it is dynamically linked against an ELF interpreter at an absolute store path. Under RootDirectory= with no host store bound in, it can only reach execve() if the image carries that path itself. Until now the closure-packing path had only ever been exercised by a build.

Measured on legosi, which is the number worth keeping:

rss-parrot   9 store entries in its namespace   host has 10197
hosty-hello  0 store entries                    (static, packs nothing)

Both serve, both run as their own DynamicUser, and neither can see the host store. rss-parrot writes all six of its state files through FUSE, including the RSA keypair, and that identity survives a restart unchanged — the one thing that must never be regenerated, since remote servers cache the public half.

LoadCredential remains untested. The plan claimed rss-parrot would cover it; it does not, because rss-parrot declares no sensitive config at all. It generates its own secrets into HOSTY_FS instead. The apps that would exercise LoadCredential are pocket-id (ENCRYPTION_KEY_FILE) and vaultwarden (which still does not build).

All three were invisible with one app installed, which is why they survived every previous test. They are recorded because the shape is the lesson: hosty's whole purpose is running several apps, so a single-app test was never testing the real configuration.

portablectl is-attached cannot be trusted. It resolves its argument loosely and reports some other image's state when the one asked about is not attached — including for a path that does not exist:

# hosty-hello attached and running, rss-parrot-throwaway detached
$ portablectl is-attached rss-parrot-throwaway        → running-runtime
$ portablectl is-attached /var/lib/hosty/images/nope  → running-runtime
$ portablectl list | grep rss-parrot-throwaway        → detached

Switching from the name form to the image-path form does not fix it. hosty now parses portablectl list --no-legend, which is the only output that reports per-image state correctly. Every other confusing symptom traced back here: detach loops that could not terminate because a detached image still looked attached, and errors naming an unrelated app ("Unit file 'hosty-hello.service' is active, can't detach" while detaching rss-parrot-throwaway).

Re-running hosty start on a running app failed — which is the upgrade path, and the only way to apply changed config. setupSystemUnits detached with a bare portablectl detach, which portabled refuses while the unit is still active; the error was ignored, and the following attach then failed hard with "Unit file exists on the host already, refusing". cleanupSystem had the correct stop-then-detach ordering all along, so the two now share detachSystem.

hosty was breaking its own detach. The teardown path had accumulated os.Remove calls for unit symlinks portabled had supposedly "left behind after a successful detach". That description was wrong, and the removes were treating a symptom hosty caused itself.

portabled refuses to detach while the attached drop-in directory holds files it does not own, so hosty cleared the directory first — with os.RemoveAll. But that directory also holds portabled's own 10-profile.conf and 20-portable.conf, which are its record of the attachment. Deleting them made it lose the image. Measured on two otherwise identical stop-then-detach runs:

removed before detach detach leftovers
our 50-hosty.conf only exit 0, removes unit + both drop-ins + dir + /run/portables/<name> enablement symlink only
the whole .service.d dir fails: "No unit files associated with '' … Image not attached?" unit symlink and enablement symlink

So detach cleans up after itself perfectly, provided hosty removes only its own file. hostyDropinName is now a named constant precisely because "which file is ours" is the load-bearing distinction.

One genuine portabled quirk remains: attach --enable creates default.target.wants/<name>.service, and detach does not remove it, even on exit 0. Verified against bare portablectl with no hosty involved — a plain attach→detach pair leaves nothing, attach --enable→detach leaves that one symlink. It does not block a later attach (also verified: attaching over it succeeds), but it dangles into an image tree about to be deleted, so hosty removes it. Both halves of this were then confirmed by deletion: dropping the enablement-symlink removal makes the test fail with three orphans; dropping the unit-symlink removal changes nothing, because detach now does it.

-name could not actually rename an app. hosty pack bakes PORTABLE_PREFIXES=<packed name> into the image's os-release, while the unit is synthesized at deploy time from the install name. When they disagreed portablectl refused outright ("Acceptable prefix matches are: rss-parrot"). The unit is deliberately synthesized rather than packed; the prefix list is part of that same "named at deploy time" contract and is now rewritten alongside it.

Debugging lesson, again. The first two of these presented as an error message naming the wrong app. The instinct is to distrust the message; the message was accurate, and it was the state query underneath that lied.

Everything hosty needs to restart a system-mode app lives in tmpfs, so a reboot leaves the app installed but not attached and not running. The persistent half is fine: the image tree (/var/lib/hosty/images/<name>), data.hosty and hosty.db are all on disk. What is lost is the wiring:

# Artifact Written to
1 attached unit + portabled's own drop-ins /run/systemd/system.attached/ (portablectl attach --runtime)
2 hosty's 50-hosty.conf drop-in /run/systemd/system.attached/<name>.service.d/
3 enablement symlink /run/systemd/system.attached/default.target.wants/
4 the <name>-fs.service companion /run/systemd/system/

This was never a deliberate decision — --runtime was simply what the first working attach used, and the companion unit went to /run because /etc/systemd/system is a read-only Nix store symlink on NixOS.

Dropping --runtime is necessary but not sufficient, which is the trap. It fixes 1–3 (portabled then writes to /etc/systemd/system.attached/ instead), but not 4 — and since the app BindsTo= + After= its -fs unit, fixing 1–3 alone yields something worse than today: a unit that is persistently enabled and then fails at every boot because its companion has vanished.

Facts checked on legosi, which say the fix is available but needs care:

Open questions before attempting it, none of which should be assumed:

Whatever the answer, it needs a VM test that genuinely reboots the machine and asserts the app comes back serving — not merely that units exist. This touches setup()/detachSystem/cleanupSystem, which is exactly where the half-installed-app class of bug has always lived.

Note this is orthogonal to Phase 2: the TLS work behaves correctly either way, because the ask endpoint keys off attachment, so after a reboot it truthfully answers "not attached".

A state directory is now described by a database at its root, hosty.db: settings that apply to the whole environment, plus one row per installed app carrying its port and its sensitive config values. It replaces the per-app runtime.hosty.

The trigger was Phase 2 having nowhere to put baseDomain. Every existing setting was either per-app (runtime.hosty) or per-invocation (a flag), so a value belonging to the environment had no home. Two bad answers were tried first, and both are worth recording as things not to repeat:

While mapping what an "environment" consists of, the per-app file turned out to be misfiled. StateDirectory= hands the app directory to the app's own DynamicUser, so on legosi:

-rw------- 1 rss-parrot rss-parrot  runtime.hosty

That file held _hosty_config_shadow, the cleartext source for every sensitive value. 0600 looks right until you notice the owner is the app: it could read its own secret store, and being the owner, rewrite it — after which hosty would re-materialise the altered values as credentials on the next start. That defeats the point of the LoadCredential path, which exists precisely so an app receives only its own secrets, through a channel hosty controls.

Not exploited: both legosi apps had zero secret rows, which is why it went unnoticed. Mode was never the problem, so no permission change could have fixed it; the file had to leave app-owned territory.

The environment root is not exposed to apps — an app's namespace shows only its own subdirectory of the state dir (verified from outside via /proc/<pid>/root, since a self-contained image has no shell). So 0600 root at the root is a boundary where inside the app directory it was not.

_hosty_env      key/value: schema_version, and (Phase 2) base_domain
_hosty_apps     name PRIMARY KEY, port UNIQUE
_hosty_secrets  (app, key) PRIMARY KEY, value,
                app REFERENCES _hosty_apps(name) ON DELETE CASCADE

Foreign keys must be requested explicitly. SQLite defaults them off, and an unenforced constraint is worse than none: it reads as a guarantee while silently accepting orphans. This codebase had already been bitten by a DSN parameter that looked right and set nothing (_journal_mode=, see openDB), so the pragma is asserted by a test rather than trusted — and the test was checked to fail without it.

Two things fall out of the move:

Done by hand rather than in code, because the whole population was two apps on legosi (ports only) and eight local user-mode apps (three ports, one secret). _hosty_runtime_state only ever held port, verified across all ten installs. Migration code would have outlived its usefulness immediately.

Goal. An app installed as <name> becomes reachable at https://<name>.hosty-test.profpatsch.de with no per-app configuration. hosty-test is a scratch domain for building this out; the production label is undecided, so the base domain must be a config option from the start.

Target host. legosi.

DNS is already in place. *.hosty-test.profpatsch.de A → 88.198.193.255 (legosi), TTL 300, at INWX. Verified resolving, including the two behaviours worth remembering: a wildcard matches at any depth (a.b.hosty-test resolves) but never its own parent (hosty-test.profpatsch.de is NXDOMAIN — so a future admin UI at the bare name needs its own record). Raise the TTL once things settle.

Phase 1c is done, so this is now unblocked: apps run sandboxed on legosi and are reachable on their assigned ports over tailscale.

hosty already assigns every app a free port and records it in the environment database (appSetPort, readable via appPort), so the missing piece is a reverse proxy that learns about apps as they come and go.

Who terminates TLS looked like the hard question, and it dissolved. The Phase 1 note said Caddy would run "alongside the existing nginx… on different ports/hosts", which does not work: legosi's nginx already owns :443 for 14 virtual hosts (website, softwaregardening ×11, decentsoftware, modular-flyer, source-forge), and on-demand TLS needs :443 both to answer TLS-ALPN-01 and to serve the app. A later revision of this document concluded from that "use DNS-01

nginx is compiled --with-stream_ssl_preread_module (verified), and NixOS exposes services.nginx.streamConfig. So nginx binds :443 in a stream block, reads the SNI hostname without decrypting, and forwards the raw TCP connection:

Caddy terminates TLS for hosty apps only. It gets :443 semantics without owning the socket, and the 14 existing vhosts are untouched.

This is also what makes on-demand TLS safe. The objection to it was that an ask endpoint is mandatory, else anyone pointing a hostname at the IP can make you mint certs — and that the simplest ask endpoint drags Phase 3's root daemon into Phase 2. But nginx's SNI map is itself the allowlist: only hostnames matching the wildcard ever reach Caddy, so the question is answered before Caddy sees the connection. Caddy still needs ask configured (it refuses on-demand TLS without one), but it is defence in depth rather than the sole gate.

Worth recording, because the wildcard looks obviously simpler and is not.

A wildcard can only be issued via DNS-01 — that is Let's Encrypt policy, not a lego limitation. So it needs INWX_USERNAME/INWX_PASSWORD on legosi permanently (certs last 90 days; "do it once" is not an option), and INWX has no scoped tokens: that is the full registrar account, transfers included.

Against that, the wildcard buys surprisingly little, because the wildcard A record already does the heavy lifting. foo.hosty-test.profpatsch.de resolves to legosi today with no DNS action, so HTTP-01 and TLS-ALPN-01 already work for any app name. Per-app certs need no registrar credentials at all. The wildcard cert's only advantage is not having to issue a cert when an app appears — which on-demand TLS handles anyway.

per-app, on-demand (chosen) wildcard via DNS-01
registrar password on host none yes, permanently
blast radius if host owned one cert all domains
cert work on app install none none
new DNS records per app none none
:443 arbitration none (SNI passthrough) none

For the record, had the wildcard been chosen: NixOS supports it natively via dnsProvider = "inwx" + credentialFiles, delivered through systemd LoadCredential, so no secret would enter the Nix store; and the cert attribute name may not contain * (there is an assertion), so it would be security.acme.certs."hosty-test".domain = "*.hosty-test.profpatsch.de".

Visiting https://bla.hosty-test.profpatsch.de in a browser gives:

Secure Connection Failed
An error occurred during a connection to bla.hosty-test.profpatsch.de.
Peer reports it experienced an internal error.
Error code: SSL_ERROR_INTERNAL_ERROR_ALERT

This is the abuse protection working, not a fault. bla is not an attached app, the ask endpoint refuses, Caddy declines to obtain a certificate, and without a certificate there is no way to complete a handshake — so the failure must happen at the TLS layer, before HTTP exists to carry a readable message. Serving a nice error page for bla would require a certificate for bla, which is precisely what is being refused.

The three cases are cleanly distinguishable, and only the last one is opaque:

Request Result
attached app with a route snippet 200 from the app
attached app, no route snippet 502 hosty: no route configured for …
not an app at all TLS alert, no HTTP response

There is a better alert available, and we are not currently sending it. RFC 6066 §3 defines unrecognized_name(112) for exactly this situation ("the server understood the SNI extension but does not recognize the server name"), whereas what goes out today is internal_error(80). Go's TLS stack does send 112 — handshake_server_tls13.go picks the alert based on the error from getCertificate:

certificate, err := c.config.getCertificate(...)
if err != nil {
    if err == errNoCertificates {
        c.sendAlert(alertUnrecognizedName)
    } else {
        c.sendAlert(alertInternalError)   // ← what we hit
    }

so Caddy's on-demand refusal returns some other error and lands in the else. This has now been investigated with the Caddy source at hand and is settled — see "The unrecognized_name(112) question, settled" below. In short: the fix is real and is not blocked on go:linkname as this document previously guessed, but it is reachable only by patching Caddy, so it was not taken.

One correction worth making here, because this section originally argued the opposite. It claimed 112 would be "differently opaque rather than clearer", judging it by its symbol name (SSL_ERROR_UNRECOGNIZED_NAME_ALERT). But the symbol is not what a user sees; the NSS error string is:

alert sent Firefox renders
today internal_error(80) "Peer reports it experienced an internal error."
with the fix unrecognized_name(112) "SSL peer has no certificate for the requested DNS name."

So today's message is not merely unhelpful, it is misleading: it reports our server as faulty when it is deliberately and correctly declining. The 112 message is accurate and points at the actual cause. That makes the fix a genuine user-facing improvement, not just a tidier log line — which is the opposite of what was assumed before looking.

Two options were considered and rejected:

The conclusion is that the browser-side error is irreducible without patching Caddy — the alert can be made accurate (next section), but no amount of configuration will produce a readable page for a name we are refusing to certify. The gap that can be closed cheaply is therefore operator-side: nothing currently tells you why a name was refused, even though the ask endpoint already knows and logs it (journalctl -u hosty-serve-ask gives "app bla is not attached", "invalid app name", "domain has more than one label", etc.).

Investigated against the Caddy checkout at ~/kot/caddy (v2.11.4+), certmagic v0.25.4 and Go 1.26. Outcome: achievable, worth more than expected, and not taken — it needs a patched Caddy, which is not worth a fork for an error message. Recorded so it is not re-investigated.

The expected blocker was not the blocker. This document previously expected the investigation to die on the grounds that the fix required returning errNoCertificates, which is unexported and reachable only via go:linkname. That framing was wrong. Look at what crypto/tls actually does (common.go:1317):

if c.GetCertificate != nil && (len(c.Certificates) == 0 || len(clientHello.ServerName) > 0) {
    cert, err := c.GetCertificate(clientHello)
    if cert != nil || err != nil { return cert, err }   // ← falls through on (nil, nil)
}
if len(c.Certificates) == 0 { return nil, errNoCertificates }

Returning (nil, nil) makes the standard library synthesize the sentinel itself, and the handshake then sends alert 112. No linkname, no unexported access. The one precondition is len(c.Certificates) == 0, which holds: Caddy never populates static Certificates on a server connection policy (the only assignments in the tree are caddyconfig/httploader.go, the reverse-proxy transport, and caddytest — all client-side or test).

The denial chain, so nobody has to trace it again. Every hop wraps with %w, so the error arrives intact and simply is not the sentinel:

hosty serve-ask → 404
  → PermissionByHTTP.CertificateAllowed  wraps ErrPermissionDenied   ondemand.go:165
  → OnDemandConfig.DecisionFunc          returns it unchanged        automation.go:338
  → checkIfCertShouldBeObtained          "certificate is not allowed for server name %s: %w"
  → getCertDuringHandshake               certmagic handshake.go:361
  → GetCertificate callback              returns the error           connpolicy.go:317
  → crypto/tls: err != errNoCertificates → alertInternalError(80)

There is no configuration or plugin route to a better alert. This was the first thing checked, since a patch is the expensive answer. Four candidate hooks, all dead ends:

Hook Why it cannot work
tls.permission.* module Returns error. No return value expresses "no certificate" — non-nil means error, nil means allowed. Writing our own permission plugin instead of tls.permission.http would not help.
Drop connection policy Goes through GetConfigForClient, which sends alertInternalError unconditionally (handshake_server.go:170) — the same alert. Also matches on ClientHello only, so it cannot consult the ask endpoint.
cert managers (tls.get_certificate.*) Run before the permission check (handshake.go:348); returning empty+nil falls through to the same path. Only useful for serving some certificate, i.e. the self-signed fallback already rejected above.
handshake_context module Its error becomes a plain wrapped error (connpolicy.go:313) → alert 80.

The reason is structural: every surface Caddy exposes as config or plugin is typed error, while the (nil, nil) signal Go keys on exists only in the raw GetCertificate callback signature — which lives inside Caddy's own closure at connpolicy.go:282-318 and is not pluggable. That closure is the only place the fix can go.

The patch, for the record (~5 lines, connpolicy.go):

cert, err := cfg.GetCertificateWithContext(ctx, hello)
if err != nil && errors.Is(err, ErrPermissionDenied) {
    // must log here: returning (nil, nil) discards the reason
    return nil, nil   // → stdlib sends unrecognized_name(112)
}
return cert, err

Narrow by construction: errors.Is(…, ErrPermissionDenied) fires only on real denials, so genuine failures (ask endpoint down, ACME broken, storage errors) keep returning 80 — which is correct, those are internal errors.

Why not taken. It means a forked Caddy on legosi, rebased at every nixpkgs bump (nixpkgs ships 2.11.2, the checkout is 2.11.4+), to change one error message on a path that is only hit by requests for apps that do not exist. The vendorHash would not move, so the mechanics are easy; the maintenance is the cost. It remains a reasonable upstream PR for anyone who wants it — the current behaviour is arguably a bug, since Caddy reports an internal error for a decision it made deliberately — but that is upstream's call to carry, not ours.

The gap actually worth closing here is still the operator-side one: nothing tells you why a name was refused. See "make a refused name diagnosable" under "Next session".

hosty writes a Caddyfile snippet (imported from the Nix-managed Caddyfile) and restarts Caddy. The admin API would avoid that, but its changes are in-memory, so something must re-register after a restart — an open question a file makes disappear rather than answers. App start/stop is a rare, human-initiated event, so the cost is not worth the extra moving part.

Corrected in B2. This section originally said systemctl reload caddy. Reload cannot work here: caddy reload POSTs the new config to the admin API, which this design disables (admin off), so the reload fails with "connection refused" while the old config keeps serving. The unit reports a failed reload and routes silently do not take effect. It must be a restart, which costs nothing extra: certificates live in Caddy's data directory and survive it, and the fourteen nginx vhosts are a different process entirely.

Caddy's dynamic upstreams were considered as a way to avoid restarts entirely and rejected: the built-in modules are DNS-based (srv, a, multi), and only SRV carries a port, so hosty would have to run a DNS server. map is static config and needs a reload anyway.

Route lifetime stays tied to service lifetime either way, the same principle as the unit coupling in Phase 1: registered in setup(), torn down in cmdStop.

Names were completely unvalidated, yet they already flowed into filesystem paths, systemd unit names and PORTABLE_PREFIXES, and become DNS labels here.

Rule: RFC 1123 label, ^[a-z0-9]([a-z0-9-]*[a-z0-9])?$, ≤63 chars. Lowercase and digits with inner dashes, which is exactly what a DNS label may be, so the subdomain, the state directory and the unit name can never disagree. Uppercase is rejected rather than accepted-and-normalised: DNS is case-insensitive while Linux paths are not, so MyApp and myapp would be one subdomain but two images — the same class of aliasing that made is-attached misreport.

Reserved: any name ending in -fs (it would collide with hosty's own <name>-fs.service companion unit), plus www, admin, api and hosty, which would shadow infrastructure once subdomains exist.

Reserving hosty contradicts the Phase 3 aside about hosting hosty itself at hosty.hosty.profpatsch.de. That is deliberate: the reservation only blocks a user install claiming the name. Phase 3 would register it from inside hosty, which does not go through validateAppName.

validateAppName is called from cmdPack and from setup() — the latter because -name overrides the packed name, and because an image packed by an older hosty may carry a name predating the check — and deliberately not from stop/info/config, so an app somehow installed under a bad name stays removable. All five packed apps and both deployed apps already satisfied the rule. Both call sites validate before creating any state, which is the half-installed-app failure mode the relative -state-dir bug had; the VM test asserts that neither the state dir, the image dir nor a systemd unit appears.

Testing lesson. The first version of that VM assertion passed vacuously for the name -lead: systemctl list-unit-files -lead.service failed with "invalid option -- 'e'" rather than because no such unit existed. A negative assertion is only worth as much as the reason the command failed, so the check now passes -- to both systemctl and grep -F.

nginx binds :443 in a stream block, reads SNI without decrypting, and forwards the raw connection to its own HTTP block on 127.0.0.1:8444. Nothing routes to Caddy yet: the point was to prove stream + ssl_preread + PROXY protocol + real-IP recovery works against the 14 live vhosts while changing no behaviour, so that any breakage has exactly one possible cause. Caddy arrives afterwards as an additive change (B2). Lives in machines/profpatsch/legosi.nix.

services.nginx.defaultListen = [
  { addr = "127.0.0.1"; port = 8444; ssl = true; proxyProtocol = true; }
  { addr = "0.0.0.0";   port = 80;   ssl = false; }   # unchanged — ACME
];
services.nginx.commonHttpConfig = ''
  set_real_ip_from 127.0.0.1;
  real_ip_header proxy_protocol;
  port_in_redirect off;          # see below — load-bearing
'';
services.nginx.eventsConfig  = "worker_connections 4096;";
services.nginx.prependConfig = "worker_rlimit_nofile 8192;";
services.nginx.streamConfig = ''
  map $ssl_preread_server_name $hosty_backend {
    default 127.0.0.1:8444;        # B2 adds the hosty-test branch → Caddy
  }
  server {
    listen 0.0.0.0:443;
    ssl_preread on;
    proxy_pass $hosty_backend;
    proxy_protocol on;
    proxy_timeout 3600s;           # see below
  }
'';

Facts this rests on, all verified on legosi:

All three would have made B1 something other than the no-op it claims to be.

port_in_redirect off; — this one broke the live site. nginx builds absolute redirects from the port it is locally bound to:

port = ngx_inet_get_port(c->local_sockaddr);              /* now 8444 */
if (clcf->port_in_redirect) { port = (port == 443) ? 0 : port; }

(ngx_http_header_filter_module.c; port_in_redirect defaults to on.) With the TLS listener moved, 8444 no longer matches the special case, so every implicit directory redirect would have started emitting https://host:8444/… — a port the firewall drops. set_real_ip_from does not rescue this: the realip module rewrites c->sockaddr, the source address, and never c->local_sockaddr. PROXY protocol does not either — it parses dst_port but never applies it. The canary is a URL that redirects implicitly:

curl -sI https://profpatsch.de/mlp/music   # must not contain :8444

proxy_timeout 3600s;. The stream module's proxy_timeout defaults to 10 minutes and becomes the binding constraint on every connection — including ones the HTTP block deliberately holds open longer. The inventory vhost sets proxy_read_timeout 3600s for its capture WebSocket and SSE stats stream, which the stream layer would otherwise cut at 10 minutes regardless.

worker_connections / worker_rlimit_nofile. Every external connection now costs two: client→stream and stream→HTTP. events {} was empty, so the default of 512 applied, with a single worker — external capacity would have quietly halved to ~256. worker_connections is in turn capped by the fd limit, whose soft value was 1024, hence raising both.

The generated nginx.conf was diffed against the running one before deploying. This works precisely because nix eval of the machine's nginx ExecStart locally reproduced the exact store path that was running (5kvrgrkyl…), making the diff authoritative rather than approximate. The diff was mechanically checked to contain only: 14 listen-line rewrites, the stream block, and the three additions above — with :80 lines, all 56 acme-challenge locations, every server_name, and every ssl_certificate byte-identical. The new config was then copied to legosi and validated in place with nginx -t against the running system, before any switch.

Verified after deploying (all against a baseline captured beforehand, so that pre-existing oddities — happy 404s, asciinema 303s — were not misread as regressions):

Rollback is generation 210 (nixos-rebuild --rollback on legosi), or reverting the commit and redeploying. Deployed as generation 211.

This was the only step in Phase 2 with production blast radius: a mistake takes profpatsch.de, sources, decentsoftware and all of softwaregardening offline simultaneously. It is a no-op by construction — but "by construction" was exactly the assumption that the port_in_redirect bug violated, and only diffing and probing caught it.

Caddy now terminates TLS for hosty apps on 127.0.0.1:8443, reached through one new branch of the SNI map. hosty-hello.hosty-test.profpatsch.de serves over a publicly trusted certificate obtained on demand, and the whole chain is live: DNS → nginx :443 → ssl_preread → PROXY protocol → Caddy → ask endpoint → TLS-ALPN-01 → reverse_proxy → the app.

The ask endpoint keys off attachment, not app state. The plan in this document said hosty serve-ask would be "unprivileged, DynamicUser" and would answer "200 if the app exists in state". Both halves were wrong:

The fix came from asking what hosty already knows. It does not track running apps itself; it asks systemd (serviceStatus). Portable services register at /run/systemd/system.attached/<name>.service, which is 0755 and world readable, created by portablectl attach and removed by detach — both already driven by setup() and cmdStop. So the endpoint stays unprivileged and gets the right lifetime for free, with no new state to keep in sync.

Attachment alone is the criterion; liveness is deliberately not checked. It is what start/stop actually toggle, and a crash-looping app should not lose its certificate over a blip that certificates outlive by months.

systemctl is-active on its own would have been wrong in the other direction: nginx is loaded/active too, so it would have authorised a certificate for nginx.hosty-test.profpatsch.de. The attached-unit path is the discriminator.

All four were found by adapting the config or by deploying, not by reading docs.

https_port 8443 is required; bind 127.0.0.1 is not enough. A site block with only bind still listens on port 443, which is the very socket nginx holds. Caught by inspecting the adapted JSON before deploying.

A per-app snippet without bind 127.0.0.1 silently creates a second server. The imported site block produced its own listener on :8443 across all interfaces, with neither the proxy_protocol wrapper nor trusted_proxies — exposing Caddy publicly and breaking PROXY parsing. Nothing warns about this; the only symptom is an extra entry in the adapted JSON's server list.

os.Stat on the attached unit fails for an unprivileged caller. portabled attaches with --copy=symlink, so the marker is a symlink into /var/lib/hosty/images/… — and Stat follows it straight into the 0700 directory, returning EACCES, which made every app look detached. os.Lstat is what the check wants, since only the link's own existence is in question. The unit tests missed this because they stub the attachment check; only the shape of the real path shows it. There is now a regression test that fails against Stat.

admin off and NixOS's services.caddy.enableReload are incompatible. caddy reload POSTs the new config to the admin API, so with the API disabled the reload fails with "connection refused" while the old config keeps serving — the worst outcome, because routes silently do not take effect and the unit merely reports a failed reload. enableReload = false (restart instead) is correct here: app install is rare and human-initiated, certificates live in Caddy's data directory and so survive a restart, and the 14 nginx vhosts are a different process entirely.

HTTP-01 is impossible (nginx owns :80 for its own 16 renewals), so it is disabled explicitly and TLS-ALPN-01 is used with alternate_port 8443. That works because the SNI passthrough already forwards the handshake there — the concrete reason B1 had to land first.

Issuance was proven against Let's Encrypt staging first, then switched to production. The staging certificate came back as CN=(STAGING) Baloney Bulgur YE2, confirming the path end to end at zero rate-limit risk; production then issued CN=YE2 for hosty-hello.hosty-test.profpatsch.de, valid to Nov 5.

State right now: https://hosty-hello.hosty-test.profpatsch.de serves over a real Let's Encrypt certificate, and an attached app without a route snippet (rss-parrot-throwaway) returns the catch-all 502 — both confirmed from outside. At the end of the B2 session the repo, the deployed closure and the running services all matched, verified by comparing nix eval of the toplevel against readlink -f /nix/var/nix/profiles/system on legosi. Re-check that before changing anything:

nix eval --raw .#nixosConfigurations.legosi.config.system.build.toplevel
ssh legosi readlink -f /nix/var/nix/profiles/system

The one thing still done by hand is the route snippet, so step A is the next piece of work. The two items after it are small and independent.

setup() writes /var/lib/hosty-routes/<name>.caddy; cmdStop removes it; both then restart Caddy. Everything needed already exists: the port is in runtime state (stateGet(runtimeDB, "port")) and the name is already validated as a DNS label, so no new state is introduced.

The snippet to generate, exactly as verified by hand:

<name>.<baseDomain> {
	bind 127.0.0.1
	tls {
		on_demand
	}
	reverse_proxy 127.0.0.1:<port>
}

Four things this must get right, each of which cost time to discover:

Worth doing in the same change, because it is the same moment in the lifecycle: apps like rss-parrot need their public URL as config before they boot (HOST drives every ActivityPub URL). Route registration and URL injection both belong inside setup(), before the service is started — see the first open question below.

Verification: install a second app, confirm it serves over TLS without any manual step; hosty stop it and confirm the snippet is gone, Caddy restarted, and the name now refused by the ask endpoint. Then re-run the B1 checks (14 vhosts, /mlp/music redirect canary, ACME token on :80, real client IPs), since every Caddy restart is a chance to disturb the shared :443.

Today nothing on the operator side explains why bla.hosty-test… fails; the browser shows only a TLS error and hosty info does not mention publishing at all. The ask endpoint already computes and logs the exact reason, so this is mostly plumbing:

Deliberately not the fix: issuing certificates for uninstalled names, or a self-signed fallback. Both are rejected above, with reasons.

Goal. A web interface where dropping a .hosty file installs and starts it, with per-app start/stop/config/export.

Everything the UI needs already exists as CLI logic — infoAll (list), setup (install/upgrade), cmdConfig (config schema and values), cmdExport (backup). The UI should be a presentation layer over that same code, not a reimplementation.

The significant change is that hosty becomes a long-running root daemon rather than a CLI invoked with sudo. That is a real shift in threat model: it exposes privileged operations over HTTP, so it needs authentication and must itself be sandboxed. If Phase 2 takes the Caddy route it also needs a daemon (for the TLS ask endpoint) and the two should then be designed as one; if Phase 2 goes with DNS-01 it needs no daemon at all, and this stays the only reason to build one — which is an argument in DNS-01's favour.

Pleasingly, hosty could then be packaged as a .hosty app and hosted at hosty.hosty.profpatsch.de by the same machinery it provides.