source-forge(1)

NAME

source-forge - on-demand source browser backed by SQLite

SYNOPSIS

source-forge project add --db path --repos dir --project name [--branch name] [--base-url url]
source-forge project set-branch --db path [--repos dir] --project name --branch name
source-forge project list --db path
source-forge ingest --db path --project name [--branch name] [--git-dir path] <archive.tar>
source-forge serve --db path [--addr host:port] [--base-url url]
source-forge git-post-receive-hook --db path [--git-dir path] [--base-url url]

DESCRIPTION

source-forge publishes the source tree of a project as a browseable, syntax-highlighted static site, rendered on demand from SQLite. It is split into subcommands: one that projects are declared in (and their bare repos created), one that ingests a snapshot of a tree into the database, one that serves that tree over HTTP, and a git-post-receive-hook that re-ingests on every push.

The design goal is scraper resilience. Only rendered static content is ever exposed – there is no git history to walk and no per-request computation after the first render of a file – so crawlers hitting every link cost almost nothing. Each project is stored as a series of generations: ingesting a new tree writes every row under a fresh generation and then atomically flips the project to it, so a reader never observes a half-written tree, and superseded generations are pruned. A project that has been declared but never pushed to sits at generation 0 and is shown on the site as pending.

Project

source-forge project declares, redirects, or lists published projects. A project must be declared with project add before anything can be pushed to it; see Git post-receive hook. The sub-subcommands are as follows:

add

Creates a bare git repository at <repos>/<project>.git, points its HEAD at --branch, and inserts the project's row into the database at generation 0 (pending). Fails if the repository or the database row already exists, so the two can never drift out of sync.

--db path

Path to the SQLite database. Created if it does not exist. Required.

--repos dir

State directory the bare repository is created under. Required.

--project name

Project name. Must start with a letter or digit and contain only letters, digits, '.', '_', or '-', since the post-receive hook derives the project name back out of the repository's directory name. Required.

--branch name

Branch that will be published on push. Defaults to canon.

--base-url url

Public base URL of the site, e.g. https://sources.example.org. Optional, and used only to print the URL the project will appear at once it is pushed to; omitted from the output when not given.

set-branch

Changes which branch a declared project publishes. Updates the database row; if --repos is also given, re-points the repository's HEAD too.

--db path

Path to the SQLite database. Required.

--repos dir

State directory the bare repository lives under. Optional; when omitted only the database row is updated (the next push still re-points HEAD, see Git post-receive hook).

--project name

Project name. Required.

--branch name

New branch to publish. Required.

list

Prints every declared project, its published branch, whether it has been ingested yet, and whether a description is set. Takes only --db path.

Ingest

source-forge ingest reads a git-archive(1) tar stream on standard input and stores the tree in SQLite under a new generation, then makes it the live tree. The arguments are as follows:

--db path

Path to the SQLite database. It is created if it does not exist. Required.

--project name

Project name the tree is stored under. A single database can hold many projects, keyed by this name. Required.

--branch name

Branch that was archived, recorded on the project row and shown in the listing.

--git-dir path

Path to the source git repository. When given, two extra artifacts of --branch are built and stored alongside the tree: a clonable git bundle (git-bundle(1)), so visitors can clone the full history offline, and a gzipped tarball (git-archive(1)) usable directly as a Nix flake (see URLS). Requires --branch, which selects what to bundle and archive.

Serve

source-forge serve serves directory listings and highlighted files over HTTP, rendering each file lazily and caching the resulting HTML back into the database. The arguments are as follows:

--db path

Path to the SQLite database to serve. Required.

--addr host:port

Address to listen on. Defaults to 127.0.0.1:8790.

--base-url url

Public base URL used for absolute links, most importantly the git bundle clone hint and the Nix flake tarball hint. When omitted, a URL is reconstructed per request from the scheme and Host header, so this is only needed when that guess would be wrong (for example behind a reverse proxy that rewrites the host).

Git post-receive hook

source-forge git-post-receive-hook is meant to be installed once, generically, via git-config(1)'s core.hooksPath, rather than symlinked into each repository's hooks/post-receive individually – see EXAMPLES. That way it applies to every bare repository under the state directory, present or future, with no per-project setup beyond project add.

Because it is installed generically, neither the project name nor the branch is passed on the command line: the project name is taken from the repository's own directory name (<project>.git), and the published branch is read from that project's row in the database, which must already exist (see project add above). A push to a repository with no such row is rejected with an error pointing at project add; this is what prevents an arbitrary push from silently publishing an undeclared repository, or an unintended branch.

The hook reads the pushed ref updates from standard input (one "old new ref" line per updated ref, as git provides) and, for every update to the declared branch, re-points the repository's HEAD at it and re-ingests it into the database using the same code path as ingest. Pushes to any other ref (including other branches of a declared project) are ignored, so the served site tracks the declared branch automatically on every push.

After a successful ingest the hook writes the project's published URL to standard error, which git relays back to whoever pushed (prefixed, as git prefixes all hook output, with 'remote'), so a push ends with a link to the page it just updated:

remote: source-forge: published at https://sources.example.org/Profpatsch/

This requires --base-url; without it the line is simply omitted, rather than a guessed URL printed. The arguments are as follows:

--db path

Path to the SQLite database. Required.

--git-dir path

Path to the bare git repository. Defaults to GIT_DIR, which git sets when it invokes a hook, so it usually need not be given explicitly.

--base-url url

Public base URL of the site, as passed to serve. Optional, and used only for the published-URL line described above.

URLS

The serve subcommand exposes the following paths:

/

The project index, listing every published project.

/<project>/

A directory listing. A trailing slash means directory. At the project root, the description declared in the root .source-forge (if any) is shown between the page header and the clone/tarball instructions. Single-child directory chains are collapsed GitHub-style, extra entries declared by a .source-forge file are spliced in (see FILES), and the directory's prose is rendered below the listing: its README.md first, if it has one, then every manpage it holds, each under a "name(section)" heading linking to that page's own source file. A README and a manpage are shown together rather than one standing in for the other, since they answer different questions – what this directory is, and how the program in it is used. Manpages are ordered with the directory's own page first (one whose name matches the directory, or the project at the root), then by section number, then by path; at most eight are rendered, so a directory holding a whole tree of pages cannot turn one request into an unbounded number of mandoc(1) runs. Any beyond that are still listed, and still readable as ordinary files.

/<project>/<path>

A single file: raw bytes for binaries (streamed straight from the database), or cached syntax-highlighted HTML for text.

/<project>/<dir>/?full=

The recursive "full contents" view, rendering every file under a directory on one page. Offered only for subtrees within a fixed size limit, so a crawler appending ?full= to every directory cannot trigger huge renders.

/<project>.bundle

The clonable git bundle for a project, if one was stored at ingest time.

/<project>.tar.gz

The project tree as a gzipped tarball, if one was stored at ingest time. The archive wraps the whole tree in a single top-level <project>/ directory, which makes it directly usable as a Nix flake, for example nix run https://host/<project>.tar.gz#<package>. It carries an ETag of the live generation and answers a matching If-None-Match with 304 Not Modified, so Nix, which re-validates flake inputs on every evaluation, transfers the body only when the tree actually changed.

IMPLEMENTATION NOTES

Syntax highlighting is done with the chroma library, using CSS classes rather than inline styles so a single stylesheet can support both light and dark themes via prefers-color-scheme. Note that chroma emits a rule only for those tokens a theme actually overrides, so the two themes do not yield the same set of rules; both theme blocks are therefore wrapped in mutually exclusive media queries. Emitting one of them unscoped, in the expectation that the other shadows it, leaves every token the other theme is silent about at the wrong theme's colour. Markdown is rendered with goldmark (GitHub-flavored, with highlighted fenced code blocks). The manpages rendered below a directory listing are formatted by shelling out to mandoc(1), so it must be available on PATH for that feature; its Markdown output is then fed through the same goldmark pipeline, so a rendered manpage looks like a rendered README. The heading above each one is built from the filename rather than taken from mandoc's own title line, which is stripped along with the date footer: the filename is always present and always right, while that line carries a volume field that means nothing outside a manual. Headings in either carry a slugified id and link to themselves, so a section can be linked to directly, as in /<project>/#implementation-notes. Ids are unique only within one rendered document, so a page showing a README and a manpage together may repeat one – both are likely to have a "NAME" heading. A browser jumps to the first, which is the price of keeping anchors short enough to be worth linking to.

Bundle downloads, tarball downloads and raw binary files are streamed incrementally from the database using SQLite's blob API, so large payloads are never buffered wholly in memory.

FILES

.source-forge

An optional, checked-in metadata file describing how the directory containing it is displayed. It is read during ingest – it arrives in the tar stream like any other file – and its effects are baked into the stored listing, so nothing is parsed at request time. The file itself remains browseable like any other file in the tree.

The format is one key = value per line; blank lines and lines beginning with '#' are ignored, and only the first '=' separates the key from the value, so a value may contain further ones. A key that takes a list is simply repeated - there is no comma-separated form, so no value need worry about containing a comma:

./.source-forge

description = Profpatsch's personal monorepo.

shortcut = users/Profpatsch shortcut = users/Profpatsch/git-blimey

The recognised keys are:

shortcut path

Add an extra entry to this directory's listing, pointing at the descendant path (given relative to the directory holding the file). The entry is placed directly after the listing entry it lives under, so a shortcut to users/Profpatsch appears just below users/, not at the end of the page. May be repeated; the entries keep their declared order.

Shortcuts are purely additive: they can only add links, never remove or replace one, so no declaration can make part of the tree unreachable. This is what distinguishes them from the automatic collapsing of single-child directory chains, which is lossless by construction: a directory with two or more children is always listed as itself, and a shortcut is how an interesting subtree deeper down is surfaced anyway. The target must exist and be a strict descendant of the declaring directory; '.', '..', and absolute paths are rejected.

description html

A short phrase describing the directory the file sits in. It is shown on that directory's entry in every listing the directory appears in - including one reached by a shortcut from further up - following the size and file count, in the same muted style, and again above the directory's own listing when it is browsed. Allowed once per file.

The value is inline HTML, not markdown, because of where it lands: on the entry's own line, inside the listing's '<span>', where only phrasing content is valid. A markdown renderer's natural output is a '<p>' block, which is not. So it may carry '<code>', '<em>', a link and so on, but not headings, lists or paragraphs. It is stored verbatim and emitted unescaped, with no sanitisation - on the understanding that whoever can push to the published branch already controls every byte the site serves, a far more powerful capability than injecting HTML into one page of it.

Keep it to a phrase. It shares a line with the entry's name and stats, so a sentence that runs long simply wraps and pushes the row apart:

  description = An interactive <code>git blame</code> TUI

Because it describes the directory rather than any particular listing, it lives with the thing it describes: a project's blurb belongs in that project's own .source-forge, where it stays correct however the directory is reached, and a parent listing becomes an annotated index of what is beneath it.

In the repository root, which has no parent listing to appear in, it is the whole project's blurb instead, wrapped in a paragraph and shown on the project's page and on the site index.

This is the only way to set that blurb, and it is applied on every push, so deleting the line and pushing removes it. A project set-description subcommand once set it by hand on the server; it has been removed, because a description that lives only in the database is invisible from the repository, impossible to review alongside the change that motivates it, and raced with this file on every push.

Note that adding this file to a directory that holds exactly one other entry stops that directory from collapsing into its child, since collapsing applies only while there is a single entry to collapse into. That is usually what is wanted - a directory worth describing is a directory worth showing - but it does mean the file changes the listing merely by existing.

Nothing in this file is ever fatal. An unparseable line, an unknown key, or an unresolvable shortcut target produces a warning on standard error and that line alone is ignored; the ingest proceeds. This is deliberate: the post-receive hook runs after git has already moved the ref, so failing an ingest would leave the repository updated but the published site stuck on the previous generation

• a considerably worse outcome than one ignored directive. Since git relays hook output back over the wire, whoever pushed still sees the warning:

remote: source-forge: users/.source-forge:3: ignoring unknown key "collapes"

EXIT STATUS

The source-forge utility exits0 on success, and>0 if an error occurs.

EXAMPLES

Declare a new project, creating its bare repository:

$ source-forge project add --db forge.db --repos /var/lib/source-forge \
    --project Profpatsch --branch canon

Install the post-receive hook once, generically, so it applies to every project's repository under the state directory:

$ git config --global core.hooksPath /var/lib/source-forge/hooks
$ mkdir -p /var/lib/source-forge/hooks
$ cat >/var/lib/source-forge/hooks/post-receive <<'EOF'
#!/bin/sh
exec source-forge git-post-receive-hook --db forge.db \
    --base-url https://sources.example.org
EOF
$ chmod +x /var/lib/source-forge/hooks/post-receive

(In the NixOS module this core.hooksPath config and the hook script are both managed declaratively; see nixos-module.nix.)

Pushing to the declared repository now ingests it automatically:

$ git push ssh://forge@host/var/lib/source-forge/Profpatsch.git canon

Give the project a short description, shown on its root page, by committing it to the tree and pushing again:

$ echo "description = Profpatsch's personal monorepo." >>.source-forge
$ git add .source-forge && git commit -m 'describe the project'
$ git push origin canon

Serve the database locally:

$ source-forge serve --db forge.db --addr 127.0.0.1:8790 \
    --base-url http://localhost:8790

Ingest a one-off snapshot by hand instead (bypassing the hook and the declared-branch check; useful for testing):

$ git archive canon | source-forge ingest --db forge.db \
    --project Profpatsch --branch canon --git-dir .git

SEE ALSO

git-archive(1), git-bundle(1), mandoc(1)

AUTHORS

Profpatsch