Profpatsch/users/Profpatsch/epubify

View full contents ↓

epubify(1)

epubify - turn a scanned book chapter into an EPUB that keeps the printed page numbers

epubify convert [flags] file.pdf | url
epubify build [flags] doc
epubify report doc
epubify list

epubify reads a PDF of page images — the scanned chapter a seminar hands out — and writes an EPUB 3 to read on an e-reader. Where a printed page ends, the text has a marker such as '|12|', at the exact place, mid-sentence or mid-word, so that what is read can be cited by page. The EPUB also lists the pages in a page-list, which KOReader offers as "reference pages", and keeps footnotes as footnotes, shown in a pop-up.

Each page is read three ways and the readings are checked against each other; see CROSS-CHECK. What could not be confirmed is marked in the text, and listed in a "Pr"ufbericht" at the end of the book.

Everything computed -- rendered scans, every model answer, every OCR reading -- is kept in a database, see DATABASE, so running again costs nothing for what is already there.

convert file.pdf | url

Convert, reusing whatever the database already has for this PDF.
A
*url*
of a campusweb(1) file page,
*https://host/file/*‌*id*,
is fetched from its
*/content*
address, with
`EPUBIFY_BEARER_TOKEN`
as a bearer token for passkey-gate(1) if set; any other URL is fetched as is.

build doc

Write the EPUB again from what is stored, without any API call. A step missing from the database is an error. For trying changes to the layout of the book.

report doc

Print the cross-check report of the last run.

list

The documents in the database, with their runs.

doc is a prefix of a document's SHA-256 or part of the name it was added under.

-o file

The EPUB to write. Default: the PDF's name with .epub, in the current directory.

-db path

Default $XDG_STATE_HOME/epubify/epubify.db, else ~/.local/state/epubify/epubify.db.

-scans n[-m]

Only these pages of the PDF.

-title s, -author s

Override the title and author; see SORTING.

-name file

The file's name as it was handed out, if the PDF was renamed since; it gives the title and the number in the series.

-course s, -semester s

The course the reading is for and its semester, such as 'WS 2026/27'. Default: asked of campusweb, for a campusweb URL.

-tag s

A tag; repeatable, or several separated by commas.

-series s, -series-index n

Default: the course, and the number the file name starts with.

-no-auto-tags

Only the tags given with
**-tag**.

-calibre library

Also add the EPUB to this calibre library; see
*SORTING*.

-lang de|en|fr

The book's language; default as Jev classifies most pages.

-first-page n

The printed number of the first book page, when the numbers cannot be read.

-min-agreement f

Default 0.9.
A page whose share of confirmed words is lower is flagged.

-gemini-model m

Default
'`gemini-3.8-flash`'.

-jev-model m

Default
'`jev-preview`',
which the image questions need.
The alias moves to newer models; the database records which model answered.

-no-jev

Do not ask Jev.
Every landscape scan is taken to be two pages, every portrait one, and every
disagreement stays suspect.

-no-retry

Do not transcribe a flagged page a second time.

-offline

Use only what is in the database.

-allow-flagged

Exit 0 even if pages are still flagged.

-j n

Scans worked on at once; default 4.

-gemini-key-file path, -typesafe-key-file path

API keys.
Default:
`GEMINI_API_KEY`
and
`TYPESAFE_API_KEY`,
else
pass(1)
entries
*internet/ai.google.dev/gemini/api-keys/interactive*
and
*internet/console.typesafe.ai/api-keys/epubify*.

  1. Each PDF page is rendered at 300 dpi with pdftoppm(1), which applies the page's rotation and corrects scans stored at different horizontal and vertical resolutions. Whether the PDF has a text layer is decided from pdffonts(1) and pdftotext(1).

  2. Jev classifies each scan: one page or two, which way up, what kind of page (running text, title page, imprint, table of contents, bibliography, blurb, picture, blank), stamps, handwriting, language; and boxes the text blocks of both halves. The fold of a two-page scan is found in code, as the column with the least ink; it is accepted if it lies between Jev's two text blocks, and otherwise moved between them. Each page is then straightened; see CAVEATS.

  3. Running text, tables of contents and bibliographies are read. The title page goes into the book as a picture; imprint, blurb, blank pages and stamps are left out, and listed in the report. The title page and imprint are also read for the book's metadata, which is used for the citation on the book's first page.

  4. Each text page is read by tesseract, by Gemini, and from the text layer if there is one, and the readings are merged; see CROSS-CHECK.

  5. Page numbers: the number Gemini read on each page, or failing that the one tesseract read, gives an offset between a page's position and its number; the offset most pages agree on is taken. A page whose printed number is missing or disagrees gets the inferred number in brackets, '[11]', which is how a page without a printed number is cited.

  6. The pages are joined into one text: a paragraph, or a footnote, that runs on over a page break is one paragraph, with the marker where the page turned; a word hyphenated over the break is one word, with the marker inside it. A new file begins at each heading.

What a book is filed under is written into the EPUB, where an e-reader finds it without calibre: tags as dc:subject, which KOReader shows as "keywords", and the series both as calibre:series and as EPUB 3's belongs-to-collection, which KOReader shows as "series". KOReader's collections can be filtered by keyword and series.

For a course reading the defaults come from where it came from:

title

The file name, which says what was assigned --
'`2 Genette - Paratexte Einleitung S. 9-21.pdf`'
gives
'`Paratexte Einleitung S. 9-21`'
-- with the number in front and the author's surname removed.
The title page only names the book.
A PDF counts as a course reading if its name starts with a number or the
course is known; any other keeps the title from its title page.

series

The course.

series index

The number in front of the file name, by which lecturers order a course's
readings.

tags

'`Uni`',
the semester, the course, and every
**-tag**.

description

The course and semester, and the citation of the printed edition.

The course and semester are asked of campusweb(1) when the PDF is given as a campusweb URL, and remembered in the database, so that build keeps them; otherwise they come from -course and -semester.

With -calibre, the EPUB is added to a calibre library with the same metadata, and with the identifier 'epubify:*sha256*' of the PDF. A book converted again replaces the EPUB of the book with that identifier, and its metadata in calibre, instead of adding a second book; what was edited in calibre by hand is overwritten. calibre cannot keep a comma in a tag or series set from the command line, and gets a semicolon in its place.

Gemini reads a page as a person does -- paragraphs, footnotes, italics -- and is the one reading likely to improve on it unasked: modernise "dass" to "dass", skip a line that seemed redundant, complete a sentence. Its text is what the book is made of, and every word of it is checked.

Gemini's words are aligned with tesseract's, by a minimum-cost alignment of the two sequences. Case, punctuation, line-end hyphenation, long s and ligatures are folded away first, and so are the confusions tesseract typically makes in old-style faces: ^o for "o, B for ss, rn for m, a guillemet read as s. Every word of Gemini's then ends up as one of:

confirmed

The readings agree.
Not marked.

decided

They disagreed, and the word was replaced: either the text layer and
tesseract agreed against Gemini, or Jev, shown the printed word cut out of
the scan, confidently picked tesseract's reading.
Marked with a faint grey underline; the word's tooltip says what Gemini had
read.

suspect

They disagreed and nothing settled it, or the word is only in Gemini's
reading.
Marked with a dotted underline; the tooltip says what the OCR read.

inserted

A run of three or more words tesseract read with confidence and Gemini does
not have.
They are put into the text, dashed and in italics, because leaving them out
silently is worse than showing them.
Shorter runs are OCR noise and are listed in the report.

Jev is asked about a disagreement only where the alignment is anchored: where there are confirmed words close on both sides, so that the place on the page the OCR gives for the word is the place of the right word. Each question is asked twice, with the options in opposite orders, since Jev leans towards the first option; an answer counts only if both orders give it with a confidence of at least 0.8. A disagreement Jev does not settle is suspect, unless it is a small difference tesseract itself was unsure of.

A text layer, if there is one, is a third vote: it settles a disagreement between the other two without Jev, and confirms words tesseract missed.

The share of Gemini's words confirmed (including those Jev confirmed) is the page's agreement. A page is flagged if it is below -min-agreement, or has a run of three or more suspect or inserted words. A flagged page is transcribed by Gemini a second time, told which passages to check; the better of the two readings -- the one with fewer marked words -- is kept. A page still flagged has '<?>' before its page marker.

One SQLite file, created mode 0600 because it holds the scans.

blobs

Bytes by SHA-256: PDFs, rendered scans, EPUBs.

documents

Each PDF added, with what
pdfinfo(1)
says, and the course and semester it came from.

steps

Every computed result.
Its
*key*
hashes everything the result depends on -- input bytes, tool and version
or model, prompt, converter version -- so that a change to any of them
computes anew, beside the old row, and nothing else does.
*stage*
is
'`render`',
'`classify`',
'`tesseract`',
'`layer`',
'`gemini`',
'`decide`',
'`merge`'
or
'`biblio`';
*scan*
and
*side*
('`L, '`R, '`S`'`'`')
say which page.
Merges are stored for inspection only and always recomputed.

runs

Each run: options, API calls and tokens per service, flagged pages, the EPUB
and the report.

Nothing is ever deleted. To compute a step again, delete its rows from steps.

GEMINI_API_KEY, TYPESAFE_API_KEY

See
*OPTIONS*.

EPUBIFY_BEARER_TOKEN

Sent with a URL to fetch.

0 on success; 1 if the EPUB was written but pages are still flagged; 2 on an error.

$ epubify convert 'Genette - Paratexte Einleitung S. 9-21.pdf'
$ epubify convert --scans 3-10 -o genette.epub genette.pdf
$ epubify convert --name '2 Genette - Paratexte Einleitung S. 9-21.pdf' \
    --course 'Thematisches Einf"uhrungsseminar: Anfangen' \
    --semester 'WS 2026/27' --tag Paratext --calibre ~/Books genette.pdf
$ epubify report genette
$ sqlite3 ~/.local/state/epubify/epubify.db \
    "select scan, side, attempt, tool from steps where stage = 'gemini'"

calibredb(1), campusweb(1), pdftoppm(1), tesseract(1)

TypeSafe's documentation for Jev: https://docs.typesafe.ai/llms.txt

A misreading both readings share is confirmed. Italics, the attribution of footnotes, the order of blocks and whether a line is a heading are Gemini's alone; the check compares words. Before quoting, compare the passage with the scan.

Gemini's recitation filter stops it reproducing text it recognises as published. Asked for the page as JSON paragraphs, it refused about one page in four of the first book tried, at every temperature; asked to reproduce it line by line for a screen reader, which is what epubify does, none. A page still refused is retried at higher temperatures and then read in horizontal strips of a few lines, cut between lines tesseract found; the report says so.

TypeSafe calls images and boxes "dreamer-ready", not ready for production, and German is not Jev's main language. Its answers decide how scans are split and which reading of a word wins; where they are not confident, code decides, and the report says what was decided how.

A page scanned at an angle of a few degrees loses most of its lines in tesseract. Every page is straightened by the angle, up to five degrees either way, at which its rows of ink are sharpest. A page with little text may be turned wrongly.

Each page is one Gemini request, and each scan one Jev request plus one per sixteen disagreements; a chapter of twenty pages costs a few cents. A second transcription of a flagged page is a second request.

Scans and transcriptions of somebody else's book are in it. It is not to be published, and neither is the EPUB.