inventory — design & vision

A mobile-first, voice+vision inventory capture app. You open a box on your phone, point the camera at what's inside and talk; a Gemini Live agent records the contents into a flexible SQLite graph via tool calls. You review and edit everything afterward in a normal web UI.

Replaces an ad-hoc Excel "what do I have where" spreadsheet.

Interaction model

Phone browser  ──WS (app protocol)──►  Go server  ──WSS (BidiGenerateContent)──►  Gemini Live
  mic PCM16 16k                         (proxy +                                   audio/video/text in
  camera JPEG frames (≤1 FPS)            tool runtime                              toolCall out
  "capture photo" (full-res)            + SQLite)                                  audio/text out
  ◄── model audio 24k / text ──────────
  ◄── UI events (item recorded, …) ────

Two hops on purpose (browser is NOT connected directly to Gemini):

the API key stays server-side, and
the tool calls must touch the DB, which only the server can do.

Open, LLM-driven data model

Rather than a rigid Locations → Boxes → Items hierarchy, the model is given a small graph and a few verbs, and organises things as it sees fit. This is deliberately non-restrictive; we tighten it into views/constraints later once we see what the model actually does.

entities(id, kind, name, notes, …) — one row per thing. kind is free text the model coins: "box", "item", "location", "tote", …
entity_attrs(entity_id, key, value) — arbitrary key/values (qty, colour, condition, brand, …). No migration needed as structure emerges.
relations(from_id, to_id, rel) — free-text edges: "contained_in", "located_at", "part_of", … Expresses "this item is in that box, which is in the cellar" without hardcoding a hierarchy.
photos(id, entity_id, blob, mime, caption, …) — multiple JPEG blobs per entity, stored in SQLite.
capture_sessions(id, started_at, transcript, event_log) — audit of what the model did in a session.

Two distinct image paths

Live frames — browser sends downscaled JPEG frames ≤1 FPS as realtimeInput video so the model can see while you talk. NOT stored.
Archival photos — on request_photo (or a manual tap) the browser captures a full-res JPEG which the server stores as a photos blob linked to the entity. Multiple per entity.

Tool set (Phase 1 — minimal, iterate)

record_entity(kind, name, notes?, attrs?) → new id
update_entity(id, name?, notes?, attrs?)
relate(from_id, to_id, rel)
search(query) → find existing entities so the model reuses "the blue crate" instead of duplicating
request_photo(entity_id, reason) → ask the browser for a full-res archival photo

We start here and add/remove tools as we watch real sessions.

Gemini Live protocol notes (from python-genai `live.py`)

URI: wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent
Auth: x-goog-api-key: <key> handshake header. Key from GEMINI_API_KEY or pass (same as hans).
Handshake: client sends {"setup":{"model":…, "generationConfig":{…}, "systemInstruction":…, "tools":[…]}}; server replies {"setupComplete":{…}}.
Client→server: {"realtimeInput":{…}} (audio/video/text; VAD-driven), {"clientContent":{…}} (turn-based), {"toolResponse":{"functionResponses":[…]}}.
Server→client: {"serverContent":{…}} (has turnComplete, inputTranscription, output audio parts), {"toolCall":{"functionCalls":[…]}}, {"toolCallCancellation":…}.
Audio: in = raw PCM16 16 kHz LE (audio/pcm;rate=16000); out = PCM16 24 kHz LE.
Default model: gemini-3.1-flash-live-preview (overridable via --model). Live-capable models must support bidiGenerateContent; list them with curl "https://generativelanguage.googleapis.com/v1beta/models?key=$KEY". NOTE: an invalid/unavailable model name makes the server close the socket right after setup with a bare EOF (no JSON error).
Response modality: AUDIO out + input transcription.

Transport uses golang.org/x/net/websocket (already vendored): Config.Header carries the auth header, Config.TlsConfig handles wss, DialContext gives cancellation.

Auth (app-level)

Invite-token → session-cookie, copied from weekplan: mint a login link with inventory gen-link, open it once for a long-lived session cookie.

Commands

inventory serve    [--addr host:port] [--base-url url] <db>
inventory gen-link [--admin] [--username name] [--base-url url] <db>

Deferred / not built yet

Excel import — schema is import-ready; importer not built.
Reconciliation / dedup beyond the search tool.
Tightening the open graph into constrained views.

Non-goals

Browser talking directly to Gemini (key exposure; DB access).
A rigid, pre-baked location/box/item hierarchy.