I froze an entire wiki into a Docker bundle — and a URL inside SQLite nearly ruined everything

I wanted something that seemed trivial: take my internal knowledge base — OpenWebUI chats, tools, wiki, and vectors — and make it travel between Docker hosts without rebuilding anything. I copy a folder to another machine, run a script, and I have the exact same assistant answering only about my corpus. No reindexing, no re-configuring the tools by hand, no “ah but the knowledge is missing here”.

The naive plan was: “I’ll tar up OpenWebUI’s data volume and off we go”. It worked for 90%. The remaining 10% cost me an afternoon, and it’s about something you don’t expect: an absolute URL buried inside OpenWebUI’s SQLite.

The idea: one bundle, three pieces, zero secrets

The corpus itself is a confidential project — it’s not relevant here and I won’t touch it. I’m only talking about the packaging technique. The bundle is a folder with three ingredients:

1. The wikipack — the wiki’s markdown tree + the LanceDB vector index already computed + a manifest.json with the (non-secret) embedding config. Frozen vectors: on import nothing gets re-embedded.
2. The wiki-tools tool server — a small FastAPI service that exposes the wiki as an OpenAPI tool server (search, page read, linked pages, page listing), so the chat navigates the wiki instead of doing blind RAG.
3. The OpenWebUI state — a snapshot of the data volume, i.e. webui.db (SQLite) containing everything: users, chats, registered models, knowledge and — above all — the tool server registration.

The snapshot is three lines of export-bundle.sh, and the important part is that it runs inside an alpine container using the volume, without touching the files underneath:

VOLUME="wiki-openwebui_openwebui-data"
docker run --rm -v "$VOLUME":/data -v "$OUT":/backup alpine \
  tar czf /backup/openwebui-data.tgz -C /data .

# .env template col segreto STRAPPATO via sed
sed 's/^GATEWAY_API_KEY=.*/GATEWAY_API_KEY=/' "$HERE/.env" > "$OUT/.env.template"

Golden rule: the gateway key never enters the bundle. It’s stripped from the template with sed, and import-bundle.sh asks for it interactively on first run. Same discipline in the wikipack: make_wikipack.py reads the embedding endpoint and model from the app state but never writes an API key into the manifest.

The manifest: the vector contract

Here’s the heart of “portable vectors”. The manifest.json (sanitized) is tiny:

{
  "wikipackVersion": 1,
  "title": "<wiki-interna>",
  "createdAt": null,
  "embedding": {
    "endpoint": "https://<GATEWAY>/v1/embeddings",
    "model": "Qwen3-Embedding-8B",
    "outputDimensionality": 4096
  },
  "chunkCount": 2468,
  "pageCount": 244,
  "markdownFiles": 249
}

Why is it the contract? Because the index travels already vectorized at 4096 dimensions with a specific model. At query time, the tool server has to embed the question with the exact same model and the same dimension, otherwise comparing the vectors is pure garbage. I made this invariant explicit in the code — if it doesn’t match, it blows up right away with an understandable error instead of silently giving wrong answers:

if self.expected_dim is not None and len(vec) != self.expected_dim:
    raise EmbeddingError(
        f"embedding dim mismatch: got {len(vec)}, index expects {self.expected_dim}. "
        f"Query embedder must use the SAME model+dim as the shipped vectors."
    )

A note on createdAt: null: it’s deliberate. I don’t stamp it at build-time, to avoid introducing nondeterminism — two exports of the same state produce the same file. The CI can stamp it if needed.

Before shipping, make_wikipack calls LanceDB’s table.optimize() on the copy (never on the original): it compacts the fragments and prunes old versions, so the bundle is lean. A nerdy detail I discovered while looking at the files: the version manifest name is 18446744073709551428.manifest — that’s 2^64 − 188, LanceDB numbers versions backwards from UINT64_MAX. Harmless, but the first time it makes you doubt you corrupted something.

The thorn: the URL that wouldn’t travel

Here’s where the “the tar is enough” plan broke. OpenWebUI, when you register a tool server, saves the URL as-is inside webui.db. In my case http://localhost:8100. When you import the bundle on another host and OpenWebUI restarts, it tries to call exactly that URL again. If on the new machine the tool server ended up on a different port, the registration is dead: the chat no longer has tools, and you only notice because the answers get worse.

The fix isn’t elegant but it’s solid: nail the ports down. The compose generated in the bundle doesn’t let anything be chosen at random — it pins HOST_PORT and TOOLS_PORT, and they’re the same values the URL was saved with in SQLite:

  wiki-tools:
    ports:
      - "${TOOLS_PORT:-8100}:8000"   # deve combaciare con l'URL salvato in webui.db
  open-webui:
    ports:
      - "${HOST_PORT:-8300}:8080"

This is the price of portability in this design: the tool registration is an absolute URL frozen in the DB, so the ports become part of the contract. Keep them the same and the bundle “reconnects on its own”; change them and you have to re-register the tool by hand. On my k8s cluster I chose the opposite road — there I inject the same registration via an environment variable (TOOL_SERVER_CONNECTIONS) instead of via SQLite snapshot, so the URL is configurable. Two worlds, two strategies, same lesson: the state that makes the import convenient is also the state that traps you.

Honest gotchas

• The “settings” snapshot weighs 272 MB, the wikipack 13. Counterintuitive surprise: the heavy part isn’t the corpus with its vectors (13 MB), but the OpenWebUI state (openwebui-data.tgz, 272 MB). The “settings” are bigger than the knowledge. Total bundle ~286 MB in the “light” version (it rebuilds the wiki-tools image and pulls OpenWebUI on import); with --with-images it gets much bigger but runs offline.
• The bundle’s compose loses the tool server auth. In the dev compose I pass TOOLS_API_KEY; the heredoc that generates the bundle’s compose doesn’t include it. Result: in the bundle the tool server starts with auth disabled (_load_keys() empty → no check). Handy for an internal demo, but it’s real drift between the two files, and on an untrusted network it’s a hole. To be fixed.
• The volume name is hardcoded. Both export and import assume wiki-openwebui_openwebui-data and --project-name wiki-openwebui. Rename the project and the restore silently points at the wrong volume.
• The export’s source path is absolute. The export script has the build machine’s path nailed inside it: you can only export from there. The bundle is portable; the script that creates it isn’t.
• README/reality diverge on the default ports. The repo’s “dev” compose uses 3000/8000, the one generated in the bundle 8300/8100. If you read the wrong README you connect the wrong port.

How it’s going

It’s shipped and in use as an internal knowledge base. The cycle export → copy → import-bundle.sh → key → up runs, and the chat answers about the corpus with the tools pre-registered. There’s even a key-protected /reload endpoint to reload the updated wikipack without restarting the container. The rough spots (auth lost in the bundle, hardcoded path, fixed volume name) are notes in the notebook, not blockers for internal use.

What I learned

That portability isn’t “copying the files”, it’s understanding where the state keeps its absolute references. A 4096-dimension vector travels just fine — it’s an array of floats, it has no opinions. It’s the http://localhost:8100 buried in a SQLite that betrays you, because it presumes a world that no longer exists on the destination host. The real solution wasn’t writing more code: it was making the ports part of the contract and writing it down in black and white in the README. Sometimes “portable” simply means being honest about what has to stay the same.