Openbricks: when the governance plan collapsed on contact with reality

Databricks is convenient. It’s also a closed box on someone else’s cloud with a bill that keeps growing. The question that got me started: how much of that convenience can you rebuild from open-source pieces, in-house, on your own iron? Openbricks is the attempt — an 8-layer lakehouse deployed 100% via GitOps on bare-metal OpenShift 4.18, air-gapped. And the real story is how the governance plan died and was reborn.

The stack in brief

Eight layers, from identity to storage: Keycloak (OIDC federated to LDAP/AD) and OPA on top; then OpenShift’s HAProxy router; the apps (JupyterHub, MLflow, Airflow 3, Superset, Kestra); the processing (Spark 4.0.2 via the K8s operator, Trino 479); the catalog (Unity Catalog OSS, OpenMetadata); the storage (Delta Lake on MinIO, PostgreSQL HA via CloudNativePG, Ceph); the control plane (ArgoCD, monitoring); and the GitOps deploy. Cluster: 3 workers, ~288 cores, ~768 Gi RAM.

How data flows (the first decision)

The pattern is “Unity Catalog as a passive metastore”:

• Write: Spark writes Delta tables with the UC connector (registering the metadata) but writes the bytes directly to MinIO with its own credentials.
• Read: Trino reads Delta tables directly — a file-based metastore over the transaction log in object storage — and pulls the bytes from MinIO. UC isn’t even in the read path: more passive than that isn’t possible.
• Why passive: UC OSS v0.4.0 doesn’t do credential vending to S3-compatible endpoints like MinIO (the upstream PR is still open). So UC neither validates nor distributes S3 tokens; the engines hold their own credentials, and storage is protected at the infra level. This keeps UC replaceable at zero cost.

The pivot: Ranger out, three pieces in

The plan was elegant: Apache Ranger as the single policy plane for Trino and Spark, plus OPA for Kubernetes. One place to say who sees what. Then reality:

Ranger 2.7 has a native plugin for Trino, but there’s no Ranger plugin for Spark 4.0. Ranger would cover the query engine, not the write engine — so it can’t be the “single plane.” Removed from the stack.

Redone as multi-layer authorization, each layer good at what it does: Keycloak+oauth2-proxy for platform access, OPA Gatekeeper for K8s admission, Trino’s file-based ACL for data access at query time, Unity Catalog GRANTs (ready but disabled — the UC UI has no Keycloak login, another upstream issue), OpenShift RBAC/SCC for the infra.

How permissions stay consistent across engines

There’s no single ABAC engine covering both Trino and Spark. Consistency is achieved by making identity the shared backbone and constraining where each engine can write:

1. Unified identity at Keycloak. Every engine authenticates the same person against the same realm, behind a single AD. The group is the source of truth: team = AD group, mapped into the token’s groups claim.
2. Trino enforcement. Trino runs with access-control.name=file plus a group-provider. Humans come in via OAuth2; non-interactive services present a Keycloak Bearer token validated via JWKS — with one necessary trick: the principal is read from a dedicated trino_user claim, not from preferred_username (which for a service account collides as service-account-…).

http-server.authentication.type=oauth2,jwt
http-server.authentication.jwt.principal-field=trino_user   # claim dedicato, non preferred_username
http-server.authentication.jwt.required-audience=trino

3. Spark enforcement. No Spark authorizer: you limit the blast radius. The only service authorized to write into the ingest schema is the governed ingest identity; everyone else has read-only on the query path. Governance sits on the query path (Trino) and on the write identity, giving consistent effective permissions without a Spark plugin.

The Trino ACL, in sanitized form — deny-by-default via a catch-all with empty privileges:

{ "tables": [
  { "group": "platform-ingest|admin", "catalog": "delta", "schema": "ingest",
    "table": ".*", "privileges": ["SELECT","INSERT","DELETE","UPDATE","OWNERSHIP"] },
  { "group": "hr|admin", "catalog": "delta", "schema": "sensitive_hr",
    "table": ".*", "privileges": ["SELECT"] },
  { "user": ".*", "catalog": "delta", "schema": "sensitive_hr",
    "table": ".*", "privileges": [] } ] }

The proof it works: an end-to-end test on the live cluster — the service JWT writes to ingest = OK; writes outside = PERMISSION_DENIED; invalid JWT = 401; and the human OAuth2 login stays intact because the two authenticators coexist.

GitOps and air-gap

Everything is ArgoCD app-of-apps: a root Application owns ~15 children, ordered with sync-waves (foundation → processing → app). Kustomize base + testing/production overlays. Being air-gapped, every Helm chart is vendored as an OCI artifact on an internal Quay — no pulls from Docker Hub at sync time. Secrets are SealedSecrets committed to git; the only one applied by hand is the repo credential to Quay (sealing it would create a circular deadlock).

Honest gotchas

• UC Iceberg REST is read-only and one of its DDLs uses BINARY(16), which PostgreSQL rejects.
• Two disjoint group systems (Keycloak vs Trino’s group.txt), additive — a CronJob that regenerates group.txt from Keycloak is the planned unification.
• group.txt mounted via subPath doesn’t hot-reload: it needs a rollout restart of the coordinator.
• The corporate CA isn’t published everywhere, handled in three different ways: a JKS truststore for the Java components, the CA baked into the image for Python (Superset), skip-verify for Go (oauth2-proxy).
• OpenShift ≠ vanilla K8s: Route not Ingress, SCC not PodSecurity, oc not kubectl.

What I’m learning

That “open-source” doesn’t mean “free of effort.” The bricks are free, but the mortar that holds them together — identity, consistent policy across engines, deploy reproducibility — you have to supply yourself. That mortar is exactly what you’re paying for when you pay for a closed platform. And that an elegant architecture plan holds up right until it meets “there’s no plugin for your version.”

I’m building it for a real context with real requirements. Names and recipient stay out: what matters here is the how, not the for-whom. Next stop: closing out governance and moving to the ML layer.