Polaris Catalog: Open Source Governance

Last year, Snowflake surprised everyone by announcing Polaris Catalog, an open-source technical catalog for Apache Iceberg.

Wait, why would a closed-source SaaS company release open-source infrastructure? Because they want to be the center of gravity for metadata, even if they aren’t storing the data.

The Problem: Catalog Chaos#

You have data in S3.

• Spark uses the Hive Metastore.
• Trino uses the Glue Catalog.
• Snowflake uses its internal catalog.

They all disagree on the schema. It’s a mess.

The Solution: REST Protocol#

Polaris implements the Iceberg REST Open API. It sits in the middle.

• Spark asks Polaris: “I want to write to Table A.”
• Snowflake asks Polaris: “I want to read Table A.”

Polaris ensures both engines see the same atomic snapshot of the transaction.

Setting it up#

Polaris can be hosted by Snowflake (managed) or run in your own Kubernetes cluster (self-hosted).

-- Connecting Snowflake to a Polaris Catalog
CREATE CATALOG INTEGRATION my_polaris
  CATALOG_SOURCE = 'ICEBERG_REST'
  TABLE_FORMAT = 'ICEBERG'
  URI = 'https://my-polaris-url/api/catalog'
  ENABLED = TRUE;

Governance at the Layer#

The killer feature of Polaris is that it brings RBAC (Role Based Access Control) to the open lake. You can define who can read/write tables in Polaris, and those rules apply whether the user is coming from Spark, Flink, or Dremio.

Conclusion#

Polaris is the “Switzerland” of the Data Wars. It allows you to pick best-of-breed engines for different tasks while engaging in a single source of truth for table structure.