How to connect

Contact data.delivery@spglobal.com to get:

Catalog URIhttps://iceberg.spglobal.com
Token<token>
Token endpointhttps://secureapi.signin.spglobal.com/gettoken/token
Scopeapi
Warehouseyour catalog's warehouse

Examples

Amazon EMR

Prerequisites:

  • An EMR cluster on emr-spark-8.1.0 or later: Spark 4.1, with EMR's Iceberg, 1.11, enabled by the iceberg-defaults classification
  • Library: org.apache.iceberg:iceberg-gcp-bundle of EMR's Iceberg version, 1.11.0 on 8.1, for Cloud Storage
  • Network access to:
    • https://iceberg.spglobal.com, the catalog
    • https://secureapi.signin.spglobal.com/gettoken/token, the token endpoint
    • https://storage.googleapis.com, the data files
  • The client's secret in Secrets Manager, set by the job, rather than in the cluster's configuration

The cluster's configuration, configurations.json:

[
  {
    "Classification": "iceberg-defaults",
    "Properties": {"iceberg.enabled": "true"}
  },
  {
    "Classification": "spark-defaults",
    "Properties": {
      "spark.jars.packages": "org.apache.iceberg:iceberg-gcp-bundle:1.11.0",
      "spark.sql.catalog.<warehouse>": "org.apache.iceberg.spark.SparkCatalog",
      "spark.sql.catalog.<warehouse>.type": "rest",
      "spark.sql.catalog.<warehouse>.uri": "https://iceberg.spglobal.com",
      "spark.sql.catalog.<warehouse>.warehouse": "<warehouse>",
      "spark.sql.catalog.<warehouse>.rest.auth.type": "oauth2",
      "spark.sql.catalog.<warehouse>.oauth2-server-uri": "https://secureapi.signin.spglobal.com/gettoken/token",
      "spark.sql.catalog.<warehouse>.scope": "api",
      "spark.sql.catalog.<warehouse>.token-refresh-enabled": "false",
      "spark.sql.catalog.<warehouse>.header.X-Iceberg-Access-Delegation": "vended-credentials",
      "spark.sql.catalog.<warehouse>.io-impl": "org.apache.iceberg.gcp.gcs.GCSFileIO"
    }
  }
]
aws emr create-cluster --release-label emr-spark-8.1.0 --applications Name=Spark \
  --configurations file://configurations.json ...

Or with Terraform:

resource "aws_emr_cluster" "iceberg" {
  name          = "<cluster>"
  release_label = "emr-spark-8.1.0"
  applications  = ["Spark"]
  service_role  = "<service role>"
  # ec2_attributes, master_instance_group and core_instance_group as for any cluster

  configurations_json = jsonencode([
    {
      Classification = "iceberg-defaults"
      Properties     = { "iceberg.enabled" = "true" }
    },
    {
      Classification = "spark-defaults"
      Properties = {
        "spark.jars.packages"                                              = "org.apache.iceberg:iceberg-gcp-bundle:1.11.0"
        "spark.sql.catalog.<warehouse>"                                    = "org.apache.iceberg.spark.SparkCatalog"
        "spark.sql.catalog.<warehouse>.type"                               = "rest"
        "spark.sql.catalog.<warehouse>.uri"                                = "https://iceberg.spglobal.com"
        "spark.sql.catalog.<warehouse>.warehouse"                          = "<warehouse>"
        "spark.sql.catalog.<warehouse>.rest.auth.type"                     = "oauth2"
        "spark.sql.catalog.<warehouse>.oauth2-server-uri"                  = "https://secureapi.signin.spglobal.com/gettoken/token"
        "spark.sql.catalog.<warehouse>.scope"                              = "api"
        "spark.sql.catalog.<warehouse>.token-refresh-enabled"              = "false"
        "spark.sql.catalog.<warehouse>.header.X-Iceberg-Access-Delegation" = "vended-credentials"
        "spark.sql.catalog.<warehouse>.io-impl"                            = "org.apache.iceberg.gcp.gcs.GCSFileIO"
      }
    },
  ])
}

In the job, before the first query:

spark.conf.set("spark.sql.catalog.<warehouse>.credential", "<client id>:<client secret>")
spark.conf.set("spark.sql.catalog.<warehouse>.token", "<token>")

Documentation:

Apache Spark

Prerequisites:

  • Spark 4.0 or later
  • Libraries: org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.12.0 and org.apache.iceberg:iceberg-gcp-bundle:1.12.0. in Gradle:
    dependencies {
        runtimeOnly("org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.12.0")
        runtimeOnly("org.apache.iceberg:iceberg-gcp-bundle:1.12.0")
    }
  • Network access to:
    • https://iceberg.spglobal.com, the catalog
    • https://secureapi.signin.spglobal.com/gettoken/token, the token endpoint
    • https://storage.googleapis.com, the data files

Session configuration:

spark.sql.catalog.<warehouse>=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.<warehouse>.type=rest
spark.sql.catalog.<warehouse>.uri=https://iceberg.spglobal.com
spark.sql.catalog.<warehouse>.warehouse=<warehouse>

spark.sql.catalog.<warehouse>.rest.auth.type=oauth2
spark.sql.catalog.<warehouse>.oauth2-server-uri=https://secureapi.signin.spglobal.com/gettoken/token
spark.sql.catalog.<warehouse>.credential=<client id>:<client secret>
spark.sql.catalog.<warehouse>.scope=api
spark.sql.catalog.<warehouse>.token=<token>
spark.sql.catalog.<warehouse>.token-refresh-enabled=false
spark.sql.catalog.<warehouse>.header.X-Iceberg-Access-Delegation=vended-credentials

spark.sql.catalog.<warehouse>.io-impl=org.apache.iceberg.gcp.gcs.GCSFileIO

Querying:

SHOW NAMESPACES IN <warehouse>;

SHOW TABLES IN <warehouse>.<namespace>;

SELECT * FROM <warehouse>.<namespace>.<table> LIMIT 10;

Documentation:

Databricks

Prerequisites:

  • Databricks Runtime 17.0 or later: Spark 4.0, Scala 2.13
  • A cluster in dedicated (single user) or no isolation shared access mode
  • Libraries: org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.12.0 and org.apache.iceberg:iceberg-gcp-bundle:1.12.0
  • The client's <client id>:<client secret> in a Databricks secret scope

In the cluster's Spark config:

spark.sql.catalog.<warehouse> org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.<warehouse>.type rest
spark.sql.catalog.<warehouse>.uri https://iceberg.spglobal.com
spark.sql.catalog.<warehouse>.warehouse <warehouse>

spark.sql.catalog.<warehouse>.rest.auth.type oauth2
spark.sql.catalog.<warehouse>.oauth2-server-uri https://secureapi.signin.spglobal.com/gettoken/token
spark.sql.catalog.<warehouse>.credential {{secrets/<scope>/<key>}}
spark.sql.catalog.<warehouse>.scope api
spark.sql.catalog.<warehouse>.token <token>
spark.sql.catalog.<warehouse>.token-refresh-enabled false
spark.sql.catalog.<warehouse>.header.X-Iceberg-Access-Delegation vended-credentials

spark.sql.catalog.<warehouse>.io-impl org.apache.iceberg.gcp.gcs.GCSFileIO

Then query as in Spark:

SELECT * FROM <warehouse>.<namespace>.<table> LIMIT 10;

Documentation:

DuckDB

Prerequisites:

  • DuckDB 1.5 or later with extensions: httpfs, avro and iceberg
  • Network access to:
    • https://iceberg.spglobal.com, the catalog
    • https://secureapi.signin.spglobal.com/gettoken/token, the token endpoint
    • https://storage.googleapis.com, the data files
INSTALL httpfs; INSTALL avro; INSTALL iceberg;
LOAD httpfs; LOAD avro; LOAD iceberg;

CREATE SECRET iceberg_client (
CREATE SECRET iceberg_token (
    TYPE iceberg,
    CLIENT_ID '<client id>',
    CLIENT_SECRET '<client secret>',
    OAUTH2_SERVER_URI 'https://secureapi.signin.spglobal.com/gettoken/token',
    OAUTH2_SCOPE 'api'
    TOKEN '<token>'
);

ATTACH '<warehouse>' AS <warehouse> (
    TYPE iceberg,
    SECRET iceberg_client,
    SECRET iceberg_token,
    ENDPOINT 'https://iceberg.spglobal.com'
);

SHOW ALL TABLES;

SELECT * FROM <warehouse>.<namespace>.<table> LIMIT 10;

Documentation:

Google Managed Service for Apache Spark (formerly Dataproc)

Prerequisites:

  • A cluster on image 3.0 or later: Spark 4.0, with the optional Iceberg component, Iceberg 1.11
  • Library: org.apache.iceberg:iceberg-gcp-bundle of the component's Iceberg version, 1.11.0 on 3.0, for Cloud Storage
  • io-impl set to GCSFileIO, as below: else Iceberg reads through the cluster's Cloud Storage connector, as its service account, not with the vended credentials
  • Network access to:
    • https://iceberg.spglobal.com, the catalog
    • https://secureapi.signin.spglobal.com/gettoken/token, the token endpoint
    • https://storage.googleapis.com, the data files
  • The client's secret in Secret Manager, set by the job, rather than in the cluster's properties

The cluster:

gcloud dataproc clusters create <cluster> --region=<region> --image-version=3.0 \
  --optional-components=ICEBERG \
  --properties="^#^spark:spark.jars.packages=org.apache.iceberg:iceberg-gcp-bundle:1.11.0#\
spark:spark.sql.catalog.<warehouse>=org.apache.iceberg.spark.SparkCatalog#\
spark:spark.sql.catalog.<warehouse>.type=rest#\
spark:spark.sql.catalog.<warehouse>.uri=https://iceberg.spglobal.com#\
spark:spark.sql.catalog.<warehouse>.warehouse=<warehouse>#\
spark:spark.sql.catalog.<warehouse>.rest.auth.type=oauth2#\
spark:spark.sql.catalog.<warehouse>.oauth2-server-uri=https://secureapi.signin.spglobal.com/gettoken/token#\
spark:spark.sql.catalog.<warehouse>.scope=api#\
spark:spark.sql.catalog.<warehouse>.token-refresh-enabled=false#\
spark:spark.sql.catalog.<warehouse>.header.X-Iceberg-Access-Delegation=vended-credentials#\
spark:spark.sql.catalog.<warehouse>.io-impl=org.apache.iceberg.gcp.gcs.GCSFileIO"

Or with Terraform:

resource "google_dataproc_cluster" "iceberg" {
  name   = "<cluster>"
  region = "<region>"

  cluster_config {
    software_config {
      image_version       = "3.0"
      optional_components = ["ICEBERG"]
      override_properties = {
        "spark:spark.jars.packages"                                              = "org.apache.iceberg:iceberg-gcp-bundle:1.11.0"
        "spark:spark.sql.catalog.<warehouse>"                                    = "org.apache.iceberg.spark.SparkCatalog"
        "spark:spark.sql.catalog.<warehouse>.type"                               = "rest"
        "spark:spark.sql.catalog.<warehouse>.uri"                                = "https://iceberg.spglobal.com"
        "spark:spark.sql.catalog.<warehouse>.warehouse"                          = "<warehouse>"
        "spark:spark.sql.catalog.<warehouse>.rest.auth.type"                     = "oauth2"
        "spark:spark.sql.catalog.<warehouse>.oauth2-server-uri"                  = "https://secureapi.signin.spglobal.com/gettoken/token"
        "spark:spark.sql.catalog.<warehouse>.scope"                              = "api"
        "spark:spark.sql.catalog.<warehouse>.token-refresh-enabled"              = "false"
        "spark:spark.sql.catalog.<warehouse>.header.X-Iceberg-Access-Delegation" = "vended-credentials"
        "spark:spark.sql.catalog.<warehouse>.io-impl"                            = "org.apache.iceberg.gcp.gcs.GCSFileIO"
      }
    }
  }
}

A job, which sets the credential before its first query:

gcloud dataproc jobs submit pyspark job.py --cluster=<cluster> --region=<region>
spark.conf.set("spark.sql.catalog.<warehouse>.credential", "<client id>:<client secret>")
spark.conf.set("spark.sql.catalog.<warehouse>.token", "<token>")

Documentation:

PyIceberg

Prerequisites:

  • PyIceberg with Cloud Storage and pandas: pip install "pyiceberg[gcsfs,pandas]"
  • Network access to:
    • https://iceberg.spglobal.com, the catalog
    • https://secureapi.signin.spglobal.com/gettoken/token, the token endpoint
    • https://storage.googleapis.com, the data files

Then:

from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "<warehouse>",
    type="rest",
    uri="https://iceberg.spglobal.com",
    warehouse="<warehouse>",
    auth={
        "type": "oauth2",
        "oauth2": {
            "client_id": "<client id>",
            "client_secret": "<client secret>",
            "token_url": "https://secureapi.signin.spglobal.com/gettoken/token",
            "scope": "api",
        },
    },
    token="<token>",
    **{"header.X-Iceberg-Access-Delegation": "vended-credentials"},
)

print(catalog.list_namespaces())
print(catalog.list_tables("<namespace>"))
print(catalog.load_table(("<namespace>", "<table>")).scan(limit=10).to_pandas())

Documentation:

Snowflake

Prerequisites:

  • A role that may create catalog integrations: ACCOUNTADMIN, or one granted CREATE INTEGRATION ON ACCOUNT
CREATE CATALOG INTEGRATION data_fabric
    CATALOG_SOURCE = ICEBERG_REST
    TABLE_FORMAT = ICEBERG
    CATALOG_NAMESPACE = '<namespace>'
    REST_CONFIG = (
        CATALOG_URI = 'https://iceberg.spglobal.com'
        WAREHOUSE = '<warehouse>'
        ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS
    )
    REST_AUTHENTICATION = (
        TYPE = OAUTH
        OAUTH_TOKEN_URI = 'https://secureapi.signin.spglobal.com/gettoken/token'
        OAUTH_CLIENT_ID = '<client id>'
        OAUTH_CLIENT_SECRET = '<client secret>'
        OAUTH_ALLOWED_SCOPES = ('api')
        TYPE = BEARER
        BEARER_TOKEN = '<token>'
    )
    ENABLED = TRUE;

CREATE ICEBERG TABLE <table>
    CATALOG = 'data_fabric'
    CATALOG_TABLE_NAME = '<table>';

SELECT * FROM <table> LIMIT 10;

Documentation:

Trino

Prerequisites:

  • Trino 481 or later
  • Network access to:
    • https://iceberg.spglobal.com, the catalog
    • https://secureapi.signin.spglobal.com/gettoken/token, the token endpoint
    • https://storage.googleapis.com, the data files

A catalog, etc/catalog/<warehouse>.properties:

connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=https://iceberg.spglobal.com
iceberg.rest-catalog.warehouse=<warehouse>
iceberg.rest-catalog.security=OAUTH2
iceberg.rest-catalog.oauth2.server-uri=https://secureapi.signin.spglobal.com/gettoken/token
iceberg.rest-catalog.oauth2.credential=<client id>:<client secret>
iceberg.rest-catalog.oauth2.scope=api
iceberg.rest-catalog.oauth2.token=<token>
iceberg.rest-catalog.oauth2.token-refresh-enabled=false
iceberg.rest-catalog.oauth2.token-exchange-enabled=false
iceberg.rest-catalog.vended-credentials-enabled=true
fs.gcs.enabled=true
SHOW SCHEMAS FROM <warehouse>;
SHOW TABLES FROM <warehouse>.<namespace>;
SELECT * FROM <warehouse>.<namespace>.<table> LIMIT 10;

Documentation: