Key takeaways
-
- Google Cloud’s borderless Lakehouse leverages the open Apache Iceberg REST catalog specification to federate Databricks Unity Catalog, Snowflake Horizon, and AWS Glue into BigQuery without data migration.
- All analytical execution runs inside Google Cloud compute engines like BigQuery or Managed Service for Apache Spark, reading remote Parquet files directly via short-lived credentials vended over Workload Identity Federation.
- BigQuery implements regional caching for remote data blocks to lower query latency and egress expenses, automatically bypassing cache when customer-managed encryption key policies require direct reads.
- Conversational analytics in BigQuery uses Gemini models to translate plain-language business questions into federated SQL queries across multi-cloud logical views.
Google Cloud’s borderless Lakehouse enables cross-cloud conversational analytics by federating external Apache Iceberg REST catalogs into BigQuery without data migration. Rather than copying datasets across providers with fragile extract, transform, and load (ETL) pipelines, this architecture queries data in place across Databricks Unity Catalog, Snowflake Horizon Catalog, and AWS Glue Data Catalog. By decoupling periodic metadata synchronization from runtime storage retrieval, Google Cloud compute engines read remote Parquet files directly from object storage such as Amazon S3 using temporary vended credentials. Enterprise analysts can then query cross-cloud tables using standard SQL or interact with unified datasets using Gemini-powered conversational analytics in natural language.
Decoupling metadata from remote storage
Enterprise data rarely resides in a single public cloud. As business divisions deploy specialized platforms, analytical assets splinter across Databricks, Snowflake, and AWS environments. Historically, reconciling this data demanded scheduled batch extraction pipelines that pulled remote tables into a central data warehouse. This pattern creates continuous operational overhead: scheduled synchronizations introduce data lag, duplicate storage budgets, generate recurring egress fees, and break downstream dashboards whenever upstream schemas shift.
The borderless Lakehouse architecture addresses this friction by bringing compute directly to existing storage repositories. Built on the open Apache Iceberg REST catalog specification, the framework separates catalog synchronization from storage file access:
- Metadata synchronization: The Lakehouse runtime catalog runs asynchronous background processes to periodically synchronize schemas, table definitions, and snapshot pointers from remote catalogs according to an administrator-defined interval.
- Direct storage retrieval: During analytical execution, BigQuery or Managed Service for Apache Spark reads raw Parquet data files directly from remote object stores, such as Amazon S3, using temporary scoped access credentials vended by the source platform.
Crucially, remote compute engines remain entirely idle. Neither Databricks clusters, Snowflake virtual warehouses, nor AWS compute instances execute query logic or incur runtime compute billing. The entire analytical processing workload runs on Google Cloud compute infrastructure.
- Catalog provisioning: The platform engineer registers the external Iceberg REST catalog within Google Cloud using BigLake APIs.
- Identity resolution: The engineer inspects the federated catalog resource to extract Google Cloud’s automatically generated service identity.
- Trust delegation: The external cloud platform assigns traversal and read privileges to this service identity via Workload Identity Federation (WIF) and credential vending policies.
Federating Databricks, Snowflake, and AWS Glue
Because Databricks Unity Catalog, Snowflake Horizon Catalog, and AWS Glue Data Catalog support the Apache Iceberg REST catalog specification, administrators can establish unified access governance through standardized configuration workflows.
| Platform | Catalog Interface | Identity & Traversal Grants | Credential Exchange Mechanism |
|---|---|---|---|
| Databricks | Unity Catalog Iceberg REST | GRANT USE CATALOG, EXTERNAL USE SCHEMA, SELECT |
Workload Identity Federation to Databricks Service Principal |
| Snowflake | Horizon Catalog Iceberg REST | USAGE on Database/Schema, SELECT on Table |
Service User bound to Google Cloud Subject ID via WIF |
| AWS Glue | AWS Glue Data Catalog | IAM Policy (glue:Get*, s3:ListBucket, s3:GetObject) |
IAM AssumeRoleWithWebIdentity (43,200s session limit) |
Databricks Unity Catalog integration
In Databricks, the target Iceberg schema must be anchored to an external storage location registered within Unity Catalog. An administrator enables external data access in the metastore settings and provisions a Databricks Service Principal. In Google Cloud, engineers execute a declarative registration:
gcloud alpha biglake iceberg catalogs create dbx_retail_db \
--project="my-lakehouse-project" \
--primary-location="us-central1" \
--catalog-type="federated" \
--federated-catalog-type="unity" \
--unity-instance-name="dbc-xxxxxxxx-xxxx.cloud.databricks.com" \
--unity-catalog-name="dbx_retail_db" \
--unity-service-principal-application-id="6ea7b861-c5a5-xxxx-xxxx-xxxxxxxxxxxx" \
--refresh-interval="300s"After describing the created catalog to retrieve Google Cloud’s numeric subject identifier, the administrator grants permissions in Databricks SQL. The EXTERNAL USE SCHEMA privilege is mandatory, as it instructs Unity Catalog to vend short-lived object storage credentials to BigQuery during query execution. Federation policy is completed by establishing an identity binding where the token issuer is https://accounts.google.com and the audience is https://accounts.cloud.databricks.com.
Snowflake Horizon Catalog configuration
Snowflake managed Iceberg tables expose an Apache Iceberg REST interface natively. Integration requires creating a dedicated reader role, granting USAGE across the target database and schema, and adding SELECT on the Iceberg tables. When registering the federated catalog via the Google Cloud CLI, the --snowflake-warehouse flag references the root database identifier (such as SF_RETAIL_DB) rather than a Snowflake virtual compute warehouse:
gcloud alpha biglake iceberg catalogs create sf_retail_db \
--project="my-lakehouse-project" \
--primary-location="us-central1" \
--catalog-type="federated" \
--federated-catalog-type="snowflake" \
--snowflake-account-identifier="xy12345-ab67890" \
--snowflake-warehouse="SF_RETAIL_DB" \
--snowflake-role="gcp_lakehouse_reader_role" \
--refresh-interval="300s"In Snowflake, administrators bind the Google Cloud subject ID to a passwordless service user using Workload Identity Federation. This configuration permits Snowflake Horizon to validate incoming Google Cloud requests over OpenID Connect and vend short-lived table read tokens without hardcoded credentials.
AWS Glue Data Catalog provisioning
Federating AWS Glue requires provisioning an IAM role configured with an sts:AssumeRoleWithWebIdentity trust document. Setting the maximum session duration to 43,200 seconds (12 hours) prevents security credential expiration during complex analytical workloads. The attached permissions policy must explicitly grant glue:Get* permissions alongside Amazon S3 s3:ListBucket and s3:GetObject actions.
When environments utilize AWS Lake Formation, administrators must verify that the IAMAllowedPrincipals group maintains access or assign explicit Lake Formation permissions to the federated IAM role. To avoid synchronization race conditions, the catalog is initially provisioned without a refresh interval, described to extract its assigned service account identity, updated in the AWS IAM trust policy, and subsequently patched to activate a 300-second metadata refresh cycle.
Executing four-part queries and conversational analysis
Once catalogs are synchronized, BigQuery addresses external Iceberg tables using a standardized four-part path syntax: project.catalog.namespace.table. This namespace convention allows engineers to construct a single SQL query that joins internal BigQuery transactions with dimension tables distributed across third-party cloud environments.
SELECT
c.first_name || ' ' || c.last_name AS customer_name,
p.product_name,
s.store_name,
SUM(f.quantity) AS total_units_sold,
SUM(f.total_revenue) AS total_sales_revenue
FROM `my-lakehouse-project.local_retail_dataset.fact_sales` f
INNER JOIN `my-lakehouse-project.sf_retail_db.CORE.DIM_CUSTOMERS` c
ON f.customer_key = c.customer_key
INNER JOIN `my-lakehouse-project.dbx_retail_db.dbx_core.dim_products` p
ON f.product_key = p.product_key
INNER JOIN `my-lakehouse-project.glue_retail_db.glue_retail_db.dim_stores` s
ON f.store_key = s.store_key
GROUP BY 1, 2, 3
ORDER BY total_sales_revenue DESC
LIMIT 10;Identifier capitalization reflects source platform semantics: Snowflake namespaces typically mandate uppercase notation (CORE.DIM_CUSTOMERS), whereas Databricks and AWS Glue schemas maintain lowercase paths.
To support business teams, data engineers encapsulate federated queries into logical views. Because federated Iceberg structures inherit native BigQuery schema governance, downstream features—including conversational analytics in BigQuery—operate directly against these views. Business stakeholders can query complex multi-cloud relationships using plain English, such as asking which customer segments generated top store revenues across regional outlets. Gemini models evaluate the federated view definition, translate the question into optimized SQL, coordinate multi-cloud data retrieval, and present synthesized tabular and narrative answers.
Caching, residency, and cryptographic constraints
Operating a federated multi-cloud architecture introduces specific networking and governance trade-offs that systems engineers must incorporate into baseline capacity planning.
Metadata refresh cycles and backpressure
The --refresh-interval flag controls synchronization frequency, with an enforced minimum of 300 seconds. For large enterprise catalogs containing thousands of partitions and tables, metadata extraction can exceed this five-minute window. When a synchronization run overruns its scheduled interval, the Lakehouse runtime catalog automatically skips the subsequent execution cycle to avoid queuing task backlogs. Systems teams should monitor synchronization duration to verify metadata freshness targets.
Regional caching and egress economics
To mitigate wide-area network latency and unpredictable cloud egress fees, BigQuery automatically caches retrieved remote storage blocks inside the Google Cloud region hosting the federated catalog (for instance, us-central1). Repeated queries against static or slowly changing dimensions read directly from local cache rather than re-fetching objects across cloud boundaries. Platform teams can inspect cache efficiency inside BigQuery execution plans by evaluating objectStorageStats to compare cacheBytesRead against total objectStorageBytesRead.
Data sovereignty and CMEK policy enforcement
While caching provides latency and egress advantages, it temporarily persists remote data on Google Cloud storage disks within the catalog’s region. Compliance officers must evaluate whether this staging aligns with corporate data residency obligations and geographic boundaries.
Furthermore, cross-cloud cache blocks are encrypted by default using Google-managed encryption keys. If an enterprise enforces strict organizational guardrails—specifically the constraints/gcp.restrictNonCmekServices constraint—BigQuery disables the regional caching layer entirely. In this scenario, every query bypasses local storage and fetches raw Parquet blocks directly from the source cloud provider over the network. Organizations adhering to multi-cloud customer-managed encryption policies must account for the resulting latency and egress cost profiles when architecting analytical workloads.
Bottom line
Google Cloud’s borderless Lakehouse eliminates the traditional trade-off between centralized pipeline maintenance and fragmented multi-cloud reporting. By adopting the open Apache Iceberg REST catalog specification as a universal control plane, engineering organizations can execute federated analytical joins across Databricks, Snowflake, and AWS Glue while restricting compute expenses to Google Cloud.
However, successful adoption requires rigorous operational discipline. Platform architects must evaluate data residency compliance when utilizing regional caching, configure robust IAM credential vending across cloud providers, and establish monitoring on catalog sync intervals. For organizations balancing multi-cloud deployments, borderless Lakehouse delivers an open, standard-based path to enterprise-wide data access without the weight of legacy extraction pipelines.
Sources
- Cross-cloud conversational analytics with borderless Lakehouse | Google Cloud Blog
- About the Lakehouse runtime catalog | Google Cloud Documentation
- About cross-cloud data access | Lakehouse | Google Cloud Documentation
- Conversational Analytics in BigQuery now GA | Google Cloud Blog
- Borderless Lakehouse cross-cloud caching and connections | Google Cloud Blog
- What is borderless Lakehouse? | Google Cloud Documentation
