Connect Databricks

CostReveal reads your Databricks account's billable usage, every workspace and region, day by day, and prices it at Databricks' list prices.

What CostReveal reads

• Usage, from system.billing.usage: each day's usage by workspace, SKU and product, such as jobs, SQL, all-purpose compute, pipelines and model serving, and the job, pipeline, SQL warehouse, serving endpoint, app or cluster it ran on. Corrections Databricks makes later are restated.
• List prices, from system.billing.list_prices: the price in effect when the usage started.
• Tags, the custom tags on clusters, jobs, warehouses, pipelines and budget policies, read with your project's tag keys: a job tagged team = payments is the payments team's.
• Workspace names, from system.access.workspaces_latest, when granted.

Who invoices you decides the figures

• Databricks invoices you. Databricks does not share what it invoices, so its usage is priced at list price: estimates before any discount in your contract. Home says so, and shows no second source for them.
• A cloud invoices you. Azure Databricks, Databricks on Google Cloud, and Databricks bought through AWS Marketplace are on that cloud's invoice, so their spend is already in the cloud's connection, exactly as invoiced. Connect the cloud in the same project too: CostReveal divides its Databricks charges by the list cost of the usage they bill for, each day on its own, and adds nothing to them.

Under the Databricks connection, the latest month is shown by workspace, product and resource either way.

Set it up

1. In Connections, choose Connect next to Databricks. CostReveal makes a key for the connection.
2. In the account console, under User management, add a service principal, and add it to a workspace.
3. On the service principal, under Credentials & secrets, Federation policies, create a policy with the issuer, subject, audience and JWKS shown, or create it with the Databricks CLI command shown.
4. As a metastore admin, run the grants shown in the SQL editor, and give the service principal Can use on a SQL warehouse.
5. Enter the workspace URL, the service principal's application ID and the warehouse ID, and say who invoices you.

CostReveal signs in and reads the account's usage before saving anything. If something is missing, it says which: a policy that does not match, a warehouse that is not there, or grants that are missing.

No password or secret of yours changes hands. Deleting the federation policy, or the service principal, ends access at once.

How a cloud's Databricks charges are divided

• Found on the bill by what each cloud says of them: an Azure Databricks workspace's own resource, a Google Cloud service named Databricks, or an AWS Marketplace product named Databricks.
• Divided by usage. On Azure each workspace's charge is divided by that workspace's usage, when its name is granted; otherwise, and on the marketplaces, by the whole account's usage that day.
• Owners named by the usage's tags are allocated their share of what the cloud line's own tags left unexplained, basis ALLOCATED. A day with no usage read yet is shown as not divided.
• Claimed once. A Databricks charge is never also shared by a split rule or by usage data. A Kubernetes cluster's billing tag claims a line first.
• Fixed per connection. Who invoices the account cannot change on a connection: remove it and connect again.

Freshness, and what it costs you

• Databricks records usage within hours, and corrects it later. CostReveal reads every six hours, and reads the current and previous month each time. The first read also brings in the twelve months before the current one, as far as Databricks holds them. Where Databricks holds no older months, Connections says how far back it reached, and each read asks again.
• A month is summarized first, and read in full only when its usage has changed. Reading takes a few seconds a day of the SQL warehouse, which Databricks charges you for.