Customer-Managed Keys & Encryption

Databricks encrypts every byte at rest with AES-256 by default. Customer-Managed Keys (CMK), also marketed as Bring-Your-Own-Key (BYOK), replace the Databricks-owned root key with one you own in AWS KMS, Azure Key Vault, or GCP KMS. You keep the key material, the rotation schedule, and the audit trail — and if you revoke the key, every dependent surface goes dark within seconds.

1. Envelope Encryption Hierarchy

Databricks uses a standard envelope-encryption scheme. The customer's KMS key never leaves the HSM — it only wraps a per-object Data Encryption Key (DEK), which in turn encrypts the actual bytes.

┌──────────────────────────────────────────────────────────────────────────────────┐
│                     LAYER 1 — KMS ROOT KEY (Customer-Owned)                      │
│        AWS KMS  |  Azure Key Vault  |  GCP KMS  |  HSM-backed, revocable         │
└──────────────────────────────────────────────────────────────────────────────────┘
                                         │  wraps
                                         ▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│                   LAYER 2 — DATA ENCRYPTION KEY (DEK, AES-256)                   │
│      Generated per object/volume  |  Wrapped by KMS root  |  Cached briefly      │
└──────────────────────────────────────────────────────────────────────────────────┘
                                         │  encrypts
                                         ▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│                              LAYER 3 — DATA AT REST                              │
│       Notebooks  |  Secrets  |  DBFS Root  |  Cluster EBS  |  Delta Tables       │
└──────────────────────────────────────────────────────────────────────────────────┘

2. CMK for Managed Services (Notebooks & Secrets)

The Managed Services CMK protects items the Databricks control plane stores on your behalf:

• Notebook source (cells, outputs).
• Personal access tokens.
• Secret-scope contents (the Databricks-backed scope, not the Key Vault scope).
• Job configuration JSON.
• Query results stored by SQL warehouses.

You configure it on the workspace at creation time:

databricks account customer-managed-keys create \
  --use-cases MANAGED_SERVICES \
  --aws-key-info '{"key_arn":"arn:aws:kms:us-east-1:123:key/abcd","key_alias":"alias/dbx-managed"}'

The resulting key configuration is then attached to the workspace.

3. CMK for Workspace Storage (DBFS Root)

The Storage CMK encrypts the workspace's root S3 bucket (AWS) or ADLS Gen2 container (Azure) — the so-called DBFS root. This is where init scripts, libraries, MLflow artifacts, and the /databricks-datasets mount live.

Note: the Storage CMK does not automatically apply to Unity Catalog managed tables. Those live in UC storage credentials pointing at customer-owned buckets — encrypt those buckets separately with their own CMK at the S3 / ADLS level.

4. CMK for Cluster EBS Volumes

Cluster nodes have local EBS volumes for shuffle spill, caching, and the operating system. The EBS CMK wraps the per-volume DEK that the EC2 hypervisor uses for transparent disk encryption. Configured once per workspace; applies to every cluster launched in that workspace.

databricks account customer-managed-keys create \
  --use-cases STORAGE \
  --aws-key-info '{"key_arn":"arn:aws:kms:us-east-1:123:key/efgh","key_alias":"alias/dbx-ebs"}'

5. Key Rotation

• Automatic rotation — AWS KMS supports yearly automatic key rotation; Databricks transparently picks up the new version (re-wraps DEKs on next access).
• Manual rotation — for compliance frameworks that require quarterly rotation, generate a new KMS key and update the workspace's key configuration via the Account API. Databricks re-wraps existing DEKs in the background.
• DEK rotation — DEKs are rotated whenever the underlying object is rewritten (e.g. a Delta OPTIMIZE rewrites Parquet files with fresh DEKs).

6. Cloud-Provider Specifics

AWS — KMS

• Use a symmetric customer-managed CMK (not a multi-region key for managed services).
• Key policy must grant the Databricks cross-account role kms:Encrypt, kms:Decrypt, kms:GenerateDataKey*, kms:DescribeKey.
• For EBS, additionally grant kms:CreateGrant with kms:GrantIsForAWSResource = true.

Azure — Key Vault / Managed HSM

• Premium tier or Managed HSM required.
• Soft delete and purge protection must be enabled.
• Grant the workspace's managed identity get / wrapKey / unwrapKey permissions on the key.

GCP — Cloud KMS

• Key ring in the same region as the workspace.
• Service account databricks-cmk@<project>.iam.gserviceaccount.com needs roles/cloudkms.cryptoKeyEncrypterDecrypter.

7. Revocation Impact

Disabling or scheduling deletion of the CMK is the nuclear option — and it works. Within seconds:

• New cluster launches fail (EBS DEKs cannot be unwrapped).
• Open notebook saves fail; the editor surfaces a KMS error.
• Running jobs continue with already-cached DEKs but cannot persist new outputs.
• Reads against Delta tables on CMK-encrypted storage start failing as cached file handles expire.

Re-enable the key and the workspace recovers without data loss. Schedule deletion (vs. permanent deletion) only after a hold period — AWS KMS enforces a 7-30 day pending-deletion window; use it.

↑ Back to Top