Delta Lake

Delta Lake is the third major open table format alongside Apache Hudi and Apache Iceberg. Originated at Databricks in 2019 and donated to the Linux Foundation in 2022, it adds ACID transactions, schema enforcement, time travel, and incremental processing to Parquet-on-object-storage. Delta is the native table format on Databricks and is increasingly first-class on Snowflake, Microsoft Fabric, and AWS.

Key Features:

• ACID Transactions. Atomic commits via the Delta transaction log (_delta_log/) — ordered JSON files (and periodic Parquet checkpoints) that record every operation against the table.
• Schema Enforcement & Evolution. Writes that violate the table schema are rejected by default; evolution is opt-in via mergeSchema or explicit DDL.
• Time Travel. Query any past version (VERSION AS OF) or timestamp (TIMESTAMP AS OF) for audit, debugging, and rollback.
• MERGE / UPSERT / DELETE. Full DML on Parquet via Delta’s rewrite-on-merge logic.
• Optimize & Z-Ordering. Compaction with optional multi-dimensional clustering for skipping data on common predicates.
• Streaming Source & Sink. Tables work as Spark Structured Streaming sources and sinks with exactly-once semantics.
• Open Protocol & Wide Engine Support. Spark, Trino, Presto, Flink, Snowflake (read), Microsoft Fabric, and the Delta Kernel libraries for Rust / Python / Java.

Architecture:

A Delta table is a directory of Parquet data files plus a _delta_log/ folder of transaction-log entries. Each commit appends a numbered JSON file describing the added and removed Parquet files; periodic Parquet checkpoints summarize the log so readers don’t have to replay every entry. Optimistic concurrency control on the log gives serializable writes without a coordinator.

Use Cases:

• The Bronze / Silver / Gold medallion architecture on Databricks.
• Streaming ETL with exactly-once delivery from Kafka into curated Gold tables.
• Slowly-changing dimensions (Type 1, 2) on a lakehouse via MERGE.
• Regulatory audit and rollback workloads that need historical state.

For deeper Databricks-specific patterns — managed vs. external tables, Delta Live Tables, and SCD Type 2 — see the Databricks Delta Lake reference.