Medallion Architecture

Storing Data in the Bronze Layer

Medallion Architecture Best Practices

• Maintain the data in its original format (e.g., JSON, Parquet, Excel) to ensure traceability and auditability.
• Storing data as-is ensures data completeness and prevents potential data loss from premature transformations.
• Provides flexibility for data adjustments and future transformations without the need for re-ingestion.

Benefits of Keeping Original Format:

• Traceability and Auditability: Original data offers a clear audit trail for data integrity checks.
• Flexibility: Allows for modifications to data processing logic without re-ingesting data.
• Data Completeness: Preserves all variations and details from source systems.

Cost Savings

• Reduced Storage Costs: Storing data incrementally (Bronze, Silver, Gold) minimizes storage costs by reducing the need to retain multiple versions or raw data copies. Only essential transformations are stored in the Gold layer, conserving storage space.
• Optimized Resource Usage: Data processing tasks can be tailored to each layer, allowing for efficient resource allocation. For instance, resource-intensive tasks can be confined to specific layers, reducing cloud costs for computing and storage.
• Lower Maintenance Costs: By standardizing data processing and quality checks across layers, maintenance overhead is reduced. This decreases the need for frequent data reconciliations and error correction, cutting down operational expenses.

Overview of Layers

• Bronze Layer (Raw Data): Stores unprocessed data from various sources in its original state, preserving data fidelity. This serves as a single source of truth, allowing for reprocessing and auditing when needed.
• Silver Layer (Cleansed Data): Data from the Bronze layer is cleaned, validated, and transformed. This intermediate layer improves data quality and ensures that only reliable data moves downstream, reducing the need for extensive processing in later stages.
• Gold Layer (Curated Data): Contains refined, aggregated data tailored to specific business needs, such as analytics and reporting. This layer is aligned with business logic and optimized for high-performance queries and dashboards.

Key Benefits

• Data Quality: Incremental processing through each layer enhances data reliability, reducing the need for costly data cleansing operations later in the workflow and enabling more accurate analytics and machine learning applications.
• Scalability: The structured approach efficiently handles large data volumes. As the data grows, the architecture scales without requiring significant infrastructure investment, lowering overall data management costs.
• Flexibility: Different teams can work on separate layers simultaneously, enabling parallel processing and faster data pipelines. This reduces development time, lowering operational costs associated with waiting on dependencies and rework.
• Enhanced Data Governance: By having distinct layers, the architecture supports robust data governance practices, including access controls, data lineage tracking, and compliance measures. This approach reduces costs associated with regulatory compliance and minimizes the risk of expensive data breaches.

In summary, the Medallion Architecture not only enhances data quality, scalability, and governance but also results in considerable cost savings by optimizing data storage, processing, and resource allocation across the entire data pipeline.

Example Code for Data Ingestion in Python

# Ingesting raw data into the Bronze layer
import pyspark
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("BronzeLayerIngestion").getOrCreate()

# Read data from different formats
json_df    = spark.read.json("/path/to/source/data.json")
parquet_df = spark.read.parquet("/path/to/source/data.parquet")
excel_df   = spark.read.format("com.crealytics.spark.excel") \\
    .option("useHeader", "true") \\
    .load("/path/to/source/data.xlsx")

# Write data to Bronze layer
json_df.write.format("delta").mode("append").save("/path/to/bronze/json")
parquet_df.write.format("delta").mode("append").save("/path/to/bronze/parquet")
excel_df.write.format("delta").mode("append").save("/path/to/bronze/excel")
    

SQL Query for Verifying Bronze Data

-- Verify data ingestion in the Bronze layer
SELECT COUNT(*) AS record_count, source_format
FROM bronze_layer_table
GROUP BY source_format;

Expected Output

| record_count | source_format |
|--------------|---------------|
|    5000      |   JSON        |
|   12000      |   Parquet     |
|     800      |   Excel       |

© Kevin Luzbetak