Amazon Data Firehose (formerly Kinesis Data Firehose)

Amazon Data Firehose is the serverless delivery service for streaming data. It captures, buffers, optionally transforms, and loads streams into S3, Redshift, OpenSearch, Splunk, Snowflake, HTTP endpoints, and Iceberg tables — with no cluster to manage and automatic scaling.

Firehose vs. Kinesis Data Streams:

• Data Streams is a durable, replayable stream consumers pull from — low latency (~200ms), custom processing, 24-hour to 365-day retention, shard-based scaling.
• Firehose is a push-and-forget delivery pipe — buffers up to ~60s / 5MiB, no replay, fully serverless, minimal code.
• Choose Firehose when you just need streaming data landed into S3/Redshift/OpenSearch; choose Data Streams when multiple consumers or custom processing need the raw feed.

Key Features:

• Managed Delivery to Many Sinks: S3 (parquet/JSON/raw), Redshift via COPY, OpenSearch, Splunk, Datadog, New Relic, Snowflake, Iceberg tables, generic HTTP.
• Format Conversion: On-the-fly JSON → Parquet/ORC with Glue Data Catalog schema.
• Lambda Transforms: Invoke a Lambda function per batch to enrich, mask, or reshape records before delivery.
• Dynamic Partitioning: Route records into partitioned S3 prefixes by record content (e.g., year=2026/month=04/tenant=acme/).
• Buffering Controls: Tune buffer size (1–128 MiB) and interval (60–900 s) to trade latency for file size and cost.
• Error Handling: Failed records delivered to a configurable S3 error prefix for reprocessing.

Common Use Cases:

• Clickstream Ingestion: Land web/mobile events into an S3 data lake in Parquet, partitioned by date.
• Centralized Logging: Fan-out application logs to OpenSearch for search and S3 for cold archive.
• CDC Landing: Deliver DMS or database CDC streams into Iceberg tables.
• SIEM Feeds: Stream security events to Splunk, Datadog, or a custom SIEM.
• Real-Time Analytics Prep: Buffer IoT telemetry into Parquet for Athena queries.