← Back to Projects Real-time Processing

Real-Time Streaming Pipeline

Scalable streaming architecture using Kafka, PySpark Structured Streaming, and Delta Lake for e-commerce clickstream processing with exactly-once semantics.

Apache Kafka
PySpark
Delta Lake
Streamlit
Docker Compose
View on GitHub →
Streaming Pipeline - Pipeline Overview
Streaming Pipeline - Throughput Metrics
Streaming Pipeline - Data Quality Scores & Trends
Streaming Pipeline - Business Metrics & Revenue

Overview

Production streaming pipeline for real-time e-commerce clickstream processing and analytics.

Architecture

  • Apache Kafka for event streaming
  • PySpark Structured Streaming for processing
  • Delta Lake for ACID-compliant storage
  • Medallion architecture for data layers
  • Dead letter queue for error handling

Key Features

  • Exactly-once processing semantics
  • Change data capture (CDC) support
  • Dead letter queue for failed events
  • Real-time analytics dashboard