WikiStream: Real-Time Wikipedia Edit Analytics Platform

WikiStream is a real-time Wikipedia edit analytics platform built on AWS to capture, process, validate, and analyze live events from Wikimedia EventStreams. A containerized Python producer running on Amazon ECS Fargate consumes the Server-Sent Events feed and publishes selected events to Amazon MSK Kafka. The platform focuses on high-activity Wikipedia editions and supports analysis of content activity, contributor behavior, bot activity, regional patterns, and potentially suspicious editing behavior. The data pipeline follows a Bronze–Silver–Gold medallion architecture using Apache Iceberg tables in Amazon S3 Tables. EMR Serverless with Spark Structured Streaming ingests Kafka events into the Bronze layer in three-minute micro-batches, with checkpointing and idempotent MERGE operations designed to provide exactly-once processing. Bronze records retain source, Kafka, event, revision, user, and content-size details. Subsequent Spark batch jobs create a cleaned and enriched Silver layer, then Gold-level hourly statistics, user risk scores, and daily analytics summaries. Data quality is enforced as a sequence of gates before downstream transformations proceed. Bronze checks cover completeness, timeliness, and validity; Silver checks include accuracy, consistency, uniqueness, and drift detection; Gold checks validate upstream results and aggregation consistency. Results are retained in dedicated Iceberg audit tables. AWS Step Functions orchestrates the continuous batch cycle, while CloudWatch, SNS, a Lambda recovery function, and local Grafana support operational monitoring, alerting, and Bronze-job recovery. Amazon Quick Suite serves and visualizes Gold and Silver data through Terraform-provisioned Athena datasets for activity, regional, contributor, risk, and platform-health dashboards. Terraform provisions the AWS infrastructure, and CI validates Python, Terraform, Docker, security, and secret-scanning checks. The project demonstrates a cost-conscious, observable, and production-inspired streaming data-engineering design.