Building an AI-Augmented News Intelligence Pipeline with Kafka, Delta Lake, and LLMs

How I built a streaming pipeline that uses LLMs as a transform layer and Delta Lake for stateful content versioning My first portfolio project (Ballistics) was batch - API calls on a schedule, Airflow orchestration, S3 landing zone. My second (Pulse) was streaming - Kafka, exactly-once delivery, session analytics in dbt. Both used the same transformation tool (dbt) with different ingestion patterns. Sentinel is the third project, and the question changed. Ballistics and Pulse processed structured data - JSON from APIs, simulated clickstream events.