Job Specifications
Lead Data Engineer (Near Real‑Time Ingestion, GCP)
Location: Birmingham (Hybrid – 2 days onsite)
Are you ready to build and lead near real‑time data platforms powering one of the UK’s largest telecom infrastructures?
We're hiring for a major national telco undergoing a high‑profile digital transformation. We're looking for a Lead Data Engineer to architect and drive their NRT ingestion platforms on GCP, working on cutting‑edge, mission‑critical data systems operating at massive scale.
If you love solving distributed systems challenges, designing low‑latency streaming pipelines, and leading high‑performing engineering teams, this is the role for you.
What You’ll Own & Deliver
Architecture & Engineering
Build and evolve near real‑time ingestion frameworks using Pub/Sub, Kafka, Dataflow (Beam), GCS, BigQuery, Cloud SQL.
Architect fast, reliable ingestion pipelines with Apache Beam (Java/Python) on Dataflow.
Deliver robust CDC ingestion using Datastream, handling schema drift and low‑latency data updates.
Apply deep BigQuery optimisation: Storage Write API, partitioning, clustering, materialised views.
Performance & Operations
Optimise streaming workloads for latency, throughput, backpressure, watermarks, and windowing.
Build self‑healing, high‑availability pipelines with enterprise-grade SLAs.
Establish strong observability: Cloud Monitoring, Logging, Trace, DLQs, replay strategies, retry frameworks.
Security, Governance & Leadership
Implement secure architectures across IAM, KMS, VPC-SC, DLP, Secret Manager.
Champion governance, data quality, and auditability.
Lead the NRT roadmap, mentor engineers, and partner with architecture and analytics teams.
Drive engineering excellence, reusable frameworks, and automation.
What You Bring
Deep hands‑on expertise with GCP — Pub/Sub, Dataflow, BigQuery, GCS, Composer.
Strong background in CDC technologies (Datastream preferred).
Advanced Apache Beam engineering skills (Java or Python).
Solid grounding in Terraform, CI/CD, GitOps, and cloud-native orchestration (e.g., GKE).
Proven leadership in designing and scaling complex streaming ingestion systems.
If you're an expert in real‑time data platforms and want to make a real impact — this is a career‑defining opportunity.