Lead Data Engineer - SRM Digital LLC

Skills

Python SQL Encryption Shell PySpark Terraform

Job Specifications

Job Summary:

We are seeking an experienced GCP Data Engineer with deep expertise in BigQuery and other Google Cloud Platform (GCP) services to design, develop, and optimize data pipelines and cloud-based data solutions. This role requires strong cloud engineering, data processing, and automation skills.

Key Responsibilities:

Design and optimize BigQuery data warehouses for performance and cost efficiency.
Develop and maintain scalable ETL pipelines using Cloud Dataflow, Dataproc, Composer (Airflow), and Pub/Sub.
Build data ingestion and transformation workflows using Python, SQL, and PySpark.
Implement secure and compliant data solutions using GCP IAM, encryption, and logging best practices.
Collaborate with data analysts and scientists to support advanced analytics and ML readiness.
Monitor and optimize query performance and BigQuery usage.
Automate infrastructure and CI/CD pipelines using Terraform and Cloud Build.
Manage and maintain GCP services ensuring reliability, scalability, and availability.

Required Skills & Experience:

10+ years of experience in data engineering with at least 5+ years on GCP.
Proven expertise in Google BigQuery, including performance tuning and cost optimization.
Hands-on experience with Cloud Dataflow, Dataproc, Cloud Composer (Airflow), and Pub/Sub.
Strong skills in Python, SQL, Shell scripting, and PySpark.
Solid understanding of GCP IAM, data governance, and security best practices.
Experience with CI/CD and IaC tools like Terraform and Cloud Build.
Strong problem-solving and analytical skills.
Ability to handle large-scale datasets and real-time streaming data.

About the Company

We at SRM Digital are focused towards connecting businesses with top talent across various industries. With a Commitment to excellence, we specialize in providing tailored recruitment services that meet the unique needs of our clients & Candidates. Know more