- Company Name
- GemPool Recruitment
- Job Title
- Senior Data Engineer
- Job Description
-
Job Title: Senior Data Engineer
Role Summary:
Lead the design, development, and maintenance of enterprise-scale data pipelines and analytics solutions. Own end‑to‑end data flows from ingestion to data quality validation, ensuring reliable, performant, and scalable data architecture across cloud platforms.
Expectations:
- Deliver robust, production‑ready ETL/ELT pipelines that support business analytics and reporting.
- Maintain clean, documented, and reproducible code with CI/CD pipelines.
- Collaborate with data scientists, architects, and business stakeholders to translate requirements into technical specifications.
- Ensure data integrity, security, and compliance across all stages of the data lifecycle.
Key Responsibilities
- Build and maintain data pipelines using Airflow, dbt, Prefect, and Python (pandas, pyarrow, pyspark).
- Design and implement data lake/warehouse solutions on AWS (S3, Athena, Snowflake).
- Model data using star, snowflake, and dimensional designs; create detailed metadata and lineage documentation.
- Implement data validation, quality checks, and anomaly detection to guarantee trustworthy data sets.
- Use version control (Git) and CI/CD practices to automate workflow deployments and tests.
- Mentor junior engineers and promote knowledge sharing within the team.
- Evaluate and recommend tooling and architecture changes to improve performance and scalability.
Required Skills
- Snowflake – 3+ years core experience.
- Data Engineering, Data Migration, Database Technologies, Advanced SQL, Python – 5+ years each.
- Strong proficiency in Python (pandas, pyarrow, pyspark).
- Expertise with Airflow, dbt, Prefect.
- Cloud: AWS (S3, Athena, Snowflake).
- Data lake/warehouse architecture (S3 + Athena, Delta Lake).
- Data modelling (star/snowflake, dimensional).
- Data quality, validation, anomaly detection.
- Git, CI/CD for data workflows.
- Excellent communication, problem‑solving, documentation, and collaboration skills.
Required Education & Certifications
- Bachelor’s degree in Computer Science, Data Engineering, Information Systems, or related field.
- Certifications: Snowflake Certified Professional, AWS Certified Data Analytics – Specialty, or equivalent are highly desirable.