- Company Name
- N Consulting Global
- Job Title
- AWS Data Engineer
- Job Description
-
Job title: AWS Data Engineer
Role Summary:
Design, develop, and maintain scalable, testable data pipelines on AWS using Python, Apache Spark, and AWS Glue/EMR. Implement lakehouse architecture with Apache Iceberg, integrate data observability and quality checks, and collaborate with business teams to translate requirements into data‑driven solutions.
Expectations:
- Deliver clean, maintainable Python code with type hints, linters, and automated tests (pytest).
- Apply modern software engineering practices: version control, CI/CD, modular design, and automated testing.
- Work effectively in Agile teams, participating in code reviews, pair programming, and architecture discussions.
- Continuously learn and provide insights on financial indices and investment analytics.
Key Responsibilities:
- Design and construct scalable ETL pipelines using Python and Spark.
- Orchestrate data workflows with AWS Glue, EMR Serverless, Lambda, and S3.
- Implement lakehouse data layers and table formats using Apache Iceberg.
- Build and maintain observability for data flows, including basic quality checks and schema evolution.
- Translate business requirements into technical solutions in collaboration with stakeholders.
- Participate in code reviews, pair programming, and architectural discussions.
- Maintain CI/CD pipelines and automated testing suites.
Required Skills:
- Proficient in Python programming (type hints, linters, pytest).
- Strong understanding of data engineering fundamentals: batch processing, ETL pipeline design, schema evolution.
- Experience with or willingness to learn Apache Spark for large‑scale processing.
- Familiarity with the AWS data stack (S3, Glue, Lambda, EMR).
- Experience with version control (Git) and CI/CD tools (GitLab CI, Jenkins, GitHub Actions).
- Knowledge of data quality frameworks such as Great Expectations or Deequ (desirable).
- Agile mindset, collaborative communication, and stakeholder engagement.
Required Education & Certifications:
- Bachelor’s degree (or equivalent) in Computer Science, Information Systems, or related field.
- AWS certifications (e.g., AWS Certified Data Analytics – Specialty, AWS Certified Big Data – Specialty) preferred but not mandatory.