- Company Name
- Sigmaways Inc
- Job Title
- Senior Big Data Engineer
- Job Description
-
**Job Title:**
Senior Big Data Engineer
**Role Summary:**
Design, build, and maintain scalable, next‑generation data platforms using Spark, Hadoop, Hive, Kafka, and cloud services (AWS, Azure, Snowflake). Develop robust pipelines, optimize big data ecosystems, and collaborate across product, engineering, and data science teams to deliver actionable business insights.
**Expectations:**
- Minimum 7 years of experience designing, developing, and operating Big Data platforms including Data Lakes, Operational Data Marts, and Analytics Data Warehouses.
- Bachelor’s degree in Computer Science, Software Engineering, or a related discipline.
- Proven proficiency in Spark, Hadoop, Hive, Kafka, and distributed data ecosystems.
- Strong background in ETL pipeline development with Hive, Spark, EMR, Glue, Snowflake, Cloudera/MR, NiFi.
- Solid understanding of SQL databases (PostgreSQL, MySQL/MariaDB).
- Deep knowledge of AWS and Azure cloud infrastructure, distributed systems, and reliability engineering.
- Experience with IaC and CI/CD (Terraform, Jenkins, Kubernetes, Docker).
- Good programming skills in Python and shell scripting.
**Key Responsibilities:**
- Design, develop, and support end‑to‑end data applications and platforms focused on Big Data/Hadoop, Python/Spark, and related technologies.
- Collaborate with leadership to conceptualize next‑generation data products and contribute to the overall technical architecture.
- Work closely with product management, business stakeholders, engineers, analysts, and data scientists to engineer solutions that meet business needs.
- Own components from inception through production release, ensuring quality, security, maintainability, and cost‑effectiveness.
- Recommend and enforce software engineering best practices with enterprise-wide impact.
- Lead continuous process improvements, troubleshoot production issues, and mentor peers on best practices.
- Stay current with emerging technologies and rapidly adopt new tools and approaches.
**Required Skills:**
- Expertise in Spark, Hadoop/MR, Hive, Kafka, and distributed data ecosystems.
- Hands‑on experience building ingestion, validation, transformation, and consumption pipelines using Hive, Spark, EMR, Glue ETL/Catalog, Snowflake, Cloudera/Hadoop, NiFi.
- Strong SQL skills and experience with PostgreSQL, MySQL/MariaDB.
- Deep knowledge of AWS and Azure cloud services (compute, storage, networking, IAM, security).
- Proficiency with infrastructure-as-code (Terraform) and CI/CD pipelines (Jenkins).
- Containerization and orchestration skills (Docker, Kubernetes).
- Familiarity with REST APIs, data integration patterns, and microservices.
- Excellent programming skills in Python and shell scripting.
- Understanding of distributed systems, reliability engineering, and production best practices.
**Required Education & Certifications:**
- Bachelor’s degree in Computer Science, Software Engineering, or related field.
- Professional certifications (e.g., AWS Certified Big Data – Specialty, Azure Data Engineer Associate) are a plus but not mandatory.
San francisco bay, United states
Hybrid
Senior
09-11-2025