- Company Name
- SOFTEAM Expertise Data & IA
- Job Title
- DATA Engineer Big Data
- Job Description
-
Job Title: Data Engineer Big Data
Role Summary:
Senior Java Big Data consultant responsible for designing, implementing, and administering large‑scale data platforms. Focus on data ingestion, processing, quality control, and performance optimization using Hadoop, Spark, and Scala. Contribute to architecture definition and support end‑to‑end data governance, visualisation, and AI workflows.
Expectations:
* Minimum 5 years of professional experience as a Java Big Data consultant.
* Strong technical expertise in Java, Hadoop, Spark, and Scala.
* Proven ability to lead data engineering projects from requirements gathering through deployment and maintenance.
* Fluency in English (written and spoken).
Key Responsibilities:
* Analyze business requirements and translate them into technical solutions.
* Participate in architecture design and definition of data platform components.
* Develop and maintain data pipelines (Spark jobs, Java services) for ingestion, transformation, and loading into the Data Lake.
* Ensure data quality through validation, monitoring, and testing (unit and load tests).
* Perform performance tuning and optimisation of Spark workflows.
* Conduct capacity planning, load tests, and performance benchmarking on big‑data clusters.
* Maintain documentation and support knowledge transfer to team members.
Required Skills:
* Java development (core, streams, concurrency).
* Big Data ecosystem: Hadoop, Spark, Scala, and related libraries (e.g., Spark SQL, DataFrames, Datasets).
* Data ingestion tools (Kafka, Flume, Sqoop, or equivalent).
* Cluster administration (YARN, Mesos, or Kubernetes on cloud/on‑premises).
* Data quality verification and testing frameworks.
* Familiarity with data governance concepts and metadata management.
* Strong problem‑solving, debugging, and optimisation capabilities.
* Excellent communication and collaboration skills.
Required Education & Certifications:
* Bachelor’s or Master’s degree in Computer Science, Information Technology, or related field.
* Professional certifications in Big Data or Cloud (e.g., Cloudera Certified Associate, Hortonworks, Microsoft Azure Data Engineer, or equivalent) are desirable.
---