- Company Name
- PROBAYES
- Job Title
- Lead Data Engineer Databricks (Paris) F/H
- Job Description
-
**Job Title**
Lead Data Engineer – Databricks
**Role Summary**
Lead the design, deployment, and industrialization of data and AI solutions on Azure Databricks. Drive MLOps, CI/CD, governance, and monitoring best practices from project scoping to production. Act as the primary technical liaison with the client, mentor internal and client teams, and standardize Databricks pipelines and models.
**Expectations**
- Transform proof‑of‑concepts into scalable, production‑ready solutions.
- Serve as the main technical point of contact for the client throughout the project lifecycle.
- Establish and enforce data‑engineering and AI best practices, ensuring security, performance, and scalability.
**Key Responsibilities**
- Design and deploy Lakehouse architectures, ELT/ETL pipelines, and streaming solutions on Azure Databricks.
- Implement MLOps workflows with MLflow, Delta Lake, and Unity Catalog; manage jobs, clusters, and automated pipelines.
- Lead CI/CD integration (Git, GitLab CI, or equivalent) for code, data, and model delivery.
- Mentor and coach development teams, enhancing skills in Databricks, Spark, and data engineering.
- Standardize industrialization processes across projects, including data quality, governance, and monitoring.
- Participate in technology scouting and knowledge sharing within the organization.
- Contribute to the recruitment of specialized Databricks talent.
**Required Skills**
- 5+ years of professional experience in data platform engineering and production industrialization.
- Deep expertise in Azure Databricks: Delta Lake, Unity Catalog, MLflow, job and cluster management.
- Proficiency in Python, PySpark, and SQL.
- Strong knowledge of modern data architectures: Lakehouse, ELT/ETL, streaming, APIs.
- Hands‑on experience with CI/CD tools (Git, GitLab CI, or similar).
- Familiarity with Azure cloud services.
- Excellent communication, client‑facing, and team‑collaboration skills.
*Preferred*
- Scala programming experience.
- Experience with IA industrialization best practices (training, serving, monitoring, MLOps).
- Exposure to BI and data catalog tools (Power BI, Tableau).
**Required Education & Certifications**
- Minimum Bachelor’s degree (Bac+5) in Computer Science, Engineering, or equivalent.
- Relevant certifications in Azure, Databricks, or data engineering are advantageous.