- Company Name
- First Point Group
- Job Title
- Software Data Engineer / Databricks
- Job Description
-
**Job Title:** Software Data Engineer – Databricks (Azure)
**Role Summary:**
Design, build, and maintain scalable data pipelines and modern data platforms on Azure using Databricks. Enable real‑time analytics and machine learning by integrating diverse data sources, ensuring data quality, and optimizing performance and cost.
**Expectations:**
- 3+ years of hands‑on experience with Databricks on Azure, Apache Spark (PySpark/Spark SQL) and Delta Lake.
- Strong proficiency in Python, SQL and Unix/Linux scripting; Java/Scala a plus.
- Experience with Azure data services (ADF, ADLS, Synapse, Azure ML) and streaming technologies (Kafka, Event Hubs, Stream Analytics).
- Ability to implement CI/CD, workflow orchestration, and monitoring solutions.
**Key Responsibilities:**
- Design and optimize ETL/ELT pipelines and lakehouse architectures on Databricks.
- Build and manage data lakes, warehouses, and lakehouses for batch and streaming workloads.
- Integrate large‑scale datasets from multiple sources using Delta Lake and related formats.
- Develop feature stores and data preparation workflows for machine‑learning models.
- Implement data validation, governance, and quality frameworks.
- Monitor, troubleshoot, and tune pipelines for performance, reliability, and cost efficiency.
- Collaborate with dev‑ops to establish CI/CD pipelines and automated orchestration (Azure DevOps, GitHub Actions, Airflow, Databricks Workflows).
**Required Skills:**
- Databricks on Azure, Apache Spark (PySpark/Spark SQL)
- Delta Lake, Azure Data Factory, Azure Data Lake Storage, Synapse Analytics, Azure ML Studio
- Python, SQL, Unix/Linux scripting (Java/Scala optional)
- Streaming: Apache Kafka, Azure Event Hubs, Azure Stream Analytics
- CI/CD: Azure DevOps, GitHub Actions; Orchestration: Airflow, ADF, Databricks Workflows
- Data quality & governance practices
**Required Education & Certifications:**
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related discipline.
- Relevant Azure certifications (e.g., Azure Data Engineer Associate) are a plus.