Job Specifications
Job Description: Senior Data Engineer (Azure | Databricks)
Experience Required: 6–8 years
Skills: Python, PySpark, Databricks, Azure Data Factory, Data Concepts & Modeling, Data Architecture
Role Summary
The Senior Data Engineer will design, build, and optimize enterprise‑grade data ingestion pipelines and analytics frameworks using Azure Data Factory, Azure Databricks, Delta Lake, and associated Azure services. This role involves developing scalable data solutions, establishing engineering best practices, conducting POCs, and supporting SDLC processes across complex data ecosystems. The engineer will collaborate with architecture, security, DevOps, and QA teams while maintaining strong documentation and governance alignment.
Key Responsibilities
1. Data Ingestion & Integration
Design high‑level and detailed ingestion architectures using Azure Data Factory and Databricks.
Determine optimal ingestion patterns (batch, streaming, event‑based) based on business and technical requirements.
Build scalable pipelines with validation, staging, logging, and metadata tracking.
Integrate with Azure services including Event Hub, ADLS, Key Vault, Synapse, and enterprise systems.
Apply best practices around schema evolution, error handling, retry logic, and parameterization.
Document end‑to‑end data flows and ingestion logic.
2. Proof of Concept (POC) Development
Conduct POCs to evaluate Databricks infrastructure performance, scalability, and cost efficiency.
Build Bronze, Silver, and Gold layers using PySpark, Delta Lake, Python, and SQL.
Assess technology trade-offs (ADF vs. other ETL tools, batch vs. streaming) to recommend target architectures.
Produce architecture diagrams, design documents, and implementation summaries.
3. Enterprise Data Engineering Best Practices
Ensure solutions comply with Databricks AI Security Framework (DASF).
Create reusable templates, frameworks, and coding standards for consistent enterprise delivery.
Produce whitepapers, sample repositories, and internal documentation.
Lead KT sessions, technical deep dives, and solution walkthroughs.
Maintain best practices for data quality, lineage, naming standards, and validation rules.
Collaborate with architecture, compliance, and security teams to ensure alignment with governance requirements.
4. Technology Research & Continuous Improvement
Stay current with the latest data engineering, warehousing, and data modeling trends.
Attend industry events and debrief teams on insights and emerging patterns.
Experiment with new tools and techniques via R&D and POCs.
Partner with Azure infrastructure teams to validate enterprise readiness of new capabilities.
5. Data Modeling
Design data models aligned to Bronze/Silver/Gold patterns.
Create SQL scripts for tables, views, and dimensional models.
Document models, lineage, relationships, and data flows for transparency.
6. SDLC & Operational Support
Partner with DevOps/MLOps teams to implement automated deployments and CI/CD workflows.
Support QA/QC teams by providing test datasets, reconciliations, and root‑cause analysis.
Troubleshoot production issues and optimize pipeline performance.
Maintain runbooks, deployment guides, and troubleshooting documentation.
Participate in release management and continuous improvement initiatives.
7. Core ADF & Databricks Development
Design and implement ADF pipelines for ingestion, transformation, and orchestration.
Develop modular, reusable Databricks notebooks using PySpark, SQL, and Delta Lake.
Implement reusable components, parameters, and patterns for consistent engineering delivery.
Essential Skills
7+ years of hands‑on Data Engineering experience.
Strong expertise with Azure Data Factory, Azure Databricks, PySpark, Delta Lake, and SQL.
Proven experience building end‑to‑end, scalable data ingestion pipelines.
Expertise in Spark performance optimization.
Strong knowledge of Azure ecosystem (Event Hub, ADLS, Blob, Key Vault).
Experience with CI/CD, Git, and DevOps practices.
Excellent communication, documentation, and cross‑team collaboration skills.
Preferred Qualifications
Experience with Databricks AI Security Framework (DASF).
Experience with Azure Monitor, Log Analytics, or data lineage tools.
Background in financial services or other regulated industries.
Hands‑on experience with POCs and architecture evaluations.
Azure Data Engineering or Databricks certifications.