Job Specifications
Job Description
We are looking for a highly skilled Azure Data Engineer to support enterprise-scale data management and analytics transformation initiatives. This role focuses on delivering data engineering pipelines, platform integrations, and supporting advanced analytics, governance, and data products across Azure-based environments. The engineer will work closely with data architects, governance leads, and analytics teams to develop secure, scalable, and reusable data infrastructure.
You'll be instrumental in the development of integrated data pipelines, cloud-native services, metadata management, and supporting self-service data products using platforms like Databricks, Microsoft Purview, Collibra, Informatica IDMC, and Power BI.
Key Responsibilities
Develop ETL/ELT pipelines using Azure Data Factory, Databricks, Synapse, etc.
Build data lake/warehouse architectures for structured and unstructured data.
Perform data curation, transformation, validation, and enrichment at scale.
Implement monitoring, logging, and data quality checks within data flows.
Enable self-service analytics by curating certified datasets and exposing them securely.
Deliver reusable, secure, and well-documented APIs for downstream applications.
Collaborate with analytics teams to support real-time and batch analytics needs.
Integrate data pipelines with visualization platforms such as Power BI or Tableau.
Configure Microsoft Purview, Collibra, or Informatica IDMC for data lineage, classification, and policy enforcement.
Enable metadata management, data catalogs, and automated lineage capture.
Support access control mechanisms (RBAC) and sensitive data tagging.
Collaborate with governance and compliance teams to implement regulatory controls.
Provide platform operations support including pipeline performance optimization.
Qualifications
6-10 years of experience in data engineering, with at least 3+ years on Azure.
Proven hands-on experience in Azure Data Factory, Azure Synapse, Azure SQL, Databricks.
Experience with metadata management tools like Microsoft Purview, Collibra, or Informatica IDMC.
Strong knowledge of data governance, data catalogs, and compliance frameworks.
Experience building scalable data lakes, warehouses, and lakehouses.
Familiarity with DevOps practices and CI/CD pipelines using Azure DevOps or GitHub Actions.
Proficiency in Python, SQL, Spark (PySpark or Scala).
Understanding of security practices (data masking, RBAC, PII handling).
Good communication and documentation skills in agile team environments.
Skills: spark (pyspark or scala),sql,problem solving,python,data governance,pyspark,microsoft purview,etl,databricks,semantic layers,adf,data engineering,collibra,azure synapse,informatica,agile,azure databricks,azure devops,informatica idmc,power bi,azure data factory,github actions,ci/cd pipelines,azure sql,data catalogs