- Company Name
- Ardent SoftSol
- Job Title
- Senior Data Engineer
- Job Description
-
Job title: Senior Data Engineer
Role Summary: Design, build, and optimize enterprise‑grade data ingestion pipelines and analytics frameworks using Azure Data Factory, Azure Databricks, Delta Lake, and related Azure services. Lead the development of scalable, secure, and maintainable data solutions, conduct proof‑of‑concepts (POCs), enforce engineering best practices, and support SDLC processes across complex data ecosystems.
Expectations: 7–8 years of hands‑on data engineering experience; proven track record delivering end‑to‑end, scalable ingestion and transformation solutions on Azure. Demonstrated ability to design for performance, cost, reliability, and governance. Strong communication and documentation skills to collaborate with architecture, security, DevOps, and QA teams.
Key Responsibilities:
- Design ingestion architectures (batch, streaming, event‑based) with Azure Data Factory and Databricks; build pipelines with validation, staging, logging, and metadata tracking.
- Conduct POCs to evaluate Databricks performance, scalability, and cost; develop Bronze/Silver/Gold data layers and recommend targeted architectures.
- Develop and maintain reusable templates, coding standards, and best‑practice frameworks; create architecture diagrams, whitepapers, and internal documentation.
- Lead technology research, attend industry events, and prototype new tools; collaborate with Azure infrastructure teams to validate enterprise readiness.
- Design data models, generate SQL scripts for tables, views, and dimensional models, and document lineage and relationships.
- Partner with DevOps/MLOps to implement CI/CD, automated deployments, and release management; provide test datasets and root‑cause analysis to QA/Given.
- Troubleshoot production issues, optimize pipeline performance, and maintain runbooks and deployment guides.
Required Skills:
- Advanced proficiency in Python, PySpark, and SQL.
- Deep experience with Azure Data Factory, Azure Databricks, Delta Lake, Azure Event Hub, ADLS, Key Vault, and Synapse.
- Expertise in Spark performance tuning, schema evolution, error handling, retry logic, and parameterization.
- Familiarity with CI/CD tooling (Git, pipelines), DevOps practices, and automated testing.
- Solid documentation, communication, and cross‑team collaboration capabilities.
Required Education & Certifications:
- Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
- Azure Data Engineer Associate (DP-203) or Databricks Academy data engineering certification (preferred).
- Experience in regulated industries (e.g., financial services) is a plus.