Skills

Communication Python C# Unity SQL GitHub DevOps Azure Data Factory Azure DevOps Azure Functions Problem-solving Architecture Windows Databases Azure Spark Databricks

Job Specifications

Required Skills & Experience

* 7+ Years experience with ADF, Databricks and Azure Functions (C#)

* In depth development experience python/notebook on Databricks/Spark/panda

* Experience in loading datasets from various sources using ADF to Databricks with Medallion architecture

* Experience on converting Databricks workspace from hive metastore to unity catalog & setup associated permissions

* Creating delta table schema/tables with proper design/properties & security permissions

* Nice to have experience on Delta Live Table.

* DevOps & deployment pipeline using Azure DevOps/GitHub Enterprise (especially on databricks on cluster/secret management and code)

* In depth experience on Unity Catalog

* In depth experience on working with/deploying to Azure PaaS resources such as Azure SQL PaaS, ADF/SHIR, ADLS, Databricks, Synapse dedicated SQL databases

* Familiar to work in lock down windows environment for highly regulated industry

* Ability to work independently and lead a small team.

* Proven ability to work independently with minimum supervision

* Ability to lead a small team and technical problem-solving sessions

* Financial Services Industry experience is an Asset.

* Excellent Communication skills

Job Description

Insight Global is looking for a Lead Data Engineer to join a team at one of Canada's largest banks. The ideal candidate will be part of a multi-year modernization initiative, with the current project being a QI Dashboard Project focused on upgrading the current data infrastructure from a tactical setup to a strategic one using Azure Data Factory (ADF) and Databricks. The goal is to improve data accessibility, processing, and visualization for business intelligence.

About the Company

Insight Global is an international professional services and staffing company specializing in delivering talent and technical solutions to Fortune 1000 companies across the IT, Non-IT, Healthcare, and Engineering industries. Fueled by staffing and talent experts, Evergreen, our professional services brand, brings technical advisors and culture consultants to help customers tackle their biggest challenges. With over 70 locations across North America, Europe, and Asia, and global staffing capabilities in 50+ countries, our tea... Know more