Skills

Communication Python C# Unity SQL Data Visualization GitHub DevOps Azure Data Factory Azure DevOps Azure Functions Problem-solving Architecture Windows Databases Azure Analytics Spark Databricks

Job Specifications

Job Title: ADF Lead Developer

Contract Duration: 6-month contract with multiple extension opportunities

Work Model: Hybrid (in-office once a week)

Job Description:

The QI Dashboard Project aims to enhance the existing data infrastructure by transitioning from a tactical setup to a more strategic framework. This transformation will facilitate improved data accessibility, processing, and visualization for business intelligence purposes.

Objectives

Data Source Transformation: Migrate the current data source from a tactical infrastructure, which involves direct pulls from virtual machines (VMs), to a strategic infrastructure utilizing Azure Data Factory (ADF) and Databricks.

ETL Process Implementation: Develop an Extract, Transform, Load (ETL) process to efficiently transfer data from various sources, including SFTP, web services, and databases, into a Delta Lake. This will enable seamless data consumption for business analytics.

Connectivity Assistance: Support the business in establishing a robust connection between Plotly Dash and Databricks, ensuring that data visualizations are powered by the latest data insights.

Databricks Setup and Management: Establish the Unity Catalog, configure clusters, define schemas, and set up permissions within Databricks to ensure secure and organized data management.

Expected Outcomes

The successful execution of this project will result in a more efficient data infrastructure, improved data accessibility for business users, and enhanced capabilities for data visualization and analysis through the QI Dashboard.

Must Have Requirement:

* 7+ Years experience with ADF, Databricks and Azure Functions (C#)

* In depth development experience python/notebook on Databricks/Spark/panda

* Experience in loading datasets from various sources using ADF to Databricks with Medallion architecture

* Experience on converting Databricks workspace from hive metastore to unity catalog & setup associated permissions

* Creating delta table schema/tables with proper design/properties & security permissions

* DevOps & deployment pipeline using Azure DevOps/GitHub Enterprise (especially on databricks on cluster/secret management and code)

* In depth experience on Azure Entra ID for SSO/RBAC setup

* In depth experience on working with/deploying to Azure PaaS resources such as Azure SQL PaaS, ADF/SHIR, ADLS, Databricks, Synapse dedicated SQL databases

Nice to Have:

* Nice to have experience on Delta Live Table.

* Familiar to work in lock down windows environment for highly regulated industry

* Financial Services Industry experience is an Asset.

Soft Skills:

* Ability to work independently and lead a small team.

* Proven ability to work independently with minimum supervision

* Ability to lead a small team and technical problem-solving sessions

* Excellent Communication skills

About the Company

Bay Street Staffing is a Leader in the boutique staffing sector--offering fast, agile, exceptional customer service. We have partnered with some of Canada's largest organizations to help them deliver the talent they are looking for. Building solid relationships built on trust and integrity is our signature at Bay Street Staffing. Bay Street offers expert, specialized recruiting practices that result in sought-after matches between candidates and employers. Today's challenging business climate demands that companies hi... Know more