Job Specifications
ROLE OVERVIEW
We are recruiting for a Data Engineer to join a data migration workstream (Dynamics program), which is rolling out D365 F&O to various UK businesses.
The data engineer role will work with the data team, D365 program team to execute the end-to-end data migration process.
This is an integral part of the data team, which consists of other data engineers, a data quality/BI consultant, and data consultants responsible for loading data to Dynamics.
The Program is rolling out D365 to multiple business units in parallel; as such, the Data Engineer may pivot across several concurrent data migrations efforts in different stages of their life cycle.
You will be responsible for implementing the code and executing the process to extract, transform, and load data from source systems (both D365 and others like Access Dimensions) to D365; leveraging and where possible, refining DM processes and tools; reporting on status and blockers to the Data Lead and project managers; collaborating with colleagues in the project team (functional, technical, change/training).
KEY TASKS & RESPONSIBILITIES
Data Migration Process and Tooling
Work with Azure Data Factory (ADF) extraction and orchestration pipelines, making enhancements where required, and integrating outputs with Azure Databricks for downstream transformation
Develop and maintain the data migration pipeline in Azure Databricks using Python/SQL
Work with the Data Lead and D365 program team to iteratively review and refine the data migration process and pipeline for efficiency, auditability, and data accuracy
Work with the Data Reporting Analyst(s) to support Front End (Power BI) use cases supported by the pipeline, including for day-to-day data quality, data migration status reporting, data migration analytics, etc.
Data Discovery, Assessment, and Profiling
Analyze system documentation (eg, system design documents, FDDs) to understand data architecture in the core model; interface directly with systems implementation partner to clarify understanding
Participate in data discovery for new source systems to document source data architecture and design considerations relevant to data mapping and transformation (especially for known non-D365 source systems, eg, AccessDimensions, Concept Evolution)
Work directly with Technical workstream to understand system architecture and integration requirements
Conduct thorough data profiling and quality assessment across source systems
Analyze and provide reporting on data volume and complexity to inform planning
Coordinate with the systems integrator to understand master data needed to support transactional data migration
Collaborate with business SMEs to validate data definitions
Data Extraction
Work with business SMEs, systems integrator, and PMO to document, validate, and support data extraction scope criteria/business rules for master and transactional data
Work with Business Analysis team to confirm master and transactional data requirements and validation approaches; confirm alignment between transactional and master data scope
Implement data extraction scope criteria within ADF and Databricks extraction logic as appropriate
Source Data Quality and Cleansing
Identify data quality issues with extracted data and develop remediation strategies (eg, cleanse in source)
Work with business SMEs to implement data cleansing requirements and thresholds/KPIs
Work directly with business SMEs to conduct iterative cycles of data cleansing, re-reporting on data quality, and further cleansing activities until thresholds/KPIs are met or exceeded
Data Conversion (Mapping and Transformation)
Work directly with business SMEs and systems integrator to:
*Implement and document source-to-target field mappings
*Define validation rules
*Define transformation logic
Work directly with business SMEs to programmatically and where required, assist with populating/refining data
Implement mappings and transformations in Python/SQL code in Databricks
Data Entity & Framework Configuration
Work with the system integrator/D365 platform team to support where needed, the configuration/update of Data Management Framework (DMF) entities within D365
Optimize performance for large dataset processing
Work with the system integrator/D365 platform team to support where needed, the creation of composite/custom data entities for complex migration scenarios as required
Data Migration Dry-Run and Cutover Management
Utilize and refine data migration run-books during dry-runs and go-live cutover
Execute trial migrations and iterative validation cycles, including:
Adhering to and supporting the planning and scheduling of dry-run cycles with the Data Lead and PMO
Coordinating with business users to conduct dry-run data validation activities
Ensuring appropriate documentation for audit purposes
Working with Testing workstream to provide test data
Support systems integrator/business SMEs with data required to support transactional data migration
Execute production data migra
About the Company
At Mentmore we understand that hiring exceptional people is important for our clients but that it is only one part of their job. We aim to make that process as enjoyable, efficient and effective as possible.
We provide high quality candidates for both permanent and contract IT positions and are specialists within our five Core Practice areas:
*Information Security
*Change & Project Management
*Infrastructure & Service Management
*Architecture
*Executive IT Leadership
Our knowledge extends across a variety of sectors inclu...
Know more