Management Information Lead Data Engineer (Azure)

We are looking for a Lead Data Engineer, who will help us design, implement and manage data solutions on the Microsoft Azure cloud platform. The Lead will work to migrate the business from legacy systems. The job involves planning and management for solution design and technical delivery, ensuring that the right tools and roadmap are available. The Lead manages a team who create and maintain data pipelines, data storage solutions, data processing and data integration for data-driven decision-making inside Halfords, including the organisation’s stakeholders. The Lead will plan, execute and manage data migration from legacy systems using Azure Data Factory and Azure Databricks.

Responsibilities

Current Platform

• To determine the technical Data Engineering roadmap that delivers the Halfords requirements using appropriate technologies and tools.
• To manage, develop and mentor other members of the Data Engineering Team.
• To act as design authority for new developments within the Data Engineering Team.
• To manage the handover of ongoing Data Engineering solutions to support teams and assist in resolution of issues where appropriate.

• Communicate effectively with management and other stakeholders to explain data concepts, solutions and benefits.
• Design, develop and maintain data pipelines using Azure Data Factory.
• Implement data validation and cleansing procedures to ensure the quality, integrity and reliability of the data.
• Optimize pipeline performance and scalability using Databricks workspace configuration, cluster and notebook optimization.
• Work with big data technologies like Azure Databricks and Apache Spark to handle and analyse large volumes of data.
• Support data analytics, machine learning and other data-driven applications using Azure ML, Logic Apps and Power BI.
• Design and implement data storage systems using Azure Blob Storage.
• Ensure data security, privacy and compliance with relevant standards and regulations.
• Stay updated with the latest trends and technologies in the data field and evaluate new tools and frameworks.

Legacy

• Assess the current state of the data and analytics solutions on the legacy systems, such as SAP Business Warehouse, Qlik, Alteryx and Business Objects
• Define the target state of the data and analytics solutions on the Azure cloud platform, using Azure Databricks.
• Design, develop and test data migration strategies and pipelines using Azure Data Factory.
• Ensure the data quality, integrity and consistency during and after the migration process.
• Monitor and troubleshoot the data migration issues and resolve them in a timely manner.
• Document and communicate the data migration progress, status and results to management.
• Coordinate and align with the report re-writing team to ensure the compatibility and integration of the data and reports on the Azure platform.

Requirements

• Degree in Computer Science, Engineering, Mathematics, Statistics or related field or 4+ years of experience as a Data Engineer or similar role.
• Microsoft Certified: Azure Data Engineer Associate certification or equivalent.
• People development, coaching and mentoring skills.
• Stakeholder management skills.
• Network externally to understand emerging trends and technologies.
• Analytical and creative approach to problem solving.
• Proficiency in SQL and data modelling concepts.
• Understanding of legacy architectures and platforms such as SAP BW, Alteryx and BusinessObjects.
• Experience with Azure data services and tools, such as Azure Data Factory, Azure Databricks, Azure Storage Account and Azure SQL DB.
• Experience with data migration projects, especially from SAP BW to Azure Databricks or similar platforms.
• Experience with data visualization and reporting tools like Power BI.
• Experience with data integration and ETL processes.
• Experience with Python, Scala, Java or other programming languages.
• Experience with Azure DevOps or other version control and CI/CD tools.
• Experience with leading and managing a team of Data Engineers.
• Excellent communication, presentation and interpersonal skills.
• Awareness of retail-specific business processes and IT solutions.
• Ability to work independently and collaboratively in a fast-paced environment.
• Ability to learn new technologies and frameworks quickly.
• Knowledge of AI and machine learning concepts and platforms, such as Google Vertex and Microsoft Fabric, is a plus.