Senior Data Engineer

Senior Data Engineer (Azure)

Glasgow

The Senior Data Engineer position combines technical hands-on expertise, effective business stakeholder management, and leadership/mentoring responsibilities for junior team members. You will be instrumental in driving the development and delivery of our client's data platform, which leverages the Microsoft Azure cloud computing service, specifically Databricks with Delta Lake, along with associated Azure storage (ADLS Gen2). The primary programming languages utilized in their Databricks notebooks are Python (PySpark) and SQL. Additionally, they employ other components like Azure Data Factory, GitHub, Azure SQL, Azure Functions, and Azure Purview to varying degrees. As part of their modernization efforts, they also aim to replace several legacy middleware systems with more sophisticated alternatives.

The Senior Data Engineer will have overall technical responsibility for the data architecture. The primary responsibility will be to take the lead role in building out, migrating to and expanding the modernised data architecture.

Leadership, Mentoring & Stakeholder Management

• Understand the business and the key business drivers for successful outcomes.
• Work with all people across the business with varying degrees of seniority and technical knowledge and experience to represent the data architecture in the best way possible.
• Take a senior role within the Data Management Office to facilitate prioritisation and action on data activities across the organisation.
• Provide leadership and mentoring for more junior members of the team with less experience in Data Engineering.

Development of Modernised Data Infrastructure (MS Azure Databricks / Delta Lake and Data Integration)

• Design, develop, test, deploy and maintain end-to-end data pipelines across this data platform to make data available across the business.
• Oversee the use of API integrations between different systems to collect new data sources.
• Manage third party vendors with respect to solutions used in the data platform.
• Improve and maintain the data platform. Including:
  • Expand upon existing CI/CD processes to ensure robust, organised delivery of solutions.
  • Provision technical environments that enable low effort implementation of CICD processes (e.g., synchronised 'dev', 'test' and 'prod' environments).
  • Identify ongoing opportunities and contribute to the long-term Data Engineering strategy.

Management, Maintenance & Migration of Existing Infrastructure

• Oversee and manage existing data pipelines, responding to failures and issues as required.
• Ensure that data pipelines from our source systems via 3rd party middleware (e.g. Fivetran, Jitterbit) are operating in a timely fashion, and liaising with 3rd parties in the event of any issues.
• Have oversight of the commercial relationship with 3rd party middleware providers.
• Help design a migration plan to manoeuvre assets onto the new architecture.

Essential Skills and Experience

• At least 3 years' experience of direct, hands-on experience with Data Engineering using the Microsoft Azure cloud computing service & Databricks.
• At least 3 years' experience of coding in Python or PySpark.
• Active, demonstrable use of software engineering / CI-CD / devops tools and methods.
• Advanced SQL and relational database knowledge.
• Experience in creating data products for consumption by different types of users with varying levels of technical knowledge and understanding.
• Ability to explain Data Engineering concepts and methods in simple terms.

Good-to-have Skills and Experience

• Experience with data and application integration software and APIs
• Experience with other data and cloud computing services such as Amazon Web Services (AWS), Google Cloud Platform (GCP) and Snowflake.
• Experience with data visualisation / business intelligence / dashboarding / reporting tools and technology.

Reward

This client have an enviable reputation of offering candidates a diverse range of enterprise project opportunities, where complexity and collaboration will ensure continual personal and technical development.

In return, the successful Senior Engineer will receive a salary and package commensurate with experience and in the upper quartile for the market + Bonus options, training and exposure to new tech will ensure career development.

Next Steps

Please hit 'Apply Now' to be considered for this or similar positions FPSG have available.

Equal Opportunities

FPSG is committed to equal opportunities regardless of gender, race, disability, sexual orientation, religion or belief and age.

Key skills terminology

Data Engineer, Microsoft Azure cloud, Databricks, Python, PySpark, CI/CD, DevOps, SQL, RDBMS, APIs, Amazon Web Services, AWS, Google Cloud Platform, GCP, Snowflake, data visualisation, business intelligence, dashboarding, reporting tools.

We are Disability Confident and neurodiverse aware. If you have a disability, please tell us if there are any reasonable adjustments we can make to assist you in your application or with your recruitment process