Senior Data Engineer - Digital And Data Solutions

NB: Police Vacancy - only for candidates with a minimum of 5 years continuous UK address history. Please note that this vacancy is subject to a strict security vetting procedure.

The Senior Data Engineer will be accountable for and have lead responsibility for designing, building, and maintaining the data infrastructure, ensuring the efficient and reliable definition, processing, storage, and analysis of large datasets. The role is integral to the Force being able to create value from all the data it collects by leading the Data Engineering activities required to improve and combine data from first contact with the public through to crime intelligence and the Back Office.

They will make a significant contribution to making sure our data, And Data shared by others is of the highest quality, available, usable and shareable by rigorous standard setting and quality control. The role requires a highly comprehensive level of knowledge, functional understanding of modern data technologies and platforms and as sole accountable Data Engineer for the Force will need to excel in the application Data Engineering tools and methodologies. They will cultivate strong and effective working relationships with individuals, teams and colleagues, both internal and external to the Force. In particular they will work performance, information And Data analysts specialising in crime, justice, vulnerable populations, public safety across the Force, from other Forces and public sector partners to build products and services which deliver insight and value to the Force and its ability to make Lincolnshire a safe place to live, work and visit

Key responsibilities:

• Lead the design and implement of scalable, efficient, and robust data pipelines to extract, transform, and load (ETL) data from various sources into data storage and platform solutions.
• Engage and collaborate with cross-functional teams, to understand their data requirements and translate them into technical solutions.
• Build and link data sets so that data scientists and analysts can easily derive insights, including accessing sensitive data held in Back End systems, extracting and loading into the data storage environment (warehouse or lake) or platform (fabric or mesh, whichever strategically appropriate) from which they be provisioned.
• Write ETL scripts and code to make sure the ETL process performs optimally, leading on extracting data, joining and helping analysts to derive insights.
• Lead the build and be accountable for the optimising the performance of pipelines and APIs that connect operational systems, data for analytics and Business Intelligence (BI) systems, troubleshooting and resolving issues as they arise, to ensure the timely delivery of high-quality data.
• Provide leadership to the business in automating its manual data flows and reporting processes, enabling their scaling up and repeatable use, working with leaders across the organisation to identify and elicit the benefits of speeding up and easing access to data.
• Lead the development and maintain data models, schemas, and structures that optimize data storage and retrieval for analytics and reporting purposes.
• Work with the Principal DBA and Principal Solutions Developer to actively optimise code within the Force's systems to ensure processes perform optimally
• Identify ways to improve data reliability, efficiency and quality
• Engage and work closely with data stakeholders to understand their data needs and provide support and guidance on how to use the Data Engineering infrastructure effectively

Qualifications:

• Degree in Computer Science, Information Systems, or a related field. or a combination of education and relevant experience
• Programming land querying languages eg SQL, R, Hive, Python, Java, Scala
• Datamodelling, ETL data pipeline tools, eg Apache Spark, Apache Kafka

Knowledge:

• Detailed and applied knowledge of modern Data Engineering tools, technologies and practices Understanding of machine learning techniques and algorithms
• Good working knowledge with data governance principles, data privacy regulations (eg, GDPR, CCPA), And Data security best practices.
• Detailed and applied understanding of relational and NoSQL databases, including database design, query optimization, and performance tuning.

Experience:

• Experience demonstrating proficiency as a Data Engineer or similar role, delivering data products and services such as Datamodelling, data schemas, warehouses and/or lakes
• Experience demonstrating proficiency working with multiple, complex data sets
• Experience demonstrating proficiency and able to give examples of successful, hands-on implementation of ETL processes and delivery of data pipelines into various data management architecture.

Skills & Abilities:

• Demonstrating detailed skills in cleaning, integrating and scaling data sets and pipelines, fixing problems in data sets, from low performance to bugs and outages
• Proficiency in programming languages such as Python, Java, or Scala, used for Scripting and automation
• Proficiency in using query languages such as SQL, Hive, R, with any ability to work with SQL and NoSQL databases
• Proficient in modelling data in a range of ways, including reverse engineering from live production systems
• Proficient in building, writing, and executing tests to ensure data pipelines and products meet the needs of the Force