AddressOur client, a leading global IT service provider,is recruiting for a Data Engineer - Streaming to join their business in the UK .
Position Title:
Data Engineer - Streaming
Position Type:
Permanent or Contract
Start Date:
ASAP
Location:
Milton Keynes, UK
Contact:
Davor Molnar | +49 (0) 89 23 88 98 63
Key Responsibilities:
- Participate in requirement gathering, analysis, solution design to build big Data Engineering applications on AWS Data services such as AWS EMR, Snowflake using Spark and AWS Glue as ETL framework.
- Migrate and build Cloudera based hadoop, hive, impala, Kafka, Sqoop, Spark based data, jobs and security policies to AWS EMR , S3 , Snowflake and other AWS services such AWS Glue, DMS, IAM.
- Hands on involvement in Low Level Design, Development and architecting of large Data projects leading developer and testing teams.
- Job Scheduling and Automation.
- Data Validation, Quality checks, profiling and data reconciliation testing.
- Work as an individual contributor as well lead teams when required.
- Mentor junior members in the team by improving their skills and knowledge and have the ability to get things delivered.
- Work with both senior and junior team members like Project Manager, Hadoop Architect, other Data Engineers, data modelers, report developers, testers to complete the task.
- Setting up security and governance policies on data, users, data pipelines on AWS data services.
- Troubleshooting application errors and ensuring that they do not occur again.
Apply agile and CI/CD methodologies and tools for development and deployment in agile mode.
Must have:
- Streaming Data Engineering and Analytics over it. Kafka, AWS Data Services, AWS, ETL, Spark/Scala, Java, Python, EMR, AWS Glue, AWS Athena
Good to have:
Python, Spark, Hive, HDFS, Impala, Sqoop. Informatica
Qualifications:
- Bachelor’s degree and 9+ years of experience in the IT industry.
- 5+ years’ experience as Big Data Engineer on Hadoop, AWS EMR and its ecosystems
- Experience in Banking Domain plus.
- Experience on AWS EMR, AWS Glue, Athena, S3, DMS, SCT and Cloudera CDH based hadoop.
- Should have extensively worked on Big Data projects on AWS EMR based Hadoop, HDFS/S3, Spark, Hive, Impala leading teams and interacting with Architect roles and Client.
- Work experience with SQL, RDBMS, complex queries.
- Understanding and experienced in data warehousing data modeling concepts.
- Has an understanding of large Batch and Stream processing.
- Knowledge of Quality Assurance methodologies, exposure to all facets of Extract, Transform and Load (ETL) processes.
- Proficiency on Linux / Unix command line.
- Experience in applying agile and CI/CD tools and methodologies for deployment and related automations.
- Experience in performance optimization techniques on Hadoop, HDFS, Hive, Spark, File Formats, and providing technical guidance to other application developers.
- Banking domain experience is mandatory
English language fluent is required
For more details contact:
sales( at)manningglobal.com
Quoting reference:
UK_MGDM_DES_
Or call
Davor
on:
Davor
on:
+49 (0) 89 23 88 98 63
View our latest jobs:
www.manningglobal.com
Follow us on twitter:
@ManningGlobal
If you know of anyone within your network who may also be interested in this position,would you please forward this ‘Email & Role’ to them.
Please note:
- That we do all we can to respond to all applications but if we have not contacted you within 2 days following your application - its likely we have not shortlisted your CV for the position.
- Manning Global cannot consider your application for this role if you are working for the customer we are recruiting for
- If you are working for the client we are representing, but have confirmation you are leaving a permanent or contract position, please discuss this with us, so we can discuss representation with you.
- This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the originator of the message. Any views expressed in this message are those of the individual sender.
Group Head Office
Manning Global AG
Hohenzollernstraße 60, 80801 Munich, Germany
Sitz der Manning Global AG: München, HRB 170058 Amtsgericht München, Vorstandsvorsitz: Rachael Manning
Munich/ Dusseldorf/ Bonn (DE) – Dublin (IRE) – Liverpool (UK) – Zurich (CH) – Lille (FR) – Changzhou (CN) – New York/ Houston (USA) – Zagreb (HR) – Timișoara (RO) – Madrid (ES) – Istanbul (TR)
