Data Engineer

- Sector: Public sector / Government

- Clearance required: SC clearance

- Contract length: 6 months

- Rate: 400-500/d

- IR35 status: Outside

- Onsite requirement: 2.d a week

- Role title: Data engineer

Job Description:

· Build streaming data ingestion pipelines using AWS MSK and batch data ingestion pipelines using APIs

· Prepare data for prescriptive & predictive modelling.

· Experience working with various data formats e.g., parquet, avro, json

· Experience in building ETL pipelines using Lambda, Step Functions and Glue

· Experience working with various AWS database services (e.g., RDS, DynamoDB and Redshift)

· Technical expertise with various data modelling techniques

· Hands-on experience with Python and SQL

· Kafka Streaming data experience to build the new data pipelines.

Supporting Skills

· 5 years' experience in an iterative and Agile development environment

· Appreciation of Software Development Environments & Languages

· Appreciation of Business Analysis Activities

Appreciation of other scripting language, eg JavaScript, Lua, Perl, PHP, Unix