Senior Big Data Engineer – Utrecht, Netherlands 6 Months +
We are looking for a software developer to be responsible for delivering data to reporting, operational systems for our finance client. You will be in charge of developing complete and scalable solutions in Spark on a Cloudera platform using Azure DevOps pipelines.
The Data and Content within the organization has the responsibility to help teams with data for all IT Systems applications. The goal is to deliver data in a secure, compliant and automated way as fast as possible meeting Security DevOps principles working together with all data teams.
These teams are responsible for data storage, data processing, data flows and data provisioning. We work in DevOps teams where we work on data modelling, data logistics, data quality, data lake and data warehousing to adjust and maintain the data flows and data services we provide for systems and departments within the company.
For one of our teams working on the data lake we are looking for a senior big data engineer.
You will be responsible for
• Development and maintenance of data pipelines mainly in spark.
• Setting up and maintain CI/CD pipelines in Azure devops.
• Setting up monitoring.
• Development of scalable and performant solutions based on open source technologies (Spark, Sqoop, Hive, Hbase, Oozie)
• Streaming flow development with HDF (hortonworks) NiFi and Spark streaming.
• Advice on and involvement in the realization of the Data Lake .
• Working in a multi-disciplinary scrum team and contributing to the T-shaping in the team.
• Providing input and contributing to experiments.
• Keeping informed on the developments in the market.
The competences/expertise we would like you to have:
• Willingness to learn and being open to new techniques.
• Willingness and capability to teach and share knowledge.
• Good communication skills.
• Strong in collaboration.
• Showing entrepreneurship.
• Knowledge on financial products and business processes.
• Being able to work in a structured manner.
• Able to advise and communicate on new solutions
The knowledge we would like you to have:
• Knowledge on (Big) Data environments and tooling (Hadoop, Cloudera, Hortonworks Microsoft Azure, NiFi tooling)
• Relevant HBO or WO education (Informatica, Informatiekunde of Bedrijfskundige richting)
• 4-6 years of experience in ICT data environment
• Extensive experience in software development in an agile environment
• Cloudera 5.12.1
• Streaming (Kafka)
• Cloud platform (preferably Azure)
• CI/CD pipelines (preferably Azure devops)
• SQL + NoSQL (at least 2 years of experience)
• Scripting languages (Python/Bash)
If you are interested, please get in touch:
Consultant: Matt Martin
Contact number: +44 20 7324 2837