Data Engineer - Azure Cloud

Marlin Selection
19 Jan 2023
16 Feb 2023
Employer Sector
Technology, ICT & Telecoms
Contract Type
Full Time
An exciting opportunity with a leading financial services platform. This is an exciting opportunity for a Data Engineer to join a team of other highly skilled data professionals responsible for managing and enhancing the data estate. You will be part of a team responsible for data as a whole, including data pipelines moving data from various on premise and cloud based source systems into the Azure based data platform for BI and analytics. The Azure data platform consists of a number of Azure services including Data Factory, Data Lake Storage, Synapse Analytics, and Power BI. This role also offers the opportunity to work with other database/data platforms like MongoDB and Splunk. Responsibilities: • Designing and implementing data pipelines using Azure Data Factory, Spark, SQL feeding data from source systems through to the data lake, data warehouse and Power BI • Monitoring and supporting the existing data pipelines • Design and evolution of the Data Lake • Data Warehouse Schema modelling (Kimball/Star) • ELT Data transformation into the Data Warehouse Schema • Proficient in database performance analysis and tuning, particularly with Synapse SQL Pools. • Developing 'best practice' standards within the data team. • Continual learning about latest technologies in the data space. • Maintaining a consistently high level of service delivery for incidents, requests, and changes. • Maintain security standards across the data platform. Skills & Experienced Required: Essential • Good communication skills with technology staff and end users. • Strong technical knowledge of the Azure Data Platform including Data Factory, Data Lake, Synapse Analytics • Strong knowledge of Azure Synapse Analytics SQL Pools and how it differs to SQL Server for performance reasons • Strong SQL skills (SQL, T-SQL, PL/SQL) • Scripting skills (Python, PowerShell) • Data Lake Design - optimal layout for performance, security etc • Experience in managing and migrating large volumes of data. Compression, partitioning and ETL • Experience of other data cloud technologies AWS, Azure, Oracle Cloud. Desirable • Spark (Databricks/Synapse Spark Pools) • MongoDB • Splunk • Experience of Oracle Cloud (OCI) • Delta Lake • Experience of working in an Agile/SCRUM environment • A desire to learn and understand more about the Business environment in which you work. • Technical knowledge of SQL Server to 2019 and Oracle to 19c