Data Engineer at Planet42

Description:

Mission

The mission of this role is to grow a strong local (RSA) capability in the Data and BI function of the business. The Data Engineer will be responsible for building, optimising, and maintaining efficient data pipelines and architectures that power data-driven decision-making across the company. This role is crucial for ensuring that data from various sources is accessible, reliable, and effectively integrated with analytics platforms to support business operations, insights, and innovation.

The Data Engineer is responsible for ensuring that all Planet42 Heads of Department are equipped with the data and insights needed to make agile business decisions, as the company navigates its crucial transformation.

Outcomes

• Maintenance and optimisation of existing data pipelines and integrations.
• Timely delivery of high-quality data pipelines with minimal downtime.
• Efficient integration and availability of data for analytics and reporting.
• Consistent improvement in data pipeline performance and cost optimization.
• High level of collaboration and communication with Heads of Department.
• Production of business critical BI and Finance reports and insights.
• Implementation of robust data quality and governance frameworks.

Responsibilities

• **Data Pipeline Maintenance: **Maintenance and optimisation of existing ETL (Extract, Transform, Load) processes that handle data from various sources and formats.
• **Data Pipeline Development: **Design, build, and maintain scalable ETL (Extract, Transform, Load) processes to handle data from various sources and formats.
• Data Integration: Integrate structured and unstructured data from diverse internal and external sources.
• Data Architecture: Design, implement, and optimise data architecture (e.g., data lakes, warehouses, and marts) that enable analytics and reporting.
• **Collaboration with Data Team: ** Work closely with the Data Scientist, and other engineers to ensure seamless data flows and high-quality datasets.
• **Collaboration with Heads of Department: ** Work closely with various Heads of Department across the business to define, and refine data pipeline requirements.
• BI Report Production: Leveraging new or existing scripts and templates to generate BI and Finance reports and insights for various business functions and Heads of Department.
• **Performance Optimisation: **Identify bottlenecks in data processes and implement solutions to ensure high performance, scalability, and reliability.
• **Data Quality & Governance: **Implement best practices for data governance, quality assurance, and compliance with data protection regulations.
• **Automation: **Automate repetitive data tasks, including data validation, transformation, and loading processes.

Competencies

• **Problem-Solving: **Strong analytical skills to solve complex data engineering challenges.
• Communication: Ability to clearly communicate technical concepts to both technical and non-technical stakeholders.
• Collaboration: Strong team player who can work cross-functionally with different teams to deliver data solutions.
• **Adaptability: **Comfort in a fast-paced environment with changing priorities.
• Proactivity: Ability to anticipate challenges, take initiative, and independently identify opportunities for improvement without waiting for direction.

Requirements:

• Maintenance and optimisation of existing data pipelines and integrations.
• Timely delivery of high-quality data pipelines with minimal downtime.
• Efficient integration and availability of data for analytics and reporting.
• Consistent improvement in data pipeline performance and cost optimization.
• High level of collaboration and communication with Heads of Department.
• Production of business critical BI and Finance reports and insights.
• Implementation of robust data quality and governance frameworks.

• **Data Pipeline Maintenance: **Maintenance and optimisation of existing ETL (Extract, Transform, Load) processes that handle data from various sources and formats.
• **Data Pipeline Development: **Design, build, and maintain scalable ETL (Extract, Transform, Load) processes to handle data from various sources and formats.
• Data Integration: Integrate structured and unstructured data from diverse internal and external sources.
• Data Architecture: Design, implement, and optimise data architecture (e.g., data lakes, warehouses, and marts) that enable analytics and reporting.
• **Collaboration with Data Team: ** Work closely with the Data Scientist, and other engineers to ensure seamless data flows and high-quality datasets.
• **Collaboration with Heads of Department: ** Work closely with various Heads of Department across the business to define, and refine data pipeline requirements.
• BI Report Production: Leveraging new or existing scripts and templates to generate BI and Finance reports and insights for various business functions and Heads of Department.
• **Performance Optimisation: **Identify bottlenecks in data processes and implement solutions to ensure high performance, scalability, and reliability.
• **Data Quality & Governance: **Implement best practices for data governance, quality assurance, and compliance with data protection regulations.
• **Automation: **Automate repetitive data tasks, including data validation, transformation, and loading processes.

• **Problem-Solving: **Strong analytical skills to solve complex data engineering challenges.
• Communication: Ability to clearly communicate technical concepts to both technical and non-technical stakeholders.
• Collaboration: Strong team player who can work cross-functionally with different teams to deliver data solutions.
• **Adaptability: **Comfort in a fast-paced environment with changing priorities.
• Proactivity: Ability to anticipate challenges, take initiative, and independently identify opportunities for improvement without waiting for direction.

• **Programming Languages: **Proficiency in Python, Java, or Scala for data manipulation and automation.
• SQL Expertise: Advanced SQL skills for querying large datasets and building efficient data models.
• **Data Pipeline Tools: **Experience with tools like Apache Airflow, Luigi, or similar orchestration frameworks, or custom edge function / serverless orchestration methods such as AWS Lambda.
• Cloud Platforms: Experience with AWS, Azure, or Google Cloud for managing cloud-based data architectures.
• Big Data Tools: Familiarity with Snowflake, Spark, Hadoop, or similar technologies for processing large-scale datasets.
• ETL Tools: Experience with ETL tools such as Talend, Informatica, or cloud-native options.
• Database Technologies: Knowledge of relational (e.g., PostgreSQL, MySQL) and non-relational databases (e.g., MongoDB, Cassandra).
• BI Reporting: Familiarity with Tableau, PowerBI, Qlik, or similar technologies for processing data sets and producing reports and insights.
• **Data Warehousing: ** Experience with Redshift, BigQuery, or Snowflake for building scalable data warehouses.
• Code Management: Experience with version control and CI/CD platforms such as BitBucket, GitHub, or GitLab.

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
• 3+ years of experience in a data engineering role or similar.
• Strong experience in cloud-based data solutions.
• Experience with big data technologies is a plus.
• Experience with Bitbucket, AWS, Snowflake, and Tableau preferred.

Skills & Experience

• **Programming Languages: **Proficiency in Python, Java, or Scala for data manipulation and automation.
• SQL Expertise: Advanced SQL skills for querying large datasets and building efficient data models.
• **Data Pipeline Tools: **Experience with tools like Apache Airflow, Luigi, or similar orchestration frameworks, or custom edge function / serverless orchestration methods such as AWS Lambda.
• Cloud Platforms: Experience with AWS, Azure, or Google Cloud for managing cloud-based data architectures.
• Big Data Tools: Familiarity with Snowflake, Spark, Hadoop, or similar technologies for processing large-scale datasets.
• ETL Tools: Experience with ETL tools such as Talend, Informatica, or cloud-native options.
• Database Technologies: Knowledge of relational (e.g., PostgreSQL, MySQL) and non-relational databases (e.g., MongoDB, Cassandra).
• BI Reporting: Familiarity with Tableau, PowerBI, Qlik, or similar technologies for processing data sets and producing reports and insights.
• **Data Warehousing: ** Experience with Redshift, BigQuery, or Snowflake for building scalable data warehouses.
• Code Management: Experience with version control and CI/CD platforms such as BitBucket, GitHub, or GitLab.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
• 3+ years of experience in a data engineering role or similar.
• Strong experience in cloud-based data solutions.
• Experience with big data technologies is a plus.
• Experience with Bitbucket, AWS, Snowflake, and Tableau preferred.