Description:
Key Responsibilities:Data Engineering & Cloud Solutions
- Architect, build, and optimize scalable and cloud-agnostic data solutions using Azure, Databricks, Spark, Lakehouse, and Delta Lake tables.
- Develop, implement, and maintain big data pipelines for ingesting, processing, and storing large volumes of structured and unstructured data.
- Manage and optimize data lake and data warehouse architectures for performance, cost, and scalability.
Cloud & DevOps
- Work within Azure environments (Azure Synapse, Data Factory, ADLS, etc.) to develop and maintain cloud-based data solutions.
- Implement best DevOps practices for CI/CD pipelines, infrastructure-as-code, and automation.
- Utilize Azure DevOps and Git for managing code repositories, version control, and continuous integration/deployment.
- Ensure high levels of security, compliance, and data governance across data engineering processes.
Big Data Processing & Development
- Utilize Spark, Databricks, and distributed computing to process and analyse large datasets efficiently.
- Write advanced Python and T-SQL scripts for data transformations, ETL/ELT processes, and real-time data processing.
- Optimize performance for data pipelines and SQL queries for efficiency and cost-effectiveness.
- Experience with Graph databases is a big advantage.
Collaboration & Leadership
- Work closely with data scientists, analysts, and business stakeholders to understand data requirements and develop solutions that meet business objectives.
- Lead initiatives to enhance data engineering capabilities, introduce new technologies, and drive best practices.
- Mentor junior engineers, conduct code reviews, and contribute to building a culture of technical excellence.
- Communicate effectively with technical and non-technical stakeholders, translating complex data concepts into actionable insights.
Required Qualifications:
- 6+ years of extensive experience.
- Education / Degrees: Computer science degree or comparable data engineering certification in one of the cloud platforms.
Minimum required skills and experience:
- Extensive experience in cloud-based data engineering, with expertise in Azure (Azure Synapse, Azure Data Factory, ADLS, etc.).
- Strong expertise in cloud-agnostic data tools such as Databricks, Spark, Delta Lake, and Lakehouse architectures.
- Advanced proficiency in T-SQL and Python for developing complex data pipelines, transformations, and optimizations.
- Hands-on experience in big data processing, ETL/ELT, and data pipeline orchestration.
- Solid understanding of data modelling, warehouse design, and data lakehouse architecture.
- Experience in setting up, managing, and processing with Azure DevOps (or similar DevOps tools).
- Strong knowledge of DevOps, CI/CD, Git, and infrastructure-as-code for automated deployments.
15 Apr 2025;
from:
gumtree.co.za