Description:
As a Senior Cloud Data Engineer, you will play a key role in designing, developing, and optimizing cloud-agnostic data solutions, with a strong focus on Azure. Your expertise in Databricks, Spark, Lakehouse architectures, Delta Lake tables, and modern data engineering technologies will be essential in building and maintaining robust data pipelines.
This senior role requires technical excellence, leadership, and the ability to mentor junior engineers while independently tackling complex data engineering challenges.
Key Responsibilities
Architect, build, and optimize scalable cloud-agnostic data solutions using Azure, Databricks, Spark, Lakehouse, and Delta Lake tables. Develop, implement, and maintain big data pipelines for ingesting, processing, and storing structured and unstructured data. Manage and optimize data lake and data warehouse architectures for performance, cost, and scalability. Work within Azure environments (Azure Synapse, Data Factory, ADLS) to develop and maintain cloud-based data solutions. Implement DevOps best practices, CI/CD pipelines, and automation. Ensure security, compliance, and governance in all data processes. Utilize big data tools for processing large datasets efficiently. Write advanced Python and T-SQL scripts for data transformation and ETL/ELT processes. Collaborate with stakeholders to develop data-driven solutions. Mentor junior engineers and lead technical initiatives.Required Qualifications & Skills
Degree in Computer Science or a related field, or equivalent data engineering certification. 8+ years of experience in cloud-based data engineering. Extensive experience with Azure (Azure Synapse, Data Factory, ADLS, etc.). Strong expertise in Databricks, Spark, Delta Lake, and Lakehouse architectures. Advanced proficiency in T-SQL and Python for complex data transformations and optimizations. Hands-on experience with big data processing, ETL/ELT, and data pipeline orchestration. Solid understanding of data modeling, warehouse design, and data Lakehouse architecture. Experience with Docker and Kubernetes for data processing workloads. (Advantageous) Knowledge of data security, encryption, and compliance regulations. (Advantageous) Exposure to machine learning and AI pipelines within data engineering workflows. Experience with multi-cloud environments (AWS, GCP in addition to Azure). (Advantageous) The Reference Number for this position is NG60322
08 Apr 2025;
from:
gumtree.co.za