Contract
Job title:
Senior Data Engineer - ML Data Pipelines
Duration:
12-month contract
Role status:
Current opening
We are seeking a highly skilled Senior Data Engineer with extensive experience designing and developing scalable data pipelines and data models to support Machine Learning (ML) use cases. The ideal candidate will bring deep technical expertise in PySpark, Azure Databricks, and Azure cloud technologies, with a strong focus on feature engineering, ML-ready data modelling, and building production-grade data pipelines.
Principal tasks and responsibilities include:
· Design, develop, and maintain scalable data pipelines using PySpark and Azure Databricks to support machine learning and advanced analytics initiatives.
· Build and optimize complex ETL/ELT processes to ingest, transform, and prepare large volumes of structured and semi-structured data for ML applications.
· Design and implement data models specifically optimized for machine learning use cases, including feature engineering, training datasets, and inference pipelines.
· Collaborate closely with Data Scientists and ML Engineers to understand data requirements and translate them into efficient, reusable, and scalable data engineering solutions.
· Develop robust feature engineering pipelines to support feature extraction, transformation, aggregation, and enrichment for ML models.
· Design and maintain historical and time-aware datasets, ensuring data consistency, accuracy, and prevention of data leakage.
· Leverage Azure Databricks, Delta Lake, and Azure Data Lake Storage to build scalable and high-performance data processing solutions.
· Optimize PySpark workloads, including complex transformations, joins, partitioning, and distributed processing performance.
· Implement automated data quality checks, validation processes, monitoring, and error-handling mechanisms to ensure reliable ML-ready datasets.
· Develop and maintain data orchestration workflows, incremental processing pipelines, and production data integrations.
· Apply data engineering best practices related to performance optimization, data governance, scalability, and maintainability.
· Support the operationalization of ML data pipelines, ensuring reliable and consistent data availability for model training and production inference.
Our client:
A complex enterprise organization leveraging advanced analytics and machine learning to drive business outcomes.
Qualifications and pre-requisites:
· 7+ years of hands-on experience in Data Engineering, with demonstrated experience building data pipelines for machine learning and advanced analytics use cases.
· Expert-level proficiency in PySpark, including complex data transformations, distributed processing, optimization, and large-scale data engineering.
· Extensive hands-on experience with Azure Databricks, including Apache Spark, Delta Lake, Databricks Workflows, and lakehouse architecture.
· Strong expertise in data modelling for machine learning, including designing ML-ready datasets, feature engineering, historical data structures, and training/inference data preparation.
· Proven experience designing and implementing end-to-end data pipelines specifically supporting ML model development and production use cases.
· Strong proficiency in Python and SQL, including complex queries, data transformations, and performance tuning.
· Hands-on experience with Azure data services, particularly Azure Data Lake Storage Gen2, Azure Data Factory, and Azure Databricks.
· Strong understanding of feature engineering concepts, including feature extraction, aggregation, transformation, point-in-time accuracy, and prevention of data leakage.
· Experience with Delta Lake, incremental data processing, and modern lakehouse architecture.
· Solid understanding of data modelling principles, including dimensional modelling, relational modelling, and analytical data structures.
· Experience implementing data quality frameworks, automated testing, monitoring, and pipeline orchestration.
· Demonstrated ability to collaborate with Data Scientists and Machine Learning Engineers to support complex ML initiatives.
· Familiarity with Databricks Feature Engineering, Feature Store, MLflow, or Azure Machine Learning is considered an asset.
· Experience with Azure DevOps, CI/CD pipelines, and infrastructure automation is an asset.
· Strong analytical, problem-solving, and communication skills with the ability to independently deliver complex data engineering solutions.
Additional information or perks:
12-month contract opportunity.
Opportunity to work on complex, enterprise-scale machine learning and data engineering initiatives.
Hands-on technical role focused on Azure Databricks, PySpark, and ML data engineering.
Clear Prism does not use artificial intelligence throughout the hiring process.