Databricks Consultant
Location: PCS CA~TORONTO
Years of Experience: 7-10 Years
Job Summary: We are seeking a highly skilled Databricks Consultant with strong hands-on experience in Databricks and Apache Spark (PySpark/Scala). The ideal candidate will have a solid background in SQL, data transformation techniques, and cloud platforms (Azure/AWS/GCP), with a particular emphasis on Azure. This role involves building and optimizing ETL pipelines, ensuring data governance, and implementing best practices in data modeling and performance tuning.
Responsibilities
- Design and develop ETL pipelines using Azure Databricks and Delta Lake, including batch processing (autoloader) and Spark structured streaming.
- Implement data transformation techniques and ensure data quality throughout the pipeline.
- Work with data warehousing concepts and develop slowly changing dimensions (SCD1 and SCD2) on dimension tables.
- Build and maintain change data capture (CDC) pipelines.
- Utilize Lakehouse federation to create foreign catalogs for accessing external data sources.
- Optimize Spark jobs through effective data partitioning and liquid clustering.
- Collaborate with cross-functional teams to ensure data governance and security practices are adhered to.
Mandatory Skills
- Strong hands-on experience with Databricks and Apache Spark (PySpark/Scala).
- Proficient in SQL and data transformation techniques.
- Experience with ETL tools and data pipeline development.
- Strong Azure cloud background.
- Understanding of data warehousing concepts.
- Strong problem-solving and analytical skills.
- Hands-on experience with Unity Catalog.
Preferred Skills
- Exposure to CI/CD pipelines and DevOps practices.
- Familiarity with data governance and security practices.
- Experience in building and optimizing data models in Spark.
Qualifications
- Bachelor's degree in Computer Science, Information Technology, or a related field.
- 7-10 years of relevant experience in data engineering or backend development.
