The client is undertaking a large-scale Data and Artificial Intelligence (AI) transformation initiative focused on developing and enabling the Ontario Data Platform (ODP). ODP is an enterprise technology ecosystem designed to support data ingestion, storage, processing, visualization, governance, data quality, and data marketplace capabilities. The selected candidate will support AI/ML initiatives within ODP, focusing on running and monitoring machine learning inference workloads, supporting Large Language Model (LLM) pipelines, evaluating model performance, and preparing datasets for experimentation. The role requires practical experience with Python, machine learning libraries, LLM frameworks, and cloud-based ML environments.
Key Responsibilities
- Execute, monitor, and troubleshoot batch inference jobs in Databricks and other ML environments.
- Support DSPy pipelines, including custom model and configuration loading.
- Conduct LLM evaluation runs and track model performance metrics.
- Prepare, clean, and validate legal text datasets for inference and evaluation.
- Annotate model outputs for error analysis and benchmarking.
- Maintain reproducible Python scripts, notebooks, and experiment documentation.
- Identify, document, and escalate model anomalies.
Must-Haves
- Background in Computer Science, Data Science, AI/ML, or a related field (students and recent graduates welcome).
- Hands-on experience with Python and libraries such as pandas, scikit-learn, or PyTorch.
- Exposure to LLM frameworks such as DSPy, LangChain, or Hugging Face.
- Familiarity with Databricks, MLflow, or equivalent cloud ML environments.
- Strong analytical, troubleshooting, organizational, and documentation skills.
Nice-to-Haves
- Hands-on experience building or executing DSPy-based LLM pipelines.
- Experience with batch inference, LLM evaluation, and experiment tracking.
- Knowledge of prompt optimization, model benchmarking, and error analysis.
- Experience processing unstructured text or legal datasets.
- Familiarity with ML workflow automation and reproducible experimentation.
