MLOps North: Building (with) Agents

A gathering for people building agents and the practitioners building with them.
About this Event

MLOps North Toronto Building with Agents

Toronto Summit · Presented by TMLS · Nov 5–6, 2026 · RBC WaterPark Place, TorontoThe people building agents, and the people building with them - Multiple tracks, on the Toronto waterfront. Two days of practitioner-curated talks, no fluff, just real actionable insights. Underlying principles, methodologies and best practices in an industry that feels like it's on a rocket ship.

Why come

This isn't another "Intro to RAG" track. These are the applications and products teams have built. Every session is peer-reviewed by working engineers, not a vendor keynote in disguise.
Across two days our tracks will center on:

Building Agents — the stack: frameworks, memory, evals, orchestration

• AI Sovereignty: When to run your own stack, and when not to
• Cost Management & ROI: Profiling, model tiering and proving the spend
• Coding Agents & Dev Workflows: Complex codebases, secure AI-assisted development and the code review bottleneck
• Agent Deployment & Observability: Taking agents from prototype to production, and knowing when they improvise

Building with Agents — what you ship on it: products, workflows and agents in regulated industries

• Evals & Benchmarks: Building evals that predict production performance, and knowing when to swap models
• Agent Memory Architectures: What agents should store, retrieve, revise and forget
• Recursive Self-Improvement: Feedback loops that improve agents without overfitting
• Big Ideas, Innovations & Creative Explorations: Bold, outside-the-box approaches to the field

What's inside

• 2 days, in person
• 8 tracks
• 35+ speakers
• Talks, hands-on sessions, and hallway conversations with the people running this in production

Who it's for

Software engineers, ML and data engineers, solution architects, infra leads, and the technical leaders (Director, VP, C-suite, founder) making the calls on how AI gets built and shipped.

Details

Nov 5–6, 2026 · RBC WaterPark Place, Toronto waterfront.

NOV 5

🕑: 10:55 AM - 11:40 AM
Agentic ML - Leveraging Harnesses to Automate Extraction Models
Host: Aryan Dear, Senior ML Engineer

Info: Model, Data and Objective Drift have traditionally ML Engineering problems that have had a ceiling where we could automate. This talk is a case study of how Wisedocs automated large portions of model development across a 1500+ document types that face standard MLOps challenges. The talk will have three parts, a focus on domain knowledge, the ROI of agentic ML development and sample harnesses and skills we use to enable these capabilities.

🕑: 10:55 AM - 11:40 AM
From Answers to Evidence: Building Auditable Agents
Host: Akash Shetty, CTO, Publicus

Info: Publicus processes procurement data spread across RFPs, contracts, amendments, award notices, invoices, task authorizations, webpages, and other records. Our early AI workflows optimized for producing a useful answer: retrieve relevant context, reason over it, generate structured output, and evaluate the result. That worked until we tried to increase automation in workflows where an answer can be semantically correct while its evidence is incomplete, stale, contradictory, or simply the wrong source.

We redesigned the system around an evidence ledger rather than the generated answer. Extracted facts and agent claims became typed artifacts carrying source document, location, provenance, transformation history, confidence, and conflicts. Retrieval became evidence acquisition rather than context stuffing. Verification became a separate stage that reconciles claims across records. Unsupported or conflicting outputs cross an explicit human-review boundary instead of being silently conver

🕑: 11:45 AM - 12:30 PM
Loop Engineering is just K8s-style Cybernetics
Host: Sayantan Das, Senior Applied AI Scientist, Manlike

Info: This session reframes "Loop Engineering" as applied cybernetics — the same observe→diff→act reconciliation pattern that powers Kubernetes controllers, now applied to AI agent loops. Attendees will learn how to map K8s design principles (declarative goals, idempotent actions, crash-only design, admission webhooks as HITL gates) directly onto agent infrastructure, and walk away with a concrete checklist for building production agent loops that self-heal, externalize state, and fail gracefully. We'll cover real failure modes — thrashing, stale reads, infinite reconciliation, conflicting controllers — and the fixes that transfer from a decade of K8s production learnings.

🕑: 11:45 AM - 12:30 PM
Automating Prompt Ops in a LLM World
Host: Afseen Syeda, Lead Prompt Engineer, Wisedocs

Info: Managing 10,000+ prompts across dozens of use cases is a challenging task when new LLMs get released weekly. This talk covers the infrastructure, automation and LLM based prompt that allows Wisedocs to automatically update prompts with new product releases, classify error categories and empower prompt engineers to focus on emerging, challenging problems, rather than maintanance.

🕑: 01:30 PM - 02:15 PM
From Explainable Evidence to Intelligence that Continually Learns
Host: Prashanth Rao, Founding AI Engineer & Researcher

Info: What if AI systems could learn continuously without repeatedly retraining the model underneath them? At HDC Labs, we are exploring hyperdimensional computing as a representation layer for online, associative learning. Neural models provide powerful learned representations and general capabilities; hyperdimensional representations provide a lightweight structure in which new observations, relationships, corrections, and outcomes can be incorporated as they arrive. Because learning can happen through simple composable operations over these representations, useful behavior can emerge from relatively few examples rather than large retraining datasets.

That same compositional structure also creates an opportunity for greater explainability. Instead of treating every update as an opaque change to model weights, hyperdimensional representations can preserve the primitives and associations that contributed to a result, making retrieval and reasoning easier to inspect. In this talk, we will

🕑: 11:45 AM - 12:30 PM
A Code-First BI Architecture Powered by AI
Host: Saeid Abolfazli, Global Head of Data Platform and AI

Info: When your BI team has 1000+ workbooks and a growing request backlog, the answer isn't more analysts — it's rethinking the architecture. In this talk we share how Rakuten Kobo's Data team replaced a centralized BI bottleneck with a code-first model where dashboards are version-controlled files authored end-to-end by AI. We cover the [long-term] strategy, tooling decision, the AI-guided authoring workflow built on Claude Code and Cloud, the real operating boundaries discovered through production-scale data. Attendees will leave with a concrete architecture pattern, an honest account of where the approach has limits, and a staged AI coding model they can adapt to any BI environment.

🕑: 01:30 PM - 02:15 PM
From Sovereignty to Strategic Independence: How does a country assemble global
Host: Adam Hendin, Co-founder and CEO Radium

Info: """No country can own every layer of the AI stack. Models, chips, infrastructure, capital, talent and customers already cross borders, making technological purity an unrealistic measure of sovereignty. Drawing on Radium’s experience as a Canadian-headquartered but globally constituted AI company (an “mutt AI company”) this talk proposes a more practical objective: strategic independence, defined as the ability to choose, adapt, operate and replace critical technologies while retaining domestic expertise, infrastructure and economic value.

Rather than asking whether every component was invented in Canada, the talk asks which capabilities Canada must originate, which it can source internationally, and what it must be able to control, operate and govern at home. It also confronts a central Canadian contradiction: while Canada talks about building sovereign AI, its capital, procurement and scaling environment often pushes its most ambitious entrepreneurs to build their companies in the

🕑: 01:30 PM - 02:15 PM
Why Your Multi-Agent Pipeline Is Slow and Expensive (And How to Fix It Systema
Host: Nadia Rauch, AVP, AI Engineering | Principal AI Engineer

Info: We built a multi-agent pipeline to automate a complex, document-intensive enterprise workflow. The initial system ran sequentially through a chain of specialized agent roles, each handling a distinct task in the process, using a single frontier model throughout. It worked. It was also slow and expensive, and we didn't know why.


Rather than optimize blindly, we profiled first. The results were not where we expected: 67% of total latency came from a small minority of the agent roles, and the primary bottleneck was not the model — it was sequential chaining: independent work was being processed one step at a time instead of concurrently. Identifying and parallelizing the roles with no inter-dependency reduced end-to-end runtime by roughly 40%.*


The second experiment compared model tiers (frontier, mid-tier, lightweight) across each agent role independently, measuring accuracy, latency, and cost per role. The finding cuts against the default assumption: the roles that appea

🕑: 02:20 PM - 02:50 PM
Beyond Prompt Engineering: Engineering AI Skills for Continuous Validation and
Host: Kiana Haddadi, Data Scientist BMO Blue Rewards

Info: "Most GenAI talks measure success by speed. We think that misses the bigger opportunity and hides a risk. An AI that accepts its inputs at face value will confidently repeat upstream mistakes. An AI that one practitioner uses well rarely makes the next practitioner any better.
At BMO, we built a reusable AI skill that turns scattered model-development material into final model documentation, starting with models that support the BMO Blue Rewards loyalty program. It did make documentation faster and more consistent. It also did something we didn't design for: when it pieced the whole model story back together, it surfaced gaps and inconsistencies that earlier development and review had missed. We also saw the same skill produce very different documentation depending on who used it and how hard they questioned it. Both patterns match published research: LLM-generated model cards are more complete than human-written ones, LLMs rarely catch errors reliably on their own, and AI assistants s

🕑: 02:55 PM - 03:25 PM
Spec-Driven PBT: Deriving Semantic Invariants from Code Specifications for Age
Host: Aditya Shukla, AI Engineer Georgian Partners

Info: Software is increasingly written by AI agents that autonomously produce an implementation from a user’s natural-language description (the specification), a practice known as agentic coding. Code is now produced faster than it can be reviewed, and code written this way has been shown to exhibit its own characteristic failure patterns. Property-based testing (PBT), a long-standing technique in which invariant properties of a program are checked against many generated inputs, has recently been revisited with LLM agents authoring the properties, thereby removing the high-effort derivation of invariants that has long hindered its adoption. Existing agentic PBT methods, however, derive these properties from the implementation alone (Code-driven PBT). Agentic coding presents an opportunity here: a thorough specification, whether a detailed prompt or a full product requirements document (PRD), has become central to the practice, yet its role in testing the resulting code remains underexplored.

🕑: 02:55 PM - 03:25 PM
Beyond the Benchmark: Adversarial Evaluation of AI Agents in Production
Host: Erfan Pirmorad, Principal AI Scientist Vanguard

Info: As AI systems evolve from single-turn models into agents that retrieve information, use tools, maintain memory, and execute multi-step tasks, traditional benchmark-based evaluation is no longer sufficient. This talk explores how the evaluation workflow must be redesigned around the full agent lifecycle: from threat modeling and adversarial scenario generation to trajectory-level evaluation, production observability, and regression testing.

Attendees will learn how to identify the expanding attack surface of agentic systems, design meaningful adversarial metrics beyond simple pass/fail benchmarks, and structure an evaluation architecture that connects pre-production testing with signals observed in production. The goal is to provide a practical framework for evaluating not just whether an agent performs well, but how it behaves when its environment, inputs, or intermediate state actively push it toward failure.

🕑: 03:55 PM - 04:25 PM
Decision Models and their place in the agentic ecosystem
Host: Suhas Pai, CTO & Co-founder, Hudson Labs

Info: Enterprise AI work has shifted from "generate text" to "make a decision": routing, scoring, classification, triage. New models built to skip autoregression entirely (a category sometimes called "System 1" AI) promise order-of-magnitude speed and cost gains for exactly these tasks, without giving up transformer-grade reasoning. For teams running AI in real-time or high-volume pipelines, this is a shift worth understanding.

🕑: 02:20 PM - 02:50 PM
It Worked on My Laptop — Productionizing Real-Time LangGraph Agents
Host: Prakriti Ailavadi, Senior Manager, Data Science, RBC

Info: " ""The talk uses a compound AutoML system built with LangGraph as a case study for what changes when an AI workflow moves from a working prototype to a production system.

The AutoML agent was redesigned from a long-lived, in-memory graph into a durable checkpoint-and-resume architecture. Model-building steps such as data profiling, feature engineering, model training, and hyperparameter search became independently checkpointed and idempotent. This allows workflows to recover from failures without restarting from scratch.

Context management was also redesigned. Instead of allowing tool-call histories to grow indefinitely, context is actively managed using bounded windows, pruning, and map-reduce summarization. Retry budgets, recursion limits, and cycle controls were added to tool-calling edges to manage the non-deterministic behavior of agentic workflows. Observability also shifted from traditional pipeline monitoring to graph and node-level tracing, testing, and evaluation with Lang

🕑: 02:20 PM - 02:50 PM
Bet on Primitives, Not Frameworks: Architecting AI for a Moving Target
Host: Yahya Mowiena, Senior Director of Engineering, RBC

Info: TBD

🕑: 03:55 PM - 04:25 PM
From Supervision to Autonomy: Building Agents You Can Trust
Host: Millin Gabani, Co-founder and CEO Worklayer

Info: How do you move from supervising every agent action to trusting a system that works in the background? This talk uses feedback control systems as a lens for designing agent architectures that can take on ongoing work with increasing autonomy. We'll explore the core primitives: an environment with tools and memory, clear objectives, observable outcomes, verification, and a controller that determines when the agent should act, pause, or ask for help. Examples such as optimizing ad performance, improving e-commerce margins, and reducing application latency will show how different jobs call for different feedback cycles and architectural choices. We'll examine how budgets, tool permissions, and limits on work awaiting review shape autonomy, alongside natural ways for people to guide and steer agents through feedback. A live demonstration will bring these concepts together, starting with an environment and an agent and developing a repeatable system for doing a job, reviewing its results, a

🕑: 04:30 PM - 05:00 PM
Lairkit: Sovereign Software Forge and Deploys Simplified
Host: Alex Tucker, Software Engineer, 49ops

Info: Right now, teams wanting to deploy ML tools, agentic workflows or web apps on their own hardware usually face a wall of Kubernetes complexity. Lairkit solves this by providing automation to establish a sovereign, GitOps-lite deployment platform on any hardware (homelab, VPS, cloud instances) using existing tools rather than inventing a new API.

Because it uses established CLIs rather than new abstractions, it is incredibly Agent-friendly and even OSS models are already highly proficient at interacting with this stack in practice.

Behind the scenes, Lairkit uses Python Cookiecutter templates and interactive scripts to wire up securely:

Forgejo & Action Runners for sovereign Git and CI (No more GitHub required)

Infisical for secrets management

Tailscale for SSH management and private networking

Kamal for agentless, K8s-free multi-server deploys

🕑: 04:30 PM - 05:00 PM
The Boring Stuff Still Breaks Your Agents
Host: Jen Barnes-Nolan, Founder and Senior AI Advisor JenAI Co

Info: "If something can go wrong in production, it eventually will.”

Many teams are vibe-coding agents with tools powerful enough to build something impressive, and just enough to be dangerous. Sooner or later, those agents end up in production, or handed to another team to run.

Models can fail spectacularly, and teams spend most of their energy there: evals, guardrails, prompt tuning. But the boring stuff can bring your solution down just as hard. A downstream API times out, a dependency goes down, a queue backs up, or a tool returns half an answer. A traditional application throws an error. An agent may carry on and make something up, confidently, inside a real business process.

This talk applies decades of enterprise architecture lessons to agentic systems: timeouts, retries, fallbacks, idempotency, volume and failover, and the observability that tells you an agent improvised instead of failing. Drawn from large-scale integration and production AI delivery in banking, government an

🕑: 03:55 PM - 04:25 PM
Beyond the Benchmark: Evaluating AI Agents for Longitudinal Care Journeys
Host: Aiden Arnold, VP, AI Models & Head of League Labs League

Info: As healthcare embraces the patient-to-agent paradigm, care companions need to be evaluated and aligned with the contexts in which they operate, from longitudinal care pathways to care coordination. This talk explores what those evaluations should cover, how they can adapt and grow in complexity over time, and how simulated patients and clinical scenarios can help us assess and improve agent behaviour across different care pathways.

🕑: 04:30 PM - 05:00 PM
The Path to High MFU - Delivering Inference Performance For Agentic Workflows
Host: Navya Mehta, Principal Engineer and Tech Lead - Compiler

Info: How do we think about optimizing the hardware for inference when it comes to agentic workloads? what is the tradeoff between interactivity and throughput and how does that affect hardware design decisions? I aim to build up an example from optimizing kernels (aka compute), up to optimizing memory management, to optimizing the topology at the data center level, to designing specific hardware itself for different tradeoffs across the interactivity-throughput paretto frontier.

NOV 6

🕑: 10:55 AM - 11:40 AM
How to Not Be Wrong About AI
Host: Greg Wilson, Consultant, Third Bit

Info: Most organizations don't know how to assess the productivity of their software developers, which means that many claims about the impact of AI on productivity are vacuous. This talk analyzes some common mistakes, and describes a few things companies can do to figure out what is and isn't actually cost-effective.

🕑: 10:55 AM - 11:40 AM
TBD
Host: Mohammad Danesh, Head of Data and AI, Tangerine

Info: TBD

🕑: 11:45 AM - 12:30 PM
How to Measure Success for your AI Agents
Host: Manav Shah, Founding ML Engineer, raindrop.ai

Info: To turn the art of building AI agents into a science, you need a way to measure progress — and measuring an agent turns out to be far harder, and far more important, than measuring a model. This talk is about how to actually evaluate agents: what to measure, how the field gets it wrong, and why good measurement is what eventually lets agents improve themselves.

What we'll explore:

- What you're really testing when you evaluate an agent — the model vs. the harness around it
- The building blocks of an eval: traces, datasets, and scoring — and the dimensions most people miss
- Where today's benchmarks and LLM-as-judge break down, with real examples
- A practical playbook for measuring agent quality in development and in production
- How measuring success well opens the door to self-improving, self-healing agents

🕑: 11:45 AM - 12:30 PM
(PANEL) AI Sovereignty
Host: Diederik van Liere, Chief Technology Officer, Wealthsim
🕑: 11:45 AM - 12:30 PM
Responsible AI Beyond Accuracy: Fairness, Safety, and Sustainability
Host: Shaina Raza, Applied ML Scientist Responsible AI

Info: TBD

🕑: 01:30 PM - 02:00 PM
When Self-Improving Agents Learn the Wrong Lessons
Host: Abhimanyu Anand, Sr. Data Scientist, Elastic

Info: Most teams building self improving agents follow a similar trajectory: the agent runs, reflects, updates an artifact, and the metric rises. In production, the metric can still rise while a later incident traces back to a learned change that passed review but was evaluated against the wrong signals.

This talk examines different self learning systems that may change the agent context, the agent harness, or model weights. Each offers different gains and failure modes. Memory can absorb sycophancy and inflate its reward signal. Skill libraries can overfit to the last task or be tainted by one bad run. Harness changes can discard useful behaviour as new tools arrive. Weight updates persist longest and are the hardest to reverse.

I’ll cover the controls that make self iteration viable in production: bounded, reversible updates; provenance for every learned artifact; promotion gates; and evaluations that separate improvement from noise.

🕑: 01:30 PM - 02:00 PM
Grading the Agent: How We Built Evals for Disco's MCP Server
Host: Wendy Foster, Principal Data Scientist, Disco

Info: Disco's MCP server lets learning operators and administrators manage and generate insight about their programs through AI agents instead of dashboards, which means those agents need to be trustworthy, not just capable. This talk walks through how we built our eval process for MCP tools, from defining what 'correct' looks like for operator-facing actions to structuring a dataset that catches failures before operators do.

🕑: 02:05 PM - 02:35 PM
Recursive Self-Improvement
Host: Shashank Shekhar, Research Engineer, Google DeepMind

Info: Will provider later

🕑: 02:05 PM - 02:35 PM
Beyond Public Benchmarks: Building Enterprise AI Evaluations That Actually Pre
Host: Erin Li, Head of AI Research, CIBC

Info: Most AI benchmarks are optimized for public datasets and generic tasks, but enterprise teams operate in a very different reality: domain-specific workflows, regulated data, multimodal inputs, and business-critical quality thresholds. This talk presents the CIBC Benchmark, an enterprise evaluation framework designed to test AI models and workflows against real enterprise tasks, enterprise-relevant datasets, and standardized evaluation metrics.