PEARC26: what a poster judge learns about running research computing

I spent a week in Minneapolis at PEARC26 with the rest of the Iowa State Research IT and HPC team. PEARC is where the people who actually operate national research computing compare notes, and going as an operator rather than a spectator changed what I paid attention to.

Judging taught me more than attending

I served as a student poster judge and evaluated three submissions on technical merit, reproducibility, and clarity. The gap between a good poster and a great one was not the idea. It was evidence. The strongest work came with a live demo, and watching someone drive a real system under questions tells you far more than a results table does. A couple of the demos ran on HTCondor, which I have read about but not operated, and seeing it work in front of me moved it from a name to a mental model.

Judging is a useful exercise for an operator. You are forced to separate what a system claims from what it demonstrates, which is the same discipline that separates a green dashboard from an actually healthy service.

The threads I kept pulling on

A few themes ran through the sessions and lined up with what I think about at work:

National cyberinfrastructure and federated compute. The interesting reliability problem is not a single cluster, it is the dependencies that span sites. Federation buys capacity and buys you new failure modes at the same time.
Cloud based ML and AI training. The demand is real, and so is the tension between the persistent, high throughput environment training wants and the batch scheduling model most HPC runs on.
Resource efficiency. Utilization is a reliability concern, not just a cost one. Idle capacity and contention are two sides of the same scheduling problem.

Where I want to take this

The idea I left with is a hybrid: run Slurm and Kubernetes on the same nodes so a cluster can serve both batch analytics and persistent, container-native workloads without maintaining two separate fabrics. Slinky, which runs Slurm inside Kubernetes so the two schedulers can share resources, is the piece that makes this plausible. Pair that with a high throughput fabric like OSPool for opportunistic capacity, and an AI researcher gets both the analytic power of a system like Nova and somewhere to burst.

The smaller, more concrete thing I am prototyping is high availability login nodes on k3s. Login nodes are usually a single point of failure that nobody treats like production: everyone lands on the same shared host, one runaway process degrades it for the whole cluster, and if it falls over, sessions die with it. Here is the sketch I am working from.

Isolation per user. On login, each user gets their own pod rather than a shared shell. A compromised or misbehaving session is contained instead of sitting next to everyone else's.
Fairness with cgroups. Each pod gets a burstable slice of CPU, a floor it can always use and a ceiling it can borrow toward when the node is idle. One person compiling aggressively can no longer starve the node for everyone else, which is the failure I see most often on a shared login host.
Session migration. When a node starts to degrade, the goal is to move a session off it before it fails rather than after. This is the hard part and the part I am least sure of, since a live shell has real state, but even draining new logins off an unhealthy node ahead of a hard failure is a win.
A real lifecycle for editor sessions. Browser based VS Code sessions get a defined lifecycle: provisioned on connect, resource capped like everything else, and cleaned up when the user leaves instead of lingering and leaking resources.

None of this replaces what Open OnDemand already does well. The delta I care about is treating the login tier like production: isolation, fairness, and the ability to lose a node without losing everyone's session. It is early, and I will write it up properly once it runs.

What changed

The through line from all of it is that research computing is quietly becoming a platform engineering problem. The researchers do not want to think about schedulers or nodes, they want a reliable place to do their work, and the job of the people operating these systems is to make the hard parts invisible. That is the part I want to keep building toward.