Working with emerging fields in HPC, AI, and ML
The Alces Flight crew worked with the Research Software Engineering (RSE) community to explore what new fields mean to supercomputing at their month-long SeptembRSE conference
A question that always included the RSE Community.
“I’ve been waiting 20 years for this.” — Dr. Andrea Townsend-Nicholson, University College London (UCL) responding to how thanks to more accessible HPC it is now easier to train and teach personalised medicine methodology at the clinical level.
Back in early 2021 the Alces Flight crew was being asked an interesting question: How can more HPC service providers start working with fields that are new to High Performance Computing? Specifically, how can HPC work with fields that were more experimental in their methodology (e.g. humanities, arts, psychology) over the ‘traditionally’ scientific? Our journey took us on a series of interviews that, over the past six months, were published by the Supercomputing (SC’21) conference. But we weren’t quite finished there. Time and time again the role of the RSE in this transition appeared in our discussions. Thanks to the UK’s Society of Research Software Engineering were were able to take the conversation to SeptembRSE, a month-long technology conference celebrating the many roles an RSE inhabits. Once there, we were able to talk to those who work on the front lines of HPC, AI, and ML projects and ask how they approach new ideas and concepts. Our panel uncovered several different approaches. Here is a quick summary of what we learned:
How can a new idea get resources?
“There is no direct path from English Literature to High Performance Computing.” — Dr. Mariann Hardey, Durham University on how experimental fields aren’t at all linear with adding computational applications to projects.
In traditional HPC, as well as AI and ML, the thought process for a resource starts at the hardware level. Simply apply for it (if you are an academic), or write a business case, or seek a type of grant funding. But there is something rather important missing: Getting resources for experimental fields isn’t just being granted access to compute and storage, there is a layer of skills needed that many of these fields simply don’t have because — up until now — they weren’t necessary. The projects that have succeeded with experimental fields won through collaboration. Going it alone can stall or kill projects, so those who realised the importance of the RSE community, as well as vendors, consultants, and the HPC network ultimately succeed. Two projects which sit on the experimental spectrum, Living with Machines and CompBioMed, have benefited from these rich networks and the ability to be practical… yet flexible… to the challenges they face in transitioning ideas from desktop to HPC and beyond.
When viewing resources as more than just hardware or software and including the talent (and budget!) needed you are not only more likely to get access to the resources to transform your idea… but you’re also more likely to get the right resources.
How can experimental and scientific methodology live side-by-side?
“It’s important we listen to each other. You can’t force your methods on them — because they might not be the right ones.” — Matthew West, University of Exeter on working with experimental HPC projects.
You may have heard the phrase, “Two nations divided by a common language.” When bringing experimental fields into HPC, AI, and ML that division can be felt at all levels. Tensions can grow between those who are transitioning the work to the supercomputing level and those who want to be able to showcase their results to their community that, until recently, didn’t talk much of things like large-scale computing, training models, or natural language processing. So how do you deal with it? By listening and keeping an open mind. Successful projects often look for common ground, as the Ordered Universe project learned when they spent the time translating Latin descriptions of a medieval universe into the mathematical equations. What was initially viewed as ‘gibberish’ turned out to have value when Physicist and Classist recognised the patterns that mimicked how today’s universe models are constructed in — resulting in fascinating simulations.
So what’s the takeaway from this? That when bringing together two different ways of thinking you aren’t there to battle for a side, or even balance out both sides. Instead, go in willing to learn the other view and look for common ground that can take a project to its next level.
So how do these projects get going? Even better, can we improve HPC, AI, and ML thanks to these new projects?
“Every project that I’ve seen succeed has put in the effort upfront.” — Dairsie Latimer, Red Oak Consulting, on the practical aspects of launching any large-scale computing project.
We often underestimate — and undervalue — time in any HPC, AI, or ML project. Planning out for the long term is usually not thought about until a project is underway and the real work is laid out for the team to come to grips with. In every interview we have done the successful teams have put the bulk of their time into constructing the right project resources, skills, and budgets — and for good reason. These groups would rather ‘fail smart’ over not understanding the reasons behind why something went wrong. Done in iterative steps and tracking their decisions helped speed them towards optimisation and not disaster, as well as turned up some insights that all fields regardless of methodology need to grow in the HPC, AI, and ML space.
Thanks to the work being done in humanities in computational biomedicine tracks are being made into the better collection of data and improving upon the diversity of resources. For example, groups such as Oceanic Exchanges have formed to follow news from the past and present globally to help us understand why we form the views we do (and perhaps unconsciously bias ourselves to what data we consider important). Early AI and ML models in medicine are pointing out gaps in our data collection and how we are excluding factors that are harder to track: such a cultural views towards medical treatment in general.
Parting thoughts
“We’ve run into barriers, sure. But, in the end, we’ve come to see that they are actually just another set of opportunities.” — Christina Last, Alan Turning Institute, on how the Living with Machines team handles computing obstacles.
New fields wanting into HPC, AI, and ML are going to continue to grow. There is a true desire to take the vast amount of information we have stored and (in some cases) transformed digitally to gain better insight into our world. Rather than attempting to force these fields into a scientific box it is best we approach with an open mind and the willingness to embrace the experimental. While there isn’t one ‘right’ way to go about welcoming these fields in we all can agree the value they bring by simply having a different viewpoint can only enrich HPC, AI, and ML. We’d like to thank all the researchers who have spoken to us, the industry experts who have leant us their advice, and the RSE and HPC communities for embracing this work.
If you would like to watch the SeptembRSE panel simply click on the link below and skip ahead to about the 57:00 minute marker.
