I participated in the 4th Visualize This! challenge, a Canada-wide competition that “aims to celebrate the innovative ways visualization can help researchers explore datasets and answer important scientific questions”. It was hosted by Compute Canada regional partners and the Compute Canada Visualization Team (which I joined the following year). My submission, that won second place, explores the flow around an airfoil from a particularly large computational fluid dynamics simulation (performed by Joshua Brinkerhoff’s team at UBC Okanagan), in particular the separation of the laminar boundary layer from the airfoil and its transition to turbulence. My visualization was a demonstration of the VTK-m library (Visualization Toolkit–many cores), an open source C++ library for scientific visualization maintained by Kitware. I chose to experiment with this library because it was specifically designed for the GPU-accelerated HPC environment. This experiment was only partially successful, it turned out the library was not mature enough at the time to achieve my goals.
VTK-m is similar to VTK in many respects but is a separate codebase and is derived from different older projects. The developers describe it as “algorithms for emerging processor architectures”; the idea is for the end user as well as the visualization expert to write hardware-agnostic code, and the library uses optimized algorithms at the backend. Some functionality of VTK-m could be used through ParaView and VisIt.
While VTK-m supports MPI through a class called
PartitionedDataSet, many if not most of its builtin algorithms do not know how to handle this class. The solution was to use yet another library called VTK-h (h for hybrid) that is basically a wrapper around VTK-m that facilitates much of the inter-process communication without much user intervention, and importantly, creates a composite image from the data stored on each rank.
The video visualizes four quantities from the dataset using four different rendering techniques.
Unfortunately, my attempt was plagued with technical issues, and in combination with lack of time, I was not able to run my pipeline with GPUs on multiple nodes (which was the entire point of using these libraries in the first place). Currently VTK-m is used by a small community for seemingly very specific purposes, and it was very hard coming from outside and trying to apply it for my needs. More than that, VTK-h is completely undocumented (or was at the time) so there was a lot of guesswork and trial and error. I wholeheartedly do not recommend these except for development or curiosity purposes.
A relatively major part of this project was writing C++ code to read OpenFOAM data and convert them to the VTK-m native representation in memory. I had to write this code from scratch after delving into the documentation of both OpenFOAM and VTK-m. It is thanks to this control over the data that I was able to downsample the dataset and transfer it to my workstation, and get any visualization at all in the little remaining time.