Brussels / 3 & 4 February 2024

schedule

PyPartMC: engineering Python-to-Fortran bindings in C++, for use in Julia and Matlab


The talk will outline the technical lessons learned in development of the PyPartMC project. The key goal of PyPartMC is to provide a maintainable multi-platform pip-installable package enabling interoperability between the Python ecosystem and the internal components of PartMC – an open-source aerosol dynamics simulation package. The wrapped PartMC package has been in active development for almost two decades and is used in computational research ranging from process studies on ice formation in clouds to large-scale simulations of air pollution evolution coupled with weather forecasting models. PartMC has a modular, test-covered, pseudo-OOP, Fortran 90 codebase; several C and Fortran dependencies (SUNDIALS, MPI, netCDF, SuiteSparse, ...); and a bespoke config-file subsystem supporting nested files for variable-size input.

Developing PyPartMC, we aim to provide added value to both the existing community of PartMC users, accustomed to the Fortran/HPC ecosystem, as well as to a wider community potentially daunted by the entry threshold pertinent to the multiple steps needed to obtain, compile, setup and run PartMC, and analyse its output. Steps which so far involved usage of diverse tools and languages, and which are meant to be achievable in a single computational research environment such as Python, Julia or Matlab.

To this end, PyPartMC is available at pypi.org in form of both source (with all 10+ dependencies in git submodules) as well as binary packages (Linux, Windows and macOS). Binary packages have all the dependencies statically linked, what is orchestrated using CMake via standard wheel-building logic. The design features: (i) employment of unmodified Fortran code of the wrapped PartMC package; (ii) binding logic implemented in C++ using pybind11; (iii) garbage collection of Fortran derived type instances; (iv) a drop-in i/o layer replacement matching the Fortran config-file subsystem API, which C++ sees as nlohmann::json objects, and which quacks like ordinary dicts from Python perspective. On top of that, a thorough test coverage including Github-Actions-hooked checks for interoperability with Matlab's built-in Python bridge and with Julia (using PyCall.jl).

For more information, see: D'Aquino et al. 2023

Speakers

Photo of Sylwester Arabas Sylwester Arabas

Attachments

Links