Introduction

Hello everyone! My name is Shubham Kumar, and I am currently pursuing a B.Tech in Computer Science and Engineering. I am delighted to present my final GSOC report/blog for the PyCxxwrap click project under the Ste||er Group.

Background

So, the initial plan was entirely different from what I have accomplished in this project. Initially, my intention was to use pybind11 to create bindings specifically for the HPX library.However, I encountered significant issues related to Python and its Global Interpreter Lock (GIL), which prevented me from proceeding as planned. At that point, my mentor, Steven R. Brandt, suggested that I rework a portion of CxxExplorer which was designed for creating Python bindings for C++ libraries. As a result, this project has transformed into a rework of CxxExplorer, now operating as a standalone library capable of facilitating the creation of Python bindings for various C++ libraries, which can subsequently be converted into Python modules.

Project Description

My project, PyCxxwrap, serves as a Python wrapper for C++ libraries. The primary objective of this project is to offer a Pythonic interface to C++ libraries, making it easy to work with and enabling the creation of Python libraries from existing C++ libraries.

Here’s a simple example of how PyCxxwrap can be used:

from pycxxwrap.pycxx import py11
@py11(headers=["<iostream>"])
def hello():
    """
    std::cout << "Hello World !"<<std::endl;
    """

hello()

The only way to use the HPX runtime in Python is to have HPX installed, along with the run_hpx.cpp .You can use run_hpx.cpp to create a wrapper for the HPX runtime, and then utilize this wrapper to generate Python bindings for the HPX runtime, allowing Python developers to work with HPX seamlessly.

from pycxxwrap.pycxx import py11
from pycxxwrap.types import type_names, create_type
from pycxxwrap.tools import set_args

# set arguments for compilation and location you want to store the library
f = set_args(flags = """-std=c++17 -I/home/matrix/GSoc/ipyhpx 
          -I/home/matrix/pyhpx/release_hpx/hpx/cmake_install/include 
          -L/home/matrix/pyhpx/release_hpx/hpx/cmake_install/lib -lhpx 
          -Wl,-rpath=/home/matrix/pyhpx/release_hpx/hpx/cmake_install/lib""")

# Create your own types
create_type("func","std::function<void()>")

# create wrapper for hpx runtime envirnmnet
@py11(headers=["<run_hpx.cpp>","<iostream>"],args=f)
def hpx_wrapper(f : func)->None:
    """
    const char *num = getenv("HPX_NUM_THREADS");
    int num_threads = num == 0 ? 4 : atoi(num);
    std::cout << "Using " << num_threads << " threads." << std::endl;
    hpx_global::submit_work(num_threads,f);
    """
@py11(headers=["<hpx/hpx.hpp>"],wrap=hpx_wrapper,args=f)
def do_future()->None:
    """
    std::cout << "Hello, World!" << std::endl;
    auto f = hpx::async([](){ return 5; });
    std::cout << "f=" << f.get() << std::endl;
    """
do_future()

For more examples you can visit here or read docs here

Performance comparison

Here is the comparison result of the native Python and HPX code in Python using PyCxxwrap for the 2D Heat Equation example. The example used is of 2D Heat Equation

def calculate(u):
    for k in range(0, max_iter_time-1, 1):
        for i in range(1, plate_length-1, delta_x):
            for j in range(1, plate_length-1, delta_x):
                u[k + 1, i, j] = gamma * (u[k][i+1][j] + u[k][i-1][j] + u[k][i][j+1] + u[k][i][j-1] - 4*u[k][i][j]) + u[k][i][j]
    return u

Performance :2288.0 ms execution time

The code snippet below represents the HPX version of the same code. All the boilerplate has been removed for the purpose of demonstrating the difference in performance between native Python and HPX code in Python

    
    auto start_time = std::chrono::high_resolution_clock::now();
    for (int k = 0; k < max_iter_time - 1; ++k) {
        hpx::for_each(hpx::execution::par_unseq, u[k].begin() + 1, u[k].end() - 1, [&](std::vector<float>& row) {
            int i = &row - &u[k][0];
            for (int j = 1; j < plate_length - 1; j += delta_x) {
                u[k + 1][i][j] = gamma * (u[k][i + 1][j] + u[k][i - 1][j] + u[k][i][j + 1] + u[k][i][j - 1] - 4 * u[k][i][j]) + u[k][i][j];
            }
        });
    }

performance : 54.0 ms execution time

As you can see, there is a significant performance difference, even in this simple example of HPX in Python. The native Python version is 40 times slower than the HPX version. Please note that this is not an accurate one-to-one comparison, but it serves as a meaningful comparison between the two languages. You can see more examples here

Challenges Faced

One of the main challenges I faced was working with two different languages simultaneously. In my project, the process of binding involves writing Pybind11 CXX files, compiling them using Python, and then importing them into Python. Consequently, I had to learn both Python and CXX concurrently, as well as acquire the skills to use Pybind11 for creating bindings for CXX libraries.

Another challenge I encountered was writing bug-free code, as testing can be quite challenging.

Future Work

• Add support for C++ Classes
• Upgrading the run_hpx.cpp to support more features of HPX

This is just a simple way to access C++ libraries in Python, but the ultimate goal is to bind HPX to Python and make it usable in Python. Therefore, I will be waiting for a no-GIL version of Python to be released, and then I will work on binding HPX with Pybind11 to make it usable in Python.