Building NGSolve Add-ons

18.1. Building NGSolve Add-ons#

To extend NGSolve’s built-in functionality by creating your own C++ add-on modules, there are multiple ways to proceed depending on how you have installed NGSolve on your system.

18.1.1. A template for C++ add-ons#

Suppose you have a project idea you want to implement. Your wishlist for the project implementation is not modest. You want all these things:

Write a (small, moderate or) large C++ extension module.
Include Python bindings to your C++ extensions.
Be compatible with a recent pip-installed NGSolve (6.2.2404 or newer).
Your add-on supports all those cool hip easy installation methods via pip,
- either as a source distribution (when user has a local build system),
- or as a binary wheel (in sync with user’s specific NGSolve version).
Your add-on is also buildable & installable by traditional make/cmake, so you keep the older folks happy.

Where to begin?

A starting point is offered by the ngsolve_addon_template in GitHub. The documentation there clarifies the multiple ways to install the add-on and make it work seamlessly with your existing ngsolve install.

18.1.1.1. A simple use-case scenario#

As an example, let us walk through the following simple scenario where you want to build an extension. You have just taken a look at the source code in ngsolve_addon_template/src/my_coefficient.hpp and you are super excited about the eigenvalue CoefficientFunction there! So you immediately install it using pip.

# remove any prior install:
!python3 -m pip uninstall -y ngsolve_addon_template &> /dev/null
# prerequisites:
!python3 -m pip install scikit-build-core pybind11_stubgen toml &> out_prereqs.log
# install the add-on:
!python3 -m pip install --no-build-isolation git+https://github.com/NGSolve/ngsolve-addon-template.git  &> out_addon_install.log
# print collected long outputs:
with open('out_prereqs.log', 'r') as f: print(f.read())
with open('out_addon_install.log', 'r') as f: print(f.read())

WARNING: Ignoring invalid distribution ~etgen-mesher (/home/jschoebe/ngs24/lib/python3.11/site-packages)
WARNING: Ignoring invalid distribution ~etgen-mesher (/home/jschoebe/ngs24/lib/python3.11/site-packages)
Requirement already satisfied: scikit-build-core in /home/jschoebe/ngs24/lib/python3.11/site-packages (0.9.6)
Requirement already satisfied: pybind11_stubgen in /home/jschoebe/ngs24/lib/python3.11/site-packages (2.5.1)
Requirement already satisfied: toml in /home/jschoebe/ngs24/lib/python3.11/site-packages (0.10.2)
Requirement already satisfied: packaging>=21.3 in /home/jschoebe/ngs24/lib/python3.11/site-packages (from scikit-build-core) (24.0)
Requirement already satisfied: pathspec>=0.10.1 in /home/jschoebe/ngs24/lib/python3.11/site-packages (from scikit-build-core) (0.12.1)
WARNING: Ignoring invalid distribution ~etgen-mesher (/home/jschoebe/ngs24/lib/python3.11/site-packages)
WARNING: Ignoring invalid distribution ~etgen-mesher (/home/jschoebe/ngs24/lib/python3.11/site-packages)

WARNING: Ignoring invalid distribution ~etgen-mesher (/home/jschoebe/ngs24/lib/python3.11/site-packages)
WARNING: Ignoring invalid distribution ~etgen-mesher (/home/jschoebe/ngs24/lib/python3.11/site-packages)
Collecting git+https://github.com/NGSolve/ngsolve-addon-template.git
  Cloning https://github.com/NGSolve/ngsolve-addon-template.git to /tmp/pip-req-build-xghazwk9
  Running command git clone --filter=blob:none --quiet https://github.com/NGSolve/ngsolve-addon-template.git /tmp/pip-req-build-xghazwk9
  Resolved https://github.com/NGSolve/ngsolve-addon-template.git to commit 9a3a87df55cd4fd7586dea495354d9207b3b1fe5
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: ngsolve==6.2.2404.post22.dev0 in /home/jschoebe/ngs24/lib/python3.11/site-packages (from ngsolve_addon_template==0.0.1) (6.2.2404.post22.dev0)
Collecting netgen-mesher==6.2.2404.post5.dev (from ngsolve==6.2.2404.post22.dev0->ngsolve_addon_template==0.0.1)
  Using cached netgen_mesher-6.2.2404.post5.dev0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (173 bytes)
Requirement already satisfied: mkl in /home/jschoebe/ngs24/lib/python3.11/site-packages (from ngsolve==6.2.2404.post22.dev0->ngsolve_addon_template==0.0.1) (2024.1.0)
Requirement already satisfied: intel-openmp==2024.* in /home/jschoebe/ngs24/lib/python3.11/site-packages (from mkl->ngsolve==6.2.2404.post22.dev0->ngsolve_addon_template==0.0.1) (2024.1.0)
Requirement already satisfied: tbb==2021.* in /home/jschoebe/ngs24/lib/python3.11/site-packages (from mkl->ngsolve==6.2.2404.post22.dev0->ngsolve_addon_template==0.0.1) (2021.12.0)
Using cached netgen_mesher-6.2.2404.post5.dev0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (28.8 MB)
Building wheels for collected packages: ngsolve_addon_template
  Building wheel for ngsolve_addon_template (pyproject.toml): started
  Building wheel for ngsolve_addon_template (pyproject.toml): finished with status 'done'
  Created wheel for ngsolve_addon_template: filename=ngsolve_addon_template-0.0.1-cp311-cp311-linux_x86_64.whl size=82342 sha256=cc620488f1dd9d7032130874d52d064f054317d2c17d7488a6ff92e0b3a4a513
  Stored in directory: /tmp/pip-ephem-wheel-cache-19a_l_m1/wheels/6e/1b/ba/79fe9c84cdea6d4926bd8c66e0f5f75083b7bf62846bc80c15
Successfully built ngsolve_addon_template
WARNING: Ignoring invalid distribution ~etgen-mesher (/home/jschoebe/ngs24/lib/python3.11/site-packages)
Installing collected packages: netgen-mesher, ngsolve_addon_template
WARNING: Ignoring invalid distribution ~etgen-mesher (/home/jschoebe/ngs24/lib/python3.11/site-packages)
Successfully installed netgen-mesher ngsolve_addon_template-0.0.1
WARNING: Ignoring invalid distribution ~etgen-mesher (/home/jschoebe/ngs24/lib/python3.11/site-packages)

Now that it is installed, you immediately try it out. There is a particular symmetric \(2 \times 2\) matrix-valued function whose difference of eigenvalues you have been wanting to plot for a long time \(\ldots\) finally, you can.

from ngsolve import CF, Mesh, unit_square, x, y, Integrate
import ngsolve_addon_template as addon
from ngsolve.webgui import Draw

mesh = Mesh(unit_square.GenerateMesh(maxh=0.3))
A = CF( (1, y,       # Matrix whose eigenvalues you desperately want
         y, x+5) ).Reshape((2,2))
ews = addon.EigH(A)  # Eigenvalues of A as a vector CF 
Draw(ews[0] - ews[1], mesh);

Loading ngsolve_addon_template

18.1.1.2. You want more#

As a typical ngsolve user, you have worked with many other ngsolve coefficient functions. So you know you can not only Draw them, but also evaluate them at a mesh point, see its expression tree, Integrate them, Differentiate them, etc.

So you are surprised when Diff applied to this new eigenvalue coefficient function results in problems:

try: 
    dews = ews[0].Diff(x)

    if abs(Integrate(dews*dews, mesh)) < 1e-15: 
        raise ValueError('Derivative is claimed to be 0')
except Exception as e:
    print('EXCEPTION RAISED! Message:\n', e)    

EXCEPTION RAISED! Message:
 Derivative is claimed to be 0

After you have recovered from this devastating news, you resolve to remedy this state of affairs.

18.1.1.3. Identifying the issues#

Printing the expression tree of the eigenvalue coefficient funtion reveals one of the problems: ews[0] depends on ews as expected, but the dependence of ews on the input matrix A is not visible:

print(ews[0])

coef ComponentCoefficientFunction 0, real
  coef N5ngfem7EigH_CFE, real, dim=2

Looking at NGSolve’s source code coefficient.cpp, you find the derivative returned is ZeroCF because the new eigenvalue coefficient function has no InputCoefficientFunctions. The lack of InputCoefficientFunctions is also the reason for the input matrix not showing up when you printed the expression tree.

A simple fix would be to provide a definition for the virtual function

virtual Array<shared_ptr<CoefficientFunction>> InputCoefficientFunctions() const

in the new derived coefficient function class.

There is another issue. Looking at NGSolve’s source code coefficient.cpp, you find that implementation of the Diff member function is left up to the developer of new coefficient functions! Instead of shooting off an email to the developer, you decide that the right move is to reuse the already written eigenvalue coefficient functions and add the Diff member function yourself. This takes some work.

18.1.1.4. Get your own fork#

So, you fork the repo ngsolve_addon_template at GitHub and prepare to add two new member functions to the eigenvalue coefficient function class, that would override the base class definitions:

virtual Array<shared_ptr<CoefficientFunction>>
InputCoefficientFunctions() const override  {
    // Input matrix "mat" is stored as a private member. Just return it:
    return Array<shared_ptr<CoefficientFunction>>({mat});
}

shared_ptr<CoefficientFunction>
Diff(const CoefficientFunction * var,
     shared_ptr<CoefficientFunction> dir) const override {

    // HOW? 
    
}

How do we do the Diff? There are a few ways of doing the math.

18.1.1.5. Doing the math#

The eigenvalues of a symmetric matrix

\[\begin{split} A = \begin{bmatrix} a & b \\ b & c \end{bmatrix} \end{split}\]

are

\[ \lambda_\pm = \frac 1 2 \left(a + c \pm \sqrt{ (a-c)^2 + b^2 } \right). \]

One approach is to differentiate \(\lambda_\pm\) as a function of \(a, b, c\) and then, per the chain rule, combine it with derivatives of \(a, b,\) and \(c\) with respect to \(V=\)var.

More elegant approaches are obtained by implicit differentiation. Since eigenvalues solve \(\det( \lambda I - A) = 0\), differentiating \(F(\lambda, A) = \det(\lambda I - A)\), we find that

\[ \frac{\partial \lambda}{\partial V} = -\frac{\partial F / \partial A } {\partial F / \partial \lambda} \frac{\partial A}{\partial V} \]

By the Jacobi formula, differentiating the determinant of \(A\) gives the cofactor matrix denoted by \(\text{cof}(A)\), so

\[ \frac{\partial \lambda}{\partial V} = \frac{\text{cof}(\lambda I - A)}{ \text{tr} (\text{cof}(\lambda I - A) )} : \frac{\partial A}{\partial V} \]

Once you implement this formula, you can differentiate the eigenvalue coefficient function.

See the fork jayggg/ngsolve-addon-template, specifically the Diff(…) member function there, for an example implementation. The key lines implementing the above formula are as follows:

shared_ptr<CoefficientFunction>
Diff(const CoefficientFunction * var,
     shared_ptr<CoefficientFunction> dir) const override {
    // ...    
	auto lam = MakeComponentCoefficientFunction(thisptr, i);
	auto cof = CofactorCF(mat - lam * IdentityCF(2));
	auto dA = mat->Diff(var, dir);
	auto tr = TraceCF(cof);
	dlam[i] = InnerProduct(cof, dA) / tr;	  
    // ...
}    

18.1.1.6. Install the fork#

To try this out, we remove the ngsolve-addon-template module we installed previously and then install the new replacement module from your fork.

!pip3  uninstall -y ngsolve-addon-template  &> out_uninstall.log
!python3 -m pip install --no-build-isolation git+https://github.com/jayggg/ngsolve-addon-template/   &> out_reinstall.log
with open('out_uninstall.log', 'r') as f:  print(f.read())
with open('out_reinstall.log', 'r') as f:  print(f.read())

WARNING: Ignoring invalid distribution ~etgen-mesher (/home/jschoebe/ngs24/lib/python3.11/site-packages)
Found existing installation: ngsolve_addon_template 0.0.1
Uninstalling ngsolve_addon_template-0.0.1:
  Successfully uninstalled ngsolve_addon_template-0.0.1

WARNING: Ignoring invalid distribution ~etgen-mesher (/home/jschoebe/ngs24/lib/python3.11/site-packages)
WARNING: Ignoring invalid distribution ~etgen-mesher (/home/jschoebe/ngs24/lib/python3.11/site-packages)
Collecting git+https://github.com/jayggg/ngsolve-addon-template/
  Cloning https://github.com/jayggg/ngsolve-addon-template/ to /tmp/pip-req-build-stu6omjg
  Running command git clone --filter=blob:none --quiet https://github.com/jayggg/ngsolve-addon-template/ /tmp/pip-req-build-stu6omjg
  Resolved https://github.com/jayggg/ngsolve-addon-template/ to commit cc4e3cf755caaaa360e6f374045b17883621dff4
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: ngsolve==6.2.2404.post22.dev0 in /home/jschoebe/ngs24/lib/python3.11/site-packages (from ngsolve_addon_template==0.0.1) (6.2.2404.post22.dev0)
Collecting netgen-mesher==6.2.2404.post5.dev (from ngsolve==6.2.2404.post22.dev0->ngsolve_addon_template==0.0.1)
  Using cached netgen_mesher-6.2.2404.post5.dev0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (173 bytes)
Requirement already satisfied: mkl in /home/jschoebe/ngs24/lib/python3.11/site-packages (from ngsolve==6.2.2404.post22.dev0->ngsolve_addon_template==0.0.1) (2024.1.0)
Requirement already satisfied: intel-openmp==2024.* in /home/jschoebe/ngs24/lib/python3.11/site-packages (from mkl->ngsolve==6.2.2404.post22.dev0->ngsolve_addon_template==0.0.1) (2024.1.0)
Requirement already satisfied: tbb==2021.* in /home/jschoebe/ngs24/lib/python3.11/site-packages (from mkl->ngsolve==6.2.2404.post22.dev0->ngsolve_addon_template==0.0.1) (2021.12.0)
Using cached netgen_mesher-6.2.2404.post5.dev0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (28.8 MB)
Building wheels for collected packages: ngsolve_addon_template
  Building wheel for ngsolve_addon_template (pyproject.toml): started
  Building wheel for ngsolve_addon_template (pyproject.toml): finished with status 'done'
  Created wheel for ngsolve_addon_template: filename=ngsolve_addon_template-0.0.1-cp311-cp311-linux_x86_64.whl size=84521 sha256=6b451d81fd8ace04e9a690addf0321f025bf5bcf30b131a0f138bb2b838fea61
  Stored in directory: /tmp/pip-ephem-wheel-cache-zvci009m/wheels/b5/0f/1e/f6c503bcbb84ad1620c9356b56e76f200c51dda8c86b05c910
Successfully built ngsolve_addon_template
WARNING: Ignoring invalid distribution ~etgen-mesher (/home/jschoebe/ngs24/lib/python3.11/site-packages)
Installing collected packages: netgen-mesher, ngsolve_addon_template
WARNING: Ignoring invalid distribution ~etgen-mesher (/home/jschoebe/ngs24/lib/python3.11/site-packages)
Successfully installed netgen-mesher ngsolve_addon_template-0.0.1
WARNING: Ignoring invalid distribution ~etgen-mesher (/home/jschoebe/ngs24/lib/python3.11/site-packages)

import ngsolve_addon_template_withdiff as addon2

ews = addon2.EigH(A)
dews = ews.Diff(x)
Draw(dews[0], mesh);

This concludes the walk-through of a simple use-case scenario. Pointers to more examples of add-ons are below.

Further examples of add-on modules:

Special functions extension: Bessel, Hankel, erf, etc.
Hypre-Interface: Install hypre and connect to ngsolve.
Teaching implementation of elements: Write your own element.

18.1.2. One-file extensions#

An alternate method to install a small C++ extension is to use NGSolve’s facility called CompilePythonModule. In this method, you write your c++ add-on (with python bindings defined using pybind11) in a single file, say “file.cpp”, which may include other C++ files. Then calling

m = CompilePythonModule(file.cpp)

in python will trigger NGSolve to compile your code and provide your extended facilities in the python workspace through the module variable m. In order for this compilation to work, the manner you installed NGSolve (built from source or by pip-installed binary) is not relevant, but it is important that your system has local build facilities, like a compiler.

Further pointers and examples for CompilePythonModule: