Using LLVM to improve CppAD

[bradbell]

Introduciton
This is really a request for help in CppAD development that is closely related to CppADCodegen.
I have questions about how to use LLVM and I hoping this group can help me.

Optimization
I am working on using the llvm optimizer as a replacement for the CppAD optimizer; see the example:
https://github.com/coin-or/CppAD/blob/llvm/example/llvm/optimize.cpp
I am close to being able to test how this compares to the CppAD optimizer for a simple example.

Complied Object Files
As a side benefit, inspired by CppADCodeGen, CppAD should be able to directly general object libraries corresponding to the llvm IR representation of an ADFun; see the example
https://github.com/coin-or/CppAD/blob/llvm/example/llvm/grad_det_minor.cpp
I am not sure how this will compare to the CppADCodeGen method for generating code.

Development
Work on this effect is happening on the llvm branch of CppAD in the following files and directories
https://github.com/coin-or/CppAD/blob/llvm/include/cppad/core/llvm_ir.hpp
https://github.com/coin-or/CppAD/blob/llvm/include/cppad/core/llvm_link.hpp
https://github.com/coin-or/CppAD/tree/llvm/cppad_lib/llvm
https://github.com/coin-or/CppAD/tree/llvm/example/llvm
https://github.com/coin-or/CppAD/blob/llvm/test_more/general/llvm_tst.cpp

LLVM Version
This work is currently only tested with the following version of llvm:
https://github.com/llvm/llvm-project/tree/release/11.x

Questions
I will post individual questons below.

[bradbell]

Linking Standard Math Functions
I cannot figure out how to link the standard math functions to the LLJIT.

In the llvm branch execute the following command to configure cppad:
bin/run_cmake.sh --no_optional
Then compile and run llvm_tst.cpp using the command
bin/test_one.sh test_more/general/llvm_tst.cpp
This will first compile cppad_lib then llvm_tst.cpp and then run the test with the following result.
... snip ...

Begin test group test_more/general
llvm_tst            OK
memory_leak         OK
All 2 tests passed.
End test group test_more/general
test_one.sh: OK

... snip ...

If you now remove the lines containing NOT WORKING in llvm_tst.cpp and run
bin/test_one.sh test_more/general/llvm_tst.cpp
you will get
... snip ...

Begin test group test_more/general
JIT session error: Symbols not found: [ acosh ]
llvm_tst            
llvm::compiled: Error searching for llvm_tst in object file
Error
memory_leak         OK
1 tests failed.
End test group test_more/general
export LD_LIBRARY_PATH=/home/bradbell/repo/cppad.git/build/cppad_lib

... snip ..
Where /home/bradbell/repo/cppad.git will be replaced by the location of the cppad git repository on your system.
You can run the progam in the debugger using the following commands

export LD_LIBRARY_PATH=/home/bradbell/repo/cppad.git/build/cppad_lib
gdb test_one.exe
run

Then you can set break points in llvm_tst.cpp to see exactly what llvm commands are being used.

[bradbell]

Linking Standard Math Functions
Correction: This also fails if we replace all occurrences of acosh by sin (I have pushed this change).

I think this is related to the following text: If no definition is found inside the JIT, it falls back to calling “dlsym("sin")” on the Kaleidoscope process itself. Since “sin” is defined within the JIT’s address space, it simply patches up calls in the module to call the libm version of sin directly. which appears in
https://llvm.org/docs/tutorial/MyFirstLanguageFrontend/LangImpl04.html

So I should be able to find the problem by looking at the llvm Kaleidoscope example.

[bradbell]

The main problem was that adding object files and adding shared libraries use different llvm routines.
I was able to solve the problem by creating separate routines for the two operations; see
https://github.com/coin-or/CppAD/blob/llvm/cppad_lib/llvm/llvm_link.cpp
https://github.com/coin-or/CppAD/blob/llvm/cppad_lib/llvm/dynamic_lib.cpp

[bradbell]

I have all the features of a ADFun working both for optimizing and jit compiling using llvm.
I implemented an llvm version of the det_minor speed test and it indicates that (for larger problems) CppADCodeGen is faster.
In addition, the conversion for llvm IR back to a CppAD graph representation is quite complicated.
You should be able to reproduce these results by checking out the llvm branch of cppad and running
bin/speed_llvm.sh

Here are a few results:

Size of the matrix:
det_minor_size = [ 2, 3, 4, 5, 6 ]

Retape for each function evaluation:
cppadcg_det_minor_rate = [ 2.4287, 2.4649, 2.3641, 1.8154, 0.8394 ]
llvm_det_minor_rate = [ 208.54, 88.94, 17.16, 2.1742, 0.1018 ]

Re-use the tape / jit complie:
cppadcg_det_minor_rate = [ 5107406, 3355366, 2155605, 1188629, 356046 ]
llvm_det_minor_rate = [ 10411684, 4708980, 2280245, 1079332, 287352 ]