PicoBlog

This is just a short blog on a “gotcha” that initially caught me while developing dynamic library code on OS X.

Suppose that you are developing a dynamic library which can act as a runtime replacement for an existing runtime library (e.g. LOMP as a, currently incomplete and risky, replacement for the CLANG or GCC OpenMP runtime).

You want to have an existing executable which has been dynamically linked against the compiler’s runtime use your runtime instead.

For instance, you want to run benchmarks such as the EPCC OpenMP micro-benchmarks with the same executable but testing the performance of each runtime library.

Since, as a software developer, you are, justifiably paranoid, you also want to be sure that the code really is using your library, not the system one you are replacing.

On Linux getting the code to use your library can be achieved by putting the directory in which your library (which has the same name as the compiler’s) can be found into the LD_LIBRARY_PATH envirable. The dynamic linker searches for libraries in the directories in $LD_LIBRARY_PATH before looking in the directories built into the image, so will find yours and stop searching.

Checking what is going on can be achieved by using ldd.

Here you can see me compiling a trivial OpenMP code (using one of the system compilers), running it, checking which OpenMP runtime library it is using, then using LD_LIBRARY_PATH to use the LOMP runtime instead (and out of real paranoia, checking that by asking LOMP to announce itself)

$ armclang -fopenmp hello_omp.c $ OMP_NUM_THREADS=2 ./a.out Hello from thread 0 Hello from thread 1 # Check the libraries used $ ldd ./a.out ... elided ... libomp.so => /lustre/software/aarch64/tools/arm-compiler/20.1//arm-linux-compiler-20.1_Generic-AArch64_SUSE-12_aarch64-linux/lib/libomp.so (0x0000400019bc3000) ... elided ... # Run using LOMP as the OpenMP runtime $ LD_LIBRARY_PATH=`echo ~/lomp/build_aarch64/src`:${LD_LIBRARY_PATH} OMP_NUM_THREADS=2 ./a.out Hello from thread 0 Hello from thread 1 # Check the libraries used $ LD_LIBRARY_PATH=`echo ~/lomp/build_aarch64/src`:${LD_LIBRARY_PATH} ldd ./a.out ... elided ... libomp.so => /home/br-jcownie/lomp/build_aarch64/src/libomp.so (0x000040002dfeb000) ... elided ... # Prove that I'm using LOMP by asking it to announce itself $ LD_LIBRARY_PATH=`echo ~/lomp/build_aarch64/src`:${LD_LIBRARY_PATH} OMP_NUM_THREADS=2 LOMP_DEBUG=1 ./a.out LOMP:runtime version 0.1 (SO version 1) compiled at 12:04:35 on Jul 20 2021 LOMP:from Git commit 5db9696 for aarch64 by LLVM:11:0:0 LOMP:with configuration -march=armv8.1a;DEBUG=10;LOMP_WARN_API_STUBS=1;LOMP_WARN_ARCH_FEATURES=1;LOMP_HAVE_LIBATOMIC=\ 1;LOMP_HAVE_LIBNUMA=1 Hello from thread 0 Hello from thread 1 $

Asking the web shows that MacOS does not use LD_LIBRARY_PATH, but rather DYLD_LIBRARY_PATH, and does not have an ldd command, but, instead provides the same information using the otool -L option.

“So, that’s all good then”.

Doing the same things as above, here’s what we see :-

$ clang -fopenmp hello_omp.c $ OMP_NUM_THREADS=2 ./a.out Hello from thread 0 Hello from thread 1 $ otool -L ./a.out ./a.out: /opt/homebrew/opt/llvm/lib/libomp.dylib (compatibility version 5.0.0, current version 5.0.0) ... elided ... # Now set DYLD_LIBRARY_PATH and check $ DYLD_LIBRARY_PATH=`echo ~/lomp/build/src`:${DYLD_LIBRARY_PATH} otool -L ./a.out ./a.out: /opt/homebrew/opt/llvm/lib/libomp.dylib (compatibility version 5.0.0, current version 5.0.0) ... elided ... # Did we mis-spell the path? Try running anyway $ DYLD_LIBRARY_PATH=`echo ~/lomp/build/src`:${DYLD_LIBRARY_PATH} LOMP_DEBUG=1 OMP_NUM_THREADS=2 ./a.out LOMP:runtime version 0.1 (SO version 1) compiled at 11:41:47 on Aug 4 2021 LOMP:from Git commit 700e26b for aarch64 by LLVM:12:0:1 LOMP:with configuration LOMP_COMPILE_OPTIONS-NOTFOUND;DEBUG=10;LOMP_WARN_API_STUBS=1;LOMP_WARN_ARCH_FEATURES=1  Hello from thread 0 Hello from thread 1 

The final execution proves that we do get the runtime we wanted, but otool -L told us the code would run with compiler’s runtime! How helpful…

We are hitting a piece of MacOS security enforcement!

By default modern instances of MacOS implement “System Integrity Protection” (SIP), and one of the things SIP does is to “sanitize” critical environment variables (such as DYLD_LIBRARY_PATH) when executing system commands (such as otool). This is done to prevent important executables from loading untrusted dynamic libraries. (Imagine what you could do if you loaded your own version of the system library into a process running with all privileges!)

Here, though, the effect is to make otool -L lie about the libraries that will actually be loaded by the process when it is not running under otool, which makes it positively misleading, rather than useful.

Instead of using ldd or otool, we can directly ask the dynamic linker to show us which libraries it loads. As ever, this is achieved in a slightly different way on OS X than on Linux.

On Linux the dynamic linker supports the LD_TRACE_LOADED_OBJECTS envirable, which, when set to 1, asks the dynamic linker to output information about all loaded code, and then exit before starting the program. In effect this is almost exactly like running ldd.

LD_LIBRARY_PATH=`echo ~/lomp/build_aarch64/src`:${LD_LIBRARY_PATH} LD_TRACE_LOADED_OBJECTS=1 ./a.out ... elided ...  libomp.so => /home/br-jcownie/lomp/build_aarch64/src/libomp.so (0x00004000314e3000) ... elided ...

Here the magic envirable is DYLD_PRINT_LIBRARIES, which causes the dynamic loader to print information about each library which is loaded if the envirable is set to 1.

One difference, though, is that the dynamic linker continues on to execute the code, so this isn’t exactly the same as ldd or otool.

$ DYLD_LIBRARY_PATH=`echo ~/lomp/build/src`:${DYLD_LIBRARY_PATH} OMP_NUM_THREADS=2 DYLD_PRINT_LIBRARIES=1 ./a.out dyld: loaded: <66F0786F-2994-39D6-9FCA-55CAC207B9B8> /Users/jcownie/tmp/./a.out dyld: loaded: <0368D8A6-C4DA-3B2C-8683-30AB51F7E8D0> /Users/jcownie/lomp/build/src/libomp.dylib ... elided ... LOMP:runtime version 0.1 (SO version 1) compiled at 11:41:47 on Aug 4 2021 LOMP:from Git commit 700e26b for aarch64 by LLVM:12:0:1 LOMP:with configuration LOMP_COMPILE_OPTIONS-NOTFOUND;DEBUG=10;LOMP_WARN_API_STUBS=1;LOMP_WARN_ARCH_FEATURES=1 Hello from thread 0 Hello from thread 1 $ 

Above you can see (both from the dyld trace and the behaviour) that the libomp we wanted is being loaded

• OS X is not Linux; although the shell may be the same there are significant differences.

• Be careful when people say “The MacOS equivalent of Linux’ X is MacOS’ Y”. The equivalence may not be complete and may be positively misleading!

• If you think you want an ldd equivalent on OS X you’re likely to be less misled if you use DYLD_PRINT_LIBRARIES instead of a command.

This work used the Isambard 2 UK National Tier-2 HPC Service operated by GW4 and the UK Met Office, and funded by EPSRC (EP/T022078/1)

ncG1vNJzZmiboKqztrqNrKybq6SWsKx6wqikaKhfrLWqr8dmo6Kaopa%2FunnDoptmsZ%2BqerSt2GauoqScYq%2Bm