Mucking around with libc++ on Android

tl;dr

libc++ on Android doesn’t work yet as of October 2013.

Getting the test suite running

The LLVM project-created libc++ library isn’t yet fully supported on Android, but I wanted to see how much of its test suite would currently work on an Android phone. Considering the weaknesses of all the other C++ Standard Template Libraries out there, libc++ becoming available would be a huge boon to developing a modern codebase. There are a few notes about this in one long-running feature request, and according to the git logs there are maybe 2 people actually working full-time on the NDK at Google, but there’s not much else to note.

Easier said than done, as usual, and it’s not really in Google’s interest to get this working, since everyone is supposed to write things specificially in Java. But it is the one stumbling block that prevents a company from writing their core logic a single time in modern C++11 (with conveniences like support for atomics, threads, conditional variables, and mutexes) and having that run across all existing desktop and mobile platforms. I’ll have to take a look at whether the g++ 4.8 STL support is complete enough to cover the C++11 cases I want to use. Everyone else (Microsoft, Apple) has a usable STL implementation on their desktop and mobile platforms. A functioning libc++ might also let people avoid pulling in the entirety of a library like Qt to plug the gaps, and as it is Qt is in an alpha stage of support for Android and iOS.

Anyway, these are some of the notes I took while getting the test suite going on a rooted smartphone.

Clone the latest Google NDK repository sources in $HOME:

git clone https://android.googlesource.com/platform/ndk git-ndk

Grab a copy of the prepackaged NDK from the Android developer website, at the time of this writing version r9. Unpack this in $HOME.

Create a symbolic link from sources/cxx-stl/llvm-libc++ folder in the Git checkout to the sources/cxx-stl folder in the officially packaged NDK folder. This saves you from having to build the cross-compilers yourself.

ln -s git-ndk/sources/cxx-stl/llvm-libc++ android-ndk-r9/sources/cxx-stl/llvm-libc++

Edit the android-ndk-r9/sources/cxx-stl/llvm-libc++/test/../../gabi++/include/gabixx_config.h file to use the __typeof__ operator instead of typeof, because of Problems 3 and 4 (detailed below) encountered in --std=c++11 mode:

After making this change, go to the android-ndk-r9/tests/device/test-libc++ folder and try building the test libc++ executable, but limit the build to armeabi, otherwise the ndk-build will try building 8 versions of the libc++ libraries (static, shared) x (armeabi, armeabi-v7a, mips, x86):

android-ndk-r9/tests/device/test-libc++$ ndk-build APP_ABI=armeabi

The code compiles, but there’s still a ton of stuff in the library not yet implemented, and you’ll see a lot of #warning Not implemented messages, wherever functionality has been stubbed out.

[...]
Compile++ thumb  : c++_static  libs/armeabi/test_1_static

OK, so the library builds, now to get the testit_android test runner running in android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test:

Set up the libc++ library include path to look like:

The sysroot setting needs to be fixed up in the testit_android script, in this case I set it to the android-9 platform, which reaches back to roughly the Android 2.2 release:

Add ANDROID_SYSROOT to the two compilation commands:

Now connect a rooted Android phone to your development computer.

Running testit_android should work, first pushing a copy of the libc++ shared library to the smartphone, then compiling, pushing, and running the test executables on the smartphone:

android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test$ ./testit_android --abi=armeabi
passed 1 tests in /home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test
passed 1 tests in /home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test/algorithms
passed 1 tests in /home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test/algorithms/alg.c.library
passed 1 tests in /home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test/algorithms/alg.modifying.operations
[...]

Unfortunately, after running several thousand tests for over an hour, testit_android seemed to hang when attempting to run tests within the support.dynamic test suite:

[...]
failed 1 tests in /home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test/language.support/support.dynamic/alloc.errors/set.new.handler
passed 1 tests in /home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test/language.support/support.dynamic/alloc.errors/set.new.handler

real	68m19.512s
user	0m0.004s
sys	0m0.000s

I removed the llvm-libc++/libcxx/test/support.dynamic test suite and reran testit_android, which resulted in the full suite completing:

****************************************************
Results for /home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test:
using arm-linux-androideabi-g++ (GCC) 4.8
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
with -std=c++11 -I/home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test/support -I/home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/include -I/home/builder/android-ndk-r9/sources/android/support/include -L/home/builder/android-ndk-r9/tests/device/test-libc++/obj/local/armeabi
----------------------------------------------------
sections without tests   : 0
sections with failures   : 199
sections without failures: 861
                       +   ----
total number of sections : 1060
----------------------------------------------------
number of tests failed   : 683
number of tests passed   : 3716
                       +   ----
total number of tests    : 4399
****************************************************

real	141m23.980s
user	67m18.716s
sys	21m46.932s

As usual, your mileage may vary, but this is the state of libc++ on Android in October 2013.

Notes

1. How about using Clang, since it should work better?

According to the NDK page, you can specify the compiler to use via the NDK_TOOLCHAIN_VERSION variable:

“Fixed the ndk-build script to enable you to specify a version of Clang as a command line option (e.g., NDK_TOOLCHAIN_VERSION=clang3.2). Previously, only specifying the version as an environment variable worked.”

So this works too:

ndk-build NDK_TOOLCHAIN_VERSION=clang3.3 APP_ABI=armeabi

And produces output like:

[...]
Compile++ thumb  : c++_shared 
The problem is that the testit_android script is currently hard-coded to use arm-linux-androideabi-g++ to build the test suite executables. The LLVM compiler that ships with Android NDK r9 seems to be an all-in-one cross-compiler, which can generate code for ARM, MIPS, and x86 depending on which config.mk files are supplied to configure the current build. So getting testit_android to build ARM executables with Clang would be a bit more work, though probably not that much more work.
2. Relation between the Android.mk files currently
android-ndk-r9/tests/device/test-libc++/jni/Android.mk (immediately imports test executable makefile) -> 
android-ndk-r9/sources/cxx-stl/llvm-libc++/test/Android.mk (builds the test executables) -> 
android-ndk-r9/sources/cxx-stl/llvm-libc++/Android.mk (builds the actual libc++ static and shared libraries)

Problems encountered
1. Attempting to build the tests in the Git working directory (git-ndk) without first creating the symbolic link in the release NDK directory (android-ndk-r9)
git-ndk/tests/device/test-libc++$ ndk-build
jni/Android.mk:1: /home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/test/Android.mk: No such file or directory
jni/Android.mk:1: /home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/test/Android.mk: No such file or directory
jni/Android.mk:1: /home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/test/Android.mk: No such file or directory
jni/Android.mk:1: /home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/test/Android.mk: No such file or directory
/home/builder/android-ndk-r9/build/core/build-all.mk:89: Android NDK: WARNING: There are no modules to build in this project!    
make: *** No rule to make target `/home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/test/Android.mk'.  Stop.

2. Attempting to build the tests by specifying NDK_ROOT with ndk-build on the PATH
git-ndk/tests/device/test-libc++$ ndk-build NDK_ROOT=$HOME/git-ndk
Android NDK: Could not find platform files (headers and libraries)    
Android NDK: Please run build/tools/build-platforms.sh to build the corresponding directory.    
/home/builder/git-ndk/build/core/init.mk:471: *** Android NDK: Aborting    .  Stop.

Problems 1 and 2 are solved by creating the symbolic link as described above and running the ndk-build command from within the android-ndk-r9 folder structure. 
3. Attempting to build with GNU g++ fails due to the use in GAbi++ of a GNU-specific 'typeof' keyword which isn't recognized in --std=c++11 mode
Compile++ thumb  : c++_shared 
4. Attempting to build with Clang 3.3 fails due to the use in GAbi++ of a GNU-specific 'typeof' keyword which isn't recognized in --std=c++11 mode
Compile++ thumb  : c++_shared 
typeof is a GNU compiler-specific keyword extension to the C standard, meaning that other compilers won't like it very much. Darn it. The compilation commands for g++ and clang++ use --std=c++11, which does not allow the use of the typeof keyword. 
However, according to the Clang user manual:
The parser recognizes "asm" and "typeof" as keywords in gnu* modes; the variants "__asm__" and "__typeof__" are recognized in all modes.
The same appears to be true of the GNU compiler, therefore replacing typeof with __typeof__ fixes the problem on both.
Running testit_android
1. Missing prebuilt library
android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test$ ./testit_android --abi=armeabi
ERROR: Missing prebuilt library: /home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libs/armeabi/libc++_static.a
Please run: build/tools/build-cxx-stl.sh --stl=libc++

But the library is already built here:
android-ndk-r9/sources/cxx-stl/llvm-libc++/libs/armeabi/libc++_static.a

So, the testit_android script just needs to be modified, where the line:
LIBCXX_LIBS=$(cd $LIBCXX_ROOT/.. && pwd)/libs/$TARGET_ABI

Needs to be changed to:
LIBCXX_LIBS=$(cd $LIBCXX_ROOT/../../../.. && pwd)/tests/device/test-libc++/obj/local/$TARGET_ABI

2. Missing C++ compiler
./testit_android --abi=armeabi
ERROR: Missing C++ compiler: arm-linux-androideabi-g++

Now, put that compiler into the PATH:
export PATH=$HOME/android-ndk-r9/toolchains/arm-linux-androideabi-4.8/prebuilt/linux-x86_64/bin:$PATH

3. Missing platform headers and libraries
The build command doesn't know which Android platform to compile and link against:
/home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test/containers/container.adaptors/queue/queue.defn/assign_move.pass.cpp failed to compile
Compile line was: ( cd /home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test/containers/container.adaptors/queue/queue.defn && arm-linux-androideabi-g++ -std=c++11 -I/home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test/support -I/home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/include -I/home/builder/android-ndk-r9/sources/android/support/include -L/home/builder/android-ndk-r9/tests/device/test-libc++/obj/local/armeabi assign_move.pass.cpp -lc++_shared )

android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test$ cd /home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test/containers/container.adaptors/queue/queue.defn && arm-linux-androideabi-g++ -std=c++11 -I/home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test/support -I/home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/include -I/home/builder/android-ndk-r9/sources/android/support/include -L/home/builder/android-ndk-r9/tests/device/test-libc++/obj/local/armeabi assign_move.pass.cpp -lc++_shared
In file included from /home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/include/queue:168:0,
                 from assign_move.pass.cpp:14:
/home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/include/__config:99:21: fatal error: endian.h: No such file or directory
 # include <endian.h>
                     ^
compilation terminated.

android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test$ cd /home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test/algorithms/alg.modifying.operations/alg.copy && arm-linux-androideabi-g++ -std=c++11 -I/home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/test/support -I/home/builder/android-ndk-r9/sources/cxx-stl/llvm-libc++/libcxx/include -I/home/builder/android-ndk-r9/sources/android/support/include -I/home/builder/android-ndk-r9/sources/android/support/../../../platforms/android-9/arch-arm/usr/include -L/home/builder/android-ndk-r9/tests/device/test-libc++/obj/local/armeabi copy_n.pass.cpp -lc++_shared
/home/builder/android-ndk-r9/toolchains/arm-linux-androideabi-4.8/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.8/../../../../arm-linux-androideabi/bin/ld: error: cannot open crtbegin_dynamic.o: No such file or directory
/home/builder/android-ndk-r9/toolchains/arm-linux-androideabi-4.8/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.8/../../../../arm-linux-androideabi/bin/ld: error: cannot open crtend_android.o: No such file or directory
/home/builder/android-ndk-r9/toolchains/arm-linux-androideabi-4.8/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.8/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lstdc++
/home/builder/android-ndk-r9/toolchains/arm-linux-androideabi-4.8/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.8/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lm
/home/builder/android-ndk-r9/toolchains/arm-linux-androideabi-4.8/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.8/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lc
/home/builder/android-ndk-r9/toolchains/arm-linux-androideabi-4.8/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.8/../../../../arm-linux-androideabi/bin/ld: error: cannot find -ldl
/tmp/cc55fvWI.o:copy_n.pass.cpp:function void test, output_iterator >(): error: undefined reference to 'memset'
/tmp/cc55fvWI.o:copy_n.pass.cpp:function void test, output_iterator >(): error: undefined reference to '__assert2'
/tmp/cc55fvWI.o:copy_n.pass.cpp:function void test, output_iterator >(): error: undefined reference to '__assert2'
/tmp/cc55fvWI.o:copy_n.pass.cpp:function void test, input_iterator >(): error: undefined reference to 'memset'
/tmp/cc55fvWI.o:copy_n.pass.cpp:function void test, input_iterator >(): error: undefined reference to '__assert2'
/tmp/cc55fvWI.o:copy_n.pass.cpp:function void test, input_iterator >(): error: undefined reference to '__assert2'
/tmp/cc55fvWI.o:copy_n.pass.cpp:function void test, forward_iterator >(): error: undefined reference to 'memset'
/tmp/cc55fvWI.o:copy_n.pass.cpp:function void test, bidirectional_iterator >(): error: undefined reference to 'memset'
/tmp/cc55fvWI.o:copy_n.pass.cpp:function void test(): error: undefined reference to 'memmove'
collect2: error: ld returned 1 exit status

As specified above, this problem is fixed by the addition of the ANDROID_SYSROOT variable to the build command, to indicate which Android platform includes and libraries the compiler should use.