Getting a sense of code coverage for open-source projects is necessary due-diligence for any dependent project. On OS X, it’s also a little more work. While doing some of my own research, I wanted to see how well tested the libevent library was, but wasn’t finding much info online. So here’s a log of what I needed to do, to get this information on a Mac.
First things first (and this should apply for many more open-source projects), after I checked out the latest code from github, I added an option to the configure.ac Autoconf source file to insert the necessary profiling code calls for code coverage:
|
1 2 3 |
AC_ARG_ENABLE(coverage, [ --enable-coverage Enable coverage testing], [CFLAGS="$CFLAGS -fprofile-arcs -ftest-coverage"]) |
With that added, I reran the autogen.sh file, which pulls in the configure.ac file, and regenerates the configure script, and then I ran ./configure --enable-coverage.
Then I ran make and specified clang as the C compiler instead of old-school gcc. Besides better code quality and error reports, only clang will generate the coverage code. The Apple version of gcc did not do so, leading to some initial confusion.
|
1 |
make CC=clang |
Once the build was complete, I ran the regression tests with:
|
1 |
make verify |
Unfortunately, the following error occurred:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
make check-am make check-TESTS Running tests: EPOLL (timerfd) ./test/test-script.sh: line 66: 62222 Trace/BPT trap: 5 $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE" Skipping test EPOLL (changelist) ./test/test-script.sh: line 66: 62237 Trace/BPT trap: 5 $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE" Skipping test EPOLL (timerfd+changelist) ./test/test-script.sh: line 66: 62251 Trace/BPT trap: 5 $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE" Skipping test EVPORT ./test/test-script.sh: line 66: 62265 Trace/BPT trap: 5 $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE" Skipping test KQUEUE ./test/test-script.sh: line 66: 62279 Trace/BPT trap: 5 $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE" Skipping test EPOLL ./test/test-script.sh: line 66: 62293 Trace/BPT trap: 5 $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE" Skipping test DEVPOLL ./test/test-script.sh: line 66: 62307 Trace/BPT trap: 5 $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE" Skipping test POLL ./test/test-script.sh: line 66: 62321 Trace/BPT trap: 5 $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE" Skipping test SELECT ./test/test-script.sh: line 66: 62335 Trace/BPT trap: 5 $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE" Skipping test WIN32 ./test/test-script.sh: line 66: 62349 Trace/BPT trap: 5 $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE" Skipping test PASS: test/test-script.sh ================== All 1 tests passed ================== |
So I tried running tests/regress directly and saw:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
$ test/regress main/methods: [forking] dyld: lazy symbol binding failed: Symbol not found: _llvm_gcda_start_file Referenced from: LibeventLibrary/source/.libs/libevent_openssl-2.1.3.dylib Expected in: flat namespace dyld: Symbol not found: _llvm_gcda_start_file Referenced from: LibeventLibrary/source/.libs/libevent_openssl-2.1.3.dylib Expected in: flat namespace OK main/version: OK main/base_features: [forking] dyld: lazy symbol binding failed: Symbol not found: _llvm_gcda_start_file Referenced from: LibeventLibrary/source/.libs/libevent_openssl-2.1.3.dylib Expected in: flat namespace dyld: Symbol not found: _llvm_gcda_start_file Referenced from: LibeventLibrary/source/.libs/libevent_openssl-2.1.3.dylib Expected in: flat namespace OK |
Oops.
Turns out that Apple uses the profile_rt library to handle code coverage instead of the former gcov library, which is why the _llvm_gcda_start_file function symbol is missing. So I linked to the libprofile_rt.dylib library by specifying LDFLAGS=-lprofile_rt on the make command line:
|
1 |
make CC=clang LDFLAGS=-lprofile_rt |
Rerunning make verify, the following was output, which indicated which of the event notification subsystems were available and being tested on the system:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 |
make check-am make check-TESTS Running tests: EPOLL (timerfd) Skipping test EPOLL (changelist) Skipping test EPOLL (timerfd+changelist) Skipping test EVPORT Skipping test KQUEUE test-eof: OKAY test-weof: OKAY test-time: OKAY test-changelist: OKAY test-fdleak: OKAY test-dumpevents: OKAY regress: WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818fOKAY EPOLL Skipping test DEVPOLL Skipping test POLL test-eof: OKAY test-weof: OKAY test-time: OKAY test-changelist: OKAY test-fdleak: OKAY test-dumpevents: OKAY regress: WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818fOKAY SELECT test-eof: OKAY test-weof: OKAY test-time: OKAY test-changelist: OKAY test-fdleak: OKAY test-dumpevents: OKAY regress: WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818fOKAY WIN32 Skipping test PASS: test/test-script.sh ================== All 1 tests passed ================== |
Once the regression tests finished, the coverage data became available in the form of *.gcno and *.gcda files in the test/ folder and in the .libs/ folder.
Running lcov generated easier-to-interpret HTML files:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 |
$ ~/Downloads/lcov-1.10/bin/lcov --capture --directory . --output-file coverage.info Capturing coverage data from . Found gcov version: 4.2.1 Scanning . for .gcda files ... Found 49 data files in . Processing .libs/buffer.gcda Processing .libs/bufferevent.gcda Processing .libs/bufferevent_filter.gcda Processing .libs/bufferevent_pair.gcda Processing .libs/bufferevent_ratelim.gcda Processing .libs/bufferevent_sock.gcda Processing .libs/evdns.gcda Processing .libs/event.gcda Processing .libs/event_tagging.gcda Processing .libs/evmap.gcda Processing .libs/evrpc.gcda Processing .libs/evthread.gcda Processing .libs/evthread_pthread.gcda Processing .libs/evutil.gcda Processing .libs/evutil_rand.gcda Processing .libs/evutil_time.gcda Processing .libs/http.gcda Processing .libs/kqueue.gcda Processing .libs/libevent_openssl_la-bufferevent_openssl.gcda Processing .libs/listener.gcda Processing .libs/log.gcda Processing .libs/poll.gcda Processing .libs/select.gcda Processing .libs/signal.gcda Processing test/test-changelist.gcda Processing test/test-dumpevents.gcda Processing test/test-eof.gcda Processing test/test-fdleak.gcda Processing test/test-init.gcda Processing test/test-time.gcda Processing test/test-weof.gcda Processing test/test_regress-regress.gcda Processing test/test_regress-regress.gen.gcda Processing test/test_regress-regress_buffer.gcda Processing test/test_regress-regress_bufferevent.gcda Processing test/test_regress-regress_dns.gcda Processing test/test_regress-regress_et.gcda Processing test/test_regress-regress_finalize.gcda Processing test/test_regress-regress_http.gcda Processing test/test_regress-regress_listener.gcda Processing test/test_regress-regress_main.gcda Processing test/test_regress-regress_minheap.gcda Processing test/test_regress-regress_rpc.gcda Processing test/test_regress-regress_ssl.gcda Processing test/test_regress-regress_testutils.gcda Processing test/test_regress-regress_thread.gcda Processing test/test_regress-regress_util.gcda Processing test/test_regress-regress_zlib.gcda Processing test/test_regress-tinytest.gcda Finished .info-file creation $ ~/Downloads/lcov-1.10/bin/genhtml coverage.info --output-directory html Reading data file coverage.info Found 54 entries. Found common filename prefix "LibeventLibrary" Writing .css and .png files. Generating output. Processing file source/bufferevent.c Processing file source/kqueue.c Processing file source/evmap.c Processing file source/http.c Processing file source/evthread-internal.h Processing file source/evthread_pthread.c Processing file source/select.c Processing file source/bufferevent_openssl.c Processing file source/buffer.c Processing file source/bufferevent_ratelim.c Processing file source/bufferevent_pair.c Processing file source/poll.c Processing file source/evutil.c Processing file source/evutil_rand.c Processing file source/evthread.c Processing file source/evutil_time.c Processing file source/evdns.c Processing file source/signal.c Processing file source/evrpc.c Processing file source/log.c Processing file source/bufferevent_sock.c Processing file source/event_tagging.c Processing file source/listener.c Processing file source/minheap-internal.h Processing file source/event.c Processing file source/bufferevent_filter.c Processing file source/test/regress_ssl.c Processing file source/test/regress_listener.c Processing file source/test/regress_thread.c Processing file source/test/test-eof.c Processing file source/test/regress.gen.c Processing file source/test/test-init.c Processing file source/test/regress_bufferevent.c Processing file source/test/test-fdleak.c Processing file source/test/regress_dns.c Processing file source/test/test-time.c Processing file source/test/regress_util.c Processing file source/test/regress.c Processing file source/test/tinytest.c Processing file source/test/regress_main.c Processing file source/test/test-weof.c Processing file source/test/regress_http.c Processing file source/test/regress_et.c Processing file source/test/regress_minheap.c Processing file source/test/regress_finalize.c Processing file source/test/regress_rpc.c Processing file source/test/test-dumpevents.c Processing file source/test/regress_buffer.c Processing file source/test/regress_zlib.c Processing file source/test/test-changelist.c Processing file source/test/regress_testutils.c Processing file /usr/include/libkern/i386/_OSByteOrder.h Processing file /usr/include/secure/_string.h Processing file /usr/include/sys/_structs.h Writing directory view page. Overall coverage rate: lines......: 14.5% (3415 of 23550 lines) functions..: 12.5% (202 of 1612 functions) |
Once lcov finished, all I had to do was open up html/index.html and see how much code the coverage tests executed. (And panic? In this case, 14.5% coverage seems pretty low!)
Here’s what the lcov summary looks like:
I reran the test/regress command to see if that would help, and it did push the coverage rate to 20%, but I need more insight into how the coverage tests are laid out, to see what else I can do. It is not clear how well the coverage tools work on multiplatform libraries like libevent, which have configurably-included backend code that may or may not run on the platform under test. In these cases, entire sections of code can be safely ignored. But it is unclear that code coverage tools in general are going to be aware of the preprocessor conditions that were used to build a piece of software (nor would I trust most coverage tools to be able to apply those rules to a piece of source code, especially if that code is written in C++).
In any case, like I said in a previous entry, coverage ultimately is not proof of correct behavior, but it is a good start to see what parts of your code may need more attention for quality assurance tests.