libevent, gcov, lcov, and OS X

Getting a sense of code coverage for open-source projects is necessary due-diligence for any dependent project. On OS X, it's also a little more work. While doing some of my own research, I wanted to see how well tested the libevent library was, but wasn't finding much info (broken link) online. So here's a log of what I needed to do, to get this information on a Mac.

First things first (and this should apply for many more open-source projects), after I checked out the latest code from github, I added an option to the configure.ac Autoconf source file to insert the necessary profiling code calls for code coverage:

AC_ARG_ENABLE(coverage,
              [  --enable-coverage       Enable coverage testing],
              [CFLAGS="$CFLAGS -fprofile-arcs -ftest-coverage"])

With that added, I reran the autogen.sh file, which pulls in the configure.ac file, and regenerates the configure script, and then I ran ./configure --enable-coverage.

Then I ran make and specified clang as the C compiler instead of old-school gcc. Besides better code quality and error reports, only clang will generate the coverage code. The Apple version of gcc did not do so, leading to some initial confusion.

make CC=clang

Once the build was complete, I ran the regression tests with:

make verify

Unfortunately, the following error occurred:

make  check-am
make  check-TESTS
Running tests:
EPOLL (timerfd)
./test/test-script.sh: line 66: 62222 Trace/BPT trap: 5       $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE"
Skipping test
EPOLL (changelist)
./test/test-script.sh: line 66: 62237 Trace/BPT trap: 5       $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE"
Skipping test
EPOLL (timerfd+changelist)
./test/test-script.sh: line 66: 62251 Trace/BPT trap: 5       $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE"
Skipping test
EVPORT
./test/test-script.sh: line 66: 62265 Trace/BPT trap: 5       $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE"
Skipping test
KQUEUE
./test/test-script.sh: line 66: 62279 Trace/BPT trap: 5       $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE"
Skipping test
EPOLL
./test/test-script.sh: line 66: 62293 Trace/BPT trap: 5       $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE"
Skipping test
DEVPOLL
./test/test-script.sh: line 66: 62307 Trace/BPT trap: 5       $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE"
Skipping test
POLL
./test/test-script.sh: line 66: 62321 Trace/BPT trap: 5       $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE"
Skipping test
SELECT
./test/test-script.sh: line 66: 62335 Trace/BPT trap: 5       $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE"
Skipping test
WIN32
./test/test-script.sh: line 66: 62349 Trace/BPT trap: 5       $TEST_DIR/test-init 2>> "$TEST_OUTPUT_FILE"
Skipping test
PASS: test/test-script.sh
==================
All 1 tests passed
==================

So I tried running tests/regress directly and saw:

$ test/regress
main/methods: [forking] dyld: lazy symbol binding failed: Symbol not found: _llvm_gcda_start_file
  Referenced from: LibeventLibrary/source/.libs/libevent_openssl-2.1.3.dylib
  Expected in: flat namespace

dyld: Symbol not found: _llvm_gcda_start_file
  Referenced from: LibeventLibrary/source/.libs/libevent_openssl-2.1.3.dylib
  Expected in: flat namespace

OK
main/version: OK
main/base_features: [forking] dyld: lazy symbol binding failed: Symbol not found: _llvm_gcda_start_file
  Referenced from: LibeventLibrary/source/.libs/libevent_openssl-2.1.3.dylib
  Expected in: flat namespace

dyld: Symbol not found: _llvm_gcda_start_file
  Referenced from: LibeventLibrary/source/.libs/libevent_openssl-2.1.3.dylib
  Expected in: flat namespace

OK

Oops.

Turns out that Apple uses the profile_rt library to handle code coverage (broken link) instead of the former gcov library, which is why the _llvm_gcda_start_file function symbol is missing. So I linked to the libprofile_rt.dylib library by specifying LDFLAGS=-lprofile_rt on the make command line:

make CC=clang LDFLAGS=-lprofile_rt

Rerunning make verify, the following was output, which indicated which of the event notification subsystems were available and being tested on the system:

make  check-am
make  check-TESTS
Running tests:
EPOLL (timerfd)
Skipping test
EPOLL (changelist)
Skipping test
EPOLL (timerfd+changelist)
Skipping test
EVPORT
Skipping test
KQUEUE
 test-eof: OKAY
 test-weof: OKAY
 test-time: OKAY
 test-changelist: OKAY
 test-fdleak: OKAY
 test-dumpevents: OKAY
 regress:
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818fOKAY
EPOLL
Skipping test
DEVPOLL
Skipping test
POLL
 test-eof: OKAY
 test-weof: OKAY
 test-time: OKAY
 test-changelist: OKAY
 test-fdleak: OKAY
 test-dumpevents: OKAY
 regress:
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818fOKAY
SELECT
 test-eof: OKAY
 test-weof: OKAY
 test-time: OKAY
 test-changelist: OKAY
 test-fdleak: OKAY
 test-dumpevents: OKAY
 regress:
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818f
  WARN test/regress_ssl.c:162: Version mismatch for openssl: compiled with 90812f but running with 90818fOKAY
WIN32
Skipping test
PASS: test/test-script.sh
==================
All 1 tests passed
==================

Once the regression tests finished, the coverage data became available in the form of *.gcno and *.gcda files in the test/ folder and in the .libs/ folder.

Running lcov generated easier-to-interpret HTML files:

$ ~/Downloads/lcov-1.10/bin/lcov --capture --directory . --output-file coverage.info
Capturing coverage data from .
Found gcov version: 4.2.1
Scanning . for .gcda files ...
Found 49 data files in .
Processing .libs/buffer.gcda
Processing .libs/bufferevent.gcda
Processing .libs/bufferevent_filter.gcda
Processing .libs/bufferevent_pair.gcda
Processing .libs/bufferevent_ratelim.gcda
Processing .libs/bufferevent_sock.gcda
Processing .libs/evdns.gcda
Processing .libs/event.gcda
Processing .libs/event_tagging.gcda
Processing .libs/evmap.gcda
Processing .libs/evrpc.gcda
Processing .libs/evthread.gcda
Processing .libs/evthread_pthread.gcda
Processing .libs/evutil.gcda
Processing .libs/evutil_rand.gcda
Processing .libs/evutil_time.gcda
Processing .libs/http.gcda
Processing .libs/kqueue.gcda
Processing .libs/libevent_openssl_la-bufferevent_openssl.gcda
Processing .libs/listener.gcda
Processing .libs/log.gcda
Processing .libs/poll.gcda
Processing .libs/select.gcda
Processing .libs/signal.gcda
Processing test/test-changelist.gcda
Processing test/test-dumpevents.gcda
Processing test/test-eof.gcda
Processing test/test-fdleak.gcda
Processing test/test-init.gcda
Processing test/test-time.gcda
Processing test/test-weof.gcda
Processing test/test_regress-regress.gcda
Processing test/test_regress-regress.gen.gcda
Processing test/test_regress-regress_buffer.gcda
Processing test/test_regress-regress_bufferevent.gcda
Processing test/test_regress-regress_dns.gcda
Processing test/test_regress-regress_et.gcda
Processing test/test_regress-regress_finalize.gcda
Processing test/test_regress-regress_http.gcda
Processing test/test_regress-regress_listener.gcda
Processing test/test_regress-regress_main.gcda
Processing test/test_regress-regress_minheap.gcda
Processing test/test_regress-regress_rpc.gcda
Processing test/test_regress-regress_ssl.gcda
Processing test/test_regress-regress_testutils.gcda
Processing test/test_regress-regress_thread.gcda
Processing test/test_regress-regress_util.gcda
Processing test/test_regress-regress_zlib.gcda
Processing test/test_regress-tinytest.gcda
Finished .info-file creation
$ ~/Downloads/lcov-1.10/bin/genhtml coverage.info --output-directory html
Reading data file coverage.info
Found 54 entries.
Found common filename prefix "LibeventLibrary"
Writing .css and .png files.
Generating output.
Processing file source/bufferevent.c
Processing file source/kqueue.c
Processing file source/evmap.c
Processing file source/http.c
Processing file source/evthread-internal.h
Processing file source/evthread_pthread.c
Processing file source/select.c
Processing file source/bufferevent_openssl.c
Processing file source/buffer.c
Processing file source/bufferevent_ratelim.c
Processing file source/bufferevent_pair.c
Processing file source/poll.c
Processing file source/evutil.c
Processing file source/evutil_rand.c
Processing file source/evthread.c
Processing file source/evutil_time.c
Processing file source/evdns.c
Processing file source/signal.c
Processing file source/evrpc.c
Processing file source/log.c
Processing file source/bufferevent_sock.c
Processing file source/event_tagging.c
Processing file source/listener.c
Processing file source/minheap-internal.h
Processing file source/event.c
Processing file source/bufferevent_filter.c
Processing file source/test/regress_ssl.c
Processing file source/test/regress_listener.c
Processing file source/test/regress_thread.c
Processing file source/test/test-eof.c
Processing file source/test/regress.gen.c
Processing file source/test/test-init.c
Processing file source/test/regress_bufferevent.c
Processing file source/test/test-fdleak.c
Processing file source/test/regress_dns.c
Processing file source/test/test-time.c
Processing file source/test/regress_util.c
Processing file source/test/regress.c
Processing file source/test/tinytest.c
Processing file source/test/regress_main.c
Processing file source/test/test-weof.c
Processing file source/test/regress_http.c
Processing file source/test/regress_et.c
Processing file source/test/regress_minheap.c
Processing file source/test/regress_finalize.c
Processing file source/test/regress_rpc.c
Processing file source/test/test-dumpevents.c
Processing file source/test/regress_buffer.c
Processing file source/test/regress_zlib.c
Processing file source/test/test-changelist.c
Processing file source/test/regress_testutils.c
Processing file /usr/include/libkern/i386/_OSByteOrder.h
Processing file /usr/include/secure/_string.h
Processing file /usr/include/sys/_structs.h
Writing directory view page.
Overall coverage rate:
  lines......: 14.5% (3415 of 23550 lines)
  functions..: 12.5% (202 of 1612 functions)

Once lcov finished, all I had to do was open up html/index.html and see how much code the coverage tests executed. (And panic? In this case, 14.5% coverage seems pretty low!)

Here's what the lcov summary looks like:

lcov-libevent

I reran the test/regress command to see if that would help, and it did push the coverage rate to 20%, but I need more insight into how the coverage tests are laid out, to see what else I can do. It is not clear how well the coverage tools work on multiplatform libraries like libevent, which have configurably-included backend code that may or may not run on the platform under test. In these cases, entire sections of code can be safely ignored. But it is unclear that code coverage tools in general are going to be aware of the preprocessor conditions that were used to build a piece of software (nor would I trust most coverage tools to be able to apply those rules to a piece of source code, especially if that code is written in C++).

In any case, like I said in a previous entry, coverage ultimately is not proof of correct behavior, but it is a good start to see what parts of your code may need more attention for quality assurance tests.