LCOV Code Coverage

TDF LibreOffice Document Liberation Project Community Blogs Weblate Nextcloud Redmine Ask LibreOffice Donate

This short tutorial aims to describe how to generate lcov reports.

Example results

Current results can be found here:

http://lcov.libreoffice.org

Details on how these are generated here: https://lists.freedesktop.org/archives/libreoffice/2015-June/068679.html

Historic results can be found here:

https://dev-builds.libreoffice.org/lcov_reports/

Pre-requisites

Set up build environment

Follow the guidelines at : How to Build to set up your LibreOffice build environment. This guide assumes you have the libreoffice source code in '/usr/local/src/libreoffice'.

Install lcov

You need to use lcov 1.10 or higher. http://ltp.sourceforge.net/coverage/lcov.php

patch 'geninfo'

You may need to apply the following patch to 'geninfo' : rhbz#829514

If you're on openSUSE >=12.2, just make sure you have a fully updated system, and lcov will just work. If you're using the latest distribution supplied lcov package on Fedora >=17, this patch may not be required anymore either.

Create lcovrc file

Create an ~/.lcovrc file in your home directory and add the following content :

geninfo_auto_base = 1

This option must be set to prevent errors like :

ERROR: could not read source file /home/user/project/sub-dir1/subdir2/subdir1/subdir2/file.c

This option is required when using lcov on projects built with libtool or similar build environments that work with multiple base directories, i.e. environments, where the current working directories when invoking the compiler are not the same directories in which the source code file is located. This option specifies that geninfo should try to automatically determine the base-directories when collecting coverage data.

Clean up from previous run

This step only needs to be performed if you already generated the reports at some earlier time. If this is the first time you generate them, this step can safely be skipped.

Remove the tracefiles from the previous run, reset the counters, and make clean.

rm -f /tmp/libreoffice_base.info /tmp/libreoffice_test.info /tmp/libreoffice_filtered.info /tmp/libreoffice_total.info
lcov --zerocounters --directory /usr/local/src/libreoffice
make distclean
./autogen.sh

Set appropriate FLAGS set and configure

You need to specify additional LDFLAGS and CFLAGS to instrument the build properly. You also need to specify some flags for the ./configure script.

Because chances are that you will not have all required prerequisite libraries installed on your system, the basic idea is to use as much of the 'internal' libreoffice libraries (included in the libreoffice source tree) as possible. You can do this by using '--without-system-libs' and '--without-system-headers'.

The build has an online update mechanism that spawns threads that dont properly join again before exit, which is a notorious problem with the tests (that often run quickly enough to terminate soffice.bin while the online update check is still in progress). In order to prevent this, you can run ./configure with '--disable-online-update'.

LDFLAGS+='-fprofile-arcs' CFLAGS+='-fprofile-arcs -ftest-coverage' \
CXXFLAGS+='-fprofile-arcs -ftest-coverage' CPPFLAGS+='-fprofile-arcs -ftest-coverage' \
./configure --disable-online-update --without-system-libs --without-system-headers

Build LibreOffice

The basic idea is to initially build the project only without running any tests (as much as possible.)

make

Run initial/baseline lcov

Now you need to create an lcov "baseline" before running any tests. The result is a coverage data file that contains zero coverage for every instrumented line of the project. At a later stage, you will combine this data file with coverage data files captured after the test run. This way the percentage of total lines covered will always be correct, even when not all source code files were loaded during the test(s).

lcov --no-external --capture --initial --directory /usr/local/src/libreoffice --output-file /tmp/libreoffice_base.info

Run tests

Now start the tests. The default behaviour is to stop the testing entirely if a single test fails, so that you can troubleshoot the failure. In order to let the tests continue even if one fails, you have to add '-k' to the 'make' command. This instructs it to keep going as much as possible after an error occurs. This is done in order to get as accurate as possible an picture of the overall code coverage of the tests. ('make check' will run 'slowcheck', 'unitcheck', and 'subsequentcheck' as well.)

make -k check

Run lcov again after tests/checks complete

lcov --no-external --capture --directory /usr/local/src/libreoffice --output-file /tmp/libreoffice_test.info

You may get some warnings about no data being found for /usr/include/*.h stuff. As we are not interested in that, this should not be a problem.

Combine lcov tracefiles

Now combine the 'before tests' and 'after tests' snapshots.

lcov --add-tracefile /tmp/libreoffice_base.info --add-tracefile /tmp/libreoffice_test.info \ 
--output-file /tmp/libreoffice_total.info

Remove / filter out remaining unwanted stuff from tracefile

Now we can remove the (remaining) stuff that we aren't interested in from the tracefile.

lcov --remove /tmp/libreoffice_total.info '/usr/include/*' '/usr/lib/*' '/usr/local/src/libreoffice/*/UnpackedTarball/*' \
'/usr/local/src/libreoffice/workdir/*' '/usr/local/src/libreoffice/instdir/*' '/usr/local/src/libreoffice/external/*' \
-o /tmp/libreoffice_filtered.info

Generate HTML reports.

Create a storage directory

Create an empty directory to store the reports.

rm -rf /tmp/libreoffice-lcov/
mkdir /tmp/libreoffice-lcov/

Retrieve software version

It is a good practice to include the version of the software you run the report on somewhere in the report. One way to do this, is to use the '--title' option with the genhtml command. If you used a git version, you could add the commit SHA1 that way. You can retreive this by using the 'git log' command on the directory where you cloned/located the libreoffice source. The first line in the output is the commit SHA1.

cd /usr/local/src/libreoffice
git log | head

Create reports

Now generate the reports. Remember to include the git commit SHA1.

genhtml --prefix /usr/local/src/libreoffice --ignore-errors source /tmp/libreoffice_filtered.info \
--legend --title "commit SHA1" --output-directory=/tmp/libreoffice-lcov

Done.

You should now have the generated HTML report files. See Development/RegressionHotspots, ~~tdf#66750~~ and tdf#70448 if they are too red for your taste.