Hello Neels Hofmeyr, Harald Welte, Jenkins Builder, I'd like you to reexamine a change. Please visit
https://gerrit.osmocom.org/7601 to look at the new patch set (#6). support ccache to speed up ttcn3 test compilation By default, if ccache is found in PATH when regen-makefile.sh runs, use cached compiled C++ object files during the build (or populate the cache in case an object is not cached yet). Objects are cached in ccache's default cache directory (~/.ccache) unless overriden by the CCACHE_DIR environment variable. The USE_CCACHE environment variable overrides auto-detection in regen-makefile.sh and disables (USE_CCACHE=0) or enables (USE_CCACHE=1) use of ccache. Rebuilding the test suite from scratch with a populated cache is an order of magnitude faster than doing an initial build. ccache only speeds up C++ (.cc) to object file (.o) compilation. Linking object (.o) files into shared object (.so) files is still performed by the real compiler during every build. There is a small downside: We need to tweak the generated C++ files to prevent cache misses due to timestamps inserted into the code by ttcn3_compiler. This is done during 'make compile' if ccache is used. This mechanism could break in case the output of ttcn3_makefilegen changes. However, the only consequence would be a slow build due to cache misses. In the long term, an upstream solution built into eclipse titan (e.g. an option to disable time stamps in generated files) would be diserable. Another thing to note is that builds using objects from ccache will contain timestamp strings from the build which populated the cache. ccache provides a way to repopulate the cache if needed (set CCACHE_RECACHE=1 in the environment). These tradeoffs seem acceptable to me since in return we can achieve much faster turnaround, both during local test development and during test builds on jenkins. Change-Id: Ibb538f602206535c06980f88191c1dabe3c4cd82 --- M regen-makefile.sh A strip-datetime-comments.sed 2 files changed, 63 insertions(+), 0 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-ttcn3-hacks refs/changes/01/7601/6 diff --git a/regen-makefile.sh b/regen-makefile.sh index 57eebff..cd7a589 100755 --- a/regen-makefile.sh +++ b/regen-makefile.sh @@ -12,6 +12,16 @@ test -x "$(which ttcn3_makefilegen 2>/dev/null)" || { echo "ERROR: ttcn3_makefilegen not in PATH"; exit 1; } +# Enable ccache if it can be found in path. +# This speeds up repeated builds of the TTCN3 tests by an order of magnitude +# since most of the generated C++ source files don't change very often. +# Roughly, for an initial build which takes N minutes, a complete rebuild +# after 'make clean' will only take N seconds with ccache. +# Note that ccache cannot speed up compilation of .o files to .so files. +if [ -z "$USE_CCACHE" ] && which ccache 2>/dev/null; then + USE_CCACHE=1 +fi + ttcn3_makefilegen -p -l -f $* sed -i -e 's/# TTCN3_DIR = /TTCN3_DIR = \/usr/' Makefile sed -i -e 's/LDFLAGS = /LDFLAGS = -L \/usr\/lib\/titan /' Makefile @@ -32,3 +42,11 @@ sed -i -e 's/TTCN3_DIR = $/TTCN3_DIR = \/usr/' Makefile fi sed -i -e 's/\/bin\/compiler/\/bin\/ttcn3_compiler/' Makefile + +if [ "x$USE_CCACHE" = "x1" ]; then + # enable ccache + sed -i -e 's/^CXX = g++ $/CXX = env CCACHE_SLOPPINESS=time_macros ccache g++/' Makefile + # The Makefile must tweak generated C++ files during the 'make compile' step. + # See the strip-datetime-comments sed script for details. + sed -i -f ../strip-datetime-comments.sed Makefile +fi diff --git a/strip-datetime-comments.sed b/strip-datetime-comments.sed new file mode 100644 index 0000000..8089b8e --- /dev/null +++ b/strip-datetime-comments.sed @@ -0,0 +1,45 @@ +#!/usr/bin/sed +# (C) 2018 by sysmocom s.f.m.c. GmbH <i...@sysmocom.de> +# Author: Stefan Sperling <ssperl...@sysmocom.de> +# All rights reserved. +# Released under the terms of GNU General Public License, Version 2 or +# (at your option) any later version. + +# This sed script edits a Makefile generated by ttcn3_makefilegen. +# We insert a sed command into the 'compile:' target in order to +# remove timestamp comments from generated C++ files. + +# The ttcn3_compiler will generate C++ comments with a timestamp, such as: +# // for Stefan Sperling (stsp@fintan) on Sat Mar 31 14:34:30 2018 +# We must remove such comments since they cause cache misses with ccache. +# +# A related problem is the use of __DATE__ and __TIME__ in generated C++ code, +# which can be worked around by setting "CCACHE_SLOPPINESS=time_macros". This +# workaround implies that we must get our cache hits from ccache's "direct" +# cache, since ccache's "preprocessor" caching method won't work. +# +# FIXME: This should really be addressed in ttcn3_compiler itself! +# It should provide an option to suppress use of timestamps in generated code. + +# The generated Makefile's compile target looks like this: +# +# compile: $(TTCN3_MODULES) $(PREPROCESSED_TTCN3_MODULES) $(ASN1_MODULES) +# $(TTCN3_DIR)/bin/ttcn3_compiler $(COMPILER_FLAGS) \ +# $(TTCN3_MODULES) $(PREPROCESSED_TTCN3_MODULES) $(ASN1_MODULES) - $? +# touch $@ +# +# Before the final 'touch $@' step, we now insert a `sed` command that strips +# offending C++ comments from the generated code. + +# match the line in the generated Makefile which starts with 'compile:' +/^compile:/ { + # go to the next line + n; + # go to the next line + n; + # go to the next line + n; + # on this line, try to match 'touch $@', and if there is a match preprend a + # sed command which deletes the offending comments from generated C++ files: + s!touch $@!sed -i -e '/\\/\\/ for .* on .*/d' $(GENERATED_HEADERS) $(GENERATED_SOURCES) \&\& &! +} -- To view, visit https://gerrit.osmocom.org/7601 To unsubscribe, visit https://gerrit.osmocom.org/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibb538f602206535c06980f88191c1dabe3c4cd82 Gerrit-PatchSet: 6 Gerrit-Project: osmo-ttcn3-hacks Gerrit-Branch: master Gerrit-Owner: Stefan Sperling <ssperl...@sysmocom.de> Gerrit-Reviewer: Harald Welte <lafo...@gnumonks.org> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: Neels Hofmeyr <nhofm...@sysmocom.de> Gerrit-Reviewer: Stefan Sperling <ssperl...@sysmocom.de>