Re: make profile issue on Git 2.1.0
Maybe we should do this: Looks good to me. As a side note, while testing this I noticed that the make perf run goes a lot faster if you set GIT_PERF_REPEAT_COUNT=1. This is bad for actually measuring things, but probably fine for profile feedback. I don't use the profile builds myself, though, so I will leave it to somebody who cares more to investigate whether such a change would be a good idea. Yes should be fine too. Another way to speed it up would be also to run the tests (both tests and benchmarks) in parallel on multiple cores. The gcc feedback mechanism won't mind, as it doesn't measure time. -Andi -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Profile feedback patchkit v2
Fix the bitrotted profile feedback support. Changes to v1: - Remove obsolete comment - Remove controversal diff script. -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/4] Use BASIC_FLAGS for profile feedback
From: Andi Kleen a...@linux.intel.com Use BASIC_CFLAGS instead of CFLAGS to set up the profile feedback option in the Makefile. This allows still overriding CFLAGS on the make command line without disabling profile feedback. Signed-off-by: Andi Kleen a...@linux.intel.com --- Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 07ea105..a9770ac 100644 --- a/Makefile +++ b/Makefile @@ -1552,13 +1552,13 @@ endif PROFILE_DIR := $(CURDIR) ifeq ($(PROFILE),GEN) - CFLAGS += -fprofile-generate=$(PROFILE_DIR) -DNO_NORETURN=1 + BASIC_CFLAGS += -fprofile-generate=$(PROFILE_DIR) -DNO_NORETURN=1 EXTLIBS += -lgcov export CCACHE_DISABLE = t V = 1 else ifneq ($(PROFILE),) - CFLAGS += -fprofile-use=$(PROFILE_DIR) -fprofile-correction -DNO_NORETURN=1 + BASIC_CFLAGS += -fprofile-use=$(PROFILE_DIR) -fprofile-correction -DNO_NORETURN=1 export CCACHE_DISABLE = t V = 1 endif -- 2.0.1 -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 4/4] Fix profile feedback with -jN and add profile-fast
From: Andi Kleen a...@linux.intel.com Profile feedback always failed for me with -jN. The problem was that there was no implicit ordering between the profile generate stage and the profile use stage. So some objects in the later stage would be linked with profile generate objects, and fail due to the missing -lgcov. This adds a new profile target that implicitely enforces the correct ordering by using submakes. Plus a profile-install target to also install. This is also nicer to type that PROFILE=... Plus I always run the performance test suite now for the full profile run. In addition I also added a profile-fast / profile-fast-install target the only runs the performance test suite instead of the whole test suite. This significantly speeds up the profile build, which was totally dominated by test suite run time. However it may have less coverage of course. Signed-off-by: Andi Kleen a...@linux.intel.com --- INSTALL | 14 -- Makefile | 21 + 2 files changed, 29 insertions(+), 6 deletions(-) diff --git a/INSTALL b/INSTALL index ba01e74..6ec7a24 100644 --- a/INSTALL +++ b/INSTALL @@ -28,7 +28,7 @@ set up install paths (via config.mak.autogen), so you can write instead If you're willing to trade off (much) longer build time for a later faster git you can also do a profile feedback build with - $ make prefix=/usr PROFILE=BUILD all + $ make prefix=/usr profile # make prefix=/usr PROFILE=BUILD install This will run the complete test suite as training workload and then @@ -36,10 +36,20 @@ rebuild git with the generated profile feedback. This results in a git which is a few percent faster on CPU intensive workloads. This may be a good tradeoff for distribution packagers. +Alternatively you can run profile feedback only with the git benchmark +suite. This runs significantly faster than the full test suite, but +has less coverage: + + $ make prefix=/usr profile-fast + # make prefix=/usr PROFILE=BUILD install + Or if you just want to install a profile-optimized version of git into your home directory, you could run: - $ make PROFILE=BUILD install + $ make profile-install + +or + $ make profile-fast-install As a caveat: a profile-optimized build takes a *lot* longer since the git tree must be built twice, and in order for the profiling diff --git a/Makefile b/Makefile index ba64be9..a760402 100644 --- a/Makefile +++ b/Makefile @@ -1643,13 +1643,20 @@ SHELL = $(SHELL_PATH) all:: shell_compatibility_test ifeq $(PROFILE) BUILD -ifeq ($(filter all,$(MAKECMDGOALS)),all) -all:: profile-clean +all:: profile +endif + +profile:: profile-clean $(MAKE) PROFILE=GEN all $(MAKE) PROFILE=GEN -j1 test $(MAKE) PROFILE=GEN -j1 perf -endif -endif + $(MAKE) PROFILE=USE all + +profile-fast: profile-clean + $(MAKE) PROFILE=GEN all + $(MAKE) PROFILE=GEN -j1 perf + $(MAKE) PROFILE=USE all + all:: $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS) GIT-BUILD-OPTIONS ifneq (,$X) @@ -2336,6 +2343,12 @@ mergetools_instdir_SQ = $(subst ','\'',$(mergetools_instdir)) install_bindir_programs := $(patsubst %,%$X,$(BINDIR_PROGRAMS_NEED_X)) $(BINDIR_PROGRAMS_NO_X) +profile-install: profile + $(MAKE) install + +profile-fast-install: profile-fast + $(MAKE) install + install: all $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(bindir_SQ)' $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(gitexec_instdir_SQ)' -- 2.0.1 -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 3/4] Run the perf test suite for profile feedback too
From: Andi Kleen a...@linux.intel.com Signed-off-by: Andi Kleen a...@linux.intel.com --- Makefile | 1 + 1 file changed, 1 insertion(+) diff --git a/Makefile b/Makefile index a9770ac..ba64be9 100644 --- a/Makefile +++ b/Makefile @@ -1647,6 +1647,7 @@ ifeq ($(filter all,$(MAKECMDGOALS)),all) all:: profile-clean $(MAKE) PROFILE=GEN all $(MAKE) PROFILE=GEN -j1 test + $(MAKE) PROFILE=GEN -j1 perf endif endif -- 2.0.1 -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/4] Don't define away __attribute__ on gcc
From: Andi Kleen a...@linux.intel.com Profile feedback sets -DNO_NORETURN, which causes the compat header file to go into a default #else block. That #else block defines away __attribute__(). Doing so causes all kinds of problems with the Linux and gcc system headers: in particular it makes the xmmintrin.h headers error out, breaking the build. Don't define away __attribute__ when __GNUC__ is set. Signed-off-by: Andi Kleen a...@linux.intel.com --- git-compat-util.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/git-compat-util.h b/git-compat-util.h index 96f5554..01e8695 100644 --- a/git-compat-util.h +++ b/git-compat-util.h @@ -291,10 +291,12 @@ extern char *gitbasename(char *); #else #define NORETURN #define NORETURN_PTR +#ifndef __GNUC__ #ifndef __attribute__ #define __attribute__(x) #endif #endif +#endif /* The sentinel attribute is valid from gcc version 4.0 */ #if defined(__GNUC__) (__GNUC__ = 4) -- 2.0.1 -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 3/5] Run the perf test suite for profile feedback too
On Mon, Jul 07, 2014 at 02:06:57PM -0700, Junio C Hamano wrote: Andi Kleen a...@firstfloor.org writes: From: Andi Kleen a...@linux.intel.com Open: If the perf test suite is representative enough it may be reasonable to only run that and skip the much longer full test suite. Thoughts? I do not think it right now is representative, nor it was meant to become so. The operations are those that people cared about and tuned, and hopefully it would cover stuff actual end users care about in the real life, though. I ended up answering the question by creating two separate makefile targets in the next patch. profile to run the full test suite and profile-fast to run only the performance test. profile-fast as the name implies is a lot faster to build, and fast enough that it's not annoying. I'll remove the Open -Andi -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 5/5] Add a little script to compare two make perf runs
a justification why the geometric mean is used here would increase my confident significantly. It's just a standard way to summarize sets of benchmarks. For example SPEC uses the same approach. Anyways the script is not essential to the rest of the profile feedback feature. Just ignore it if it's controversal. -Andi -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/5] Use BASIC_FLAGS for profile feedback
From: Andi Kleen a...@linux.intel.com Use BASIC_CFLAGS instead of CFLAGS to set up the profile feedback option in the Makefile. This allows still overriding CFLAGS on the make command line without disabling profile feedback. Signed-off-by: Andi Kleen a...@linux.intel.com --- Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 07ea105..a9770ac 100644 --- a/Makefile +++ b/Makefile @@ -1552,13 +1552,13 @@ endif PROFILE_DIR := $(CURDIR) ifeq ($(PROFILE),GEN) - CFLAGS += -fprofile-generate=$(PROFILE_DIR) -DNO_NORETURN=1 + BASIC_CFLAGS += -fprofile-generate=$(PROFILE_DIR) -DNO_NORETURN=1 EXTLIBS += -lgcov export CCACHE_DISABLE = t V = 1 else ifneq ($(PROFILE),) - CFLAGS += -fprofile-use=$(PROFILE_DIR) -fprofile-correction -DNO_NORETURN=1 + BASIC_CFLAGS += -fprofile-use=$(PROFILE_DIR) -fprofile-correction -DNO_NORETURN=1 export CCACHE_DISABLE = t V = 1 endif -- 2.0.1 -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/5] Don't define away __attribute__ on gcc
From: Andi Kleen a...@linux.intel.com Profile feedback sets -DNO_NORETURN, which causes the compat header file to go into a default #else block. That #else block defines away __attribute__(). Doing so causes all kinds of problems with the Linux and gcc system headers: in particular it makes the xmmintrin.h headers error out, breaking the build. Don't define away __attribute__ when __GNUC__ is set. Signed-off-by: Andi Kleen a...@linux.intel.com --- git-compat-util.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/git-compat-util.h b/git-compat-util.h index 96f5554..01e8695 100644 --- a/git-compat-util.h +++ b/git-compat-util.h @@ -291,10 +291,12 @@ extern char *gitbasename(char *); #else #define NORETURN #define NORETURN_PTR +#ifndef __GNUC__ #ifndef __attribute__ #define __attribute__(x) #endif #endif +#endif /* The sentinel attribute is valid from gcc version 4.0 */ #if defined(__GNUC__) (__GNUC__ = 4) -- 2.0.1 -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 3/5] Run the perf test suite for profile feedback too
From: Andi Kleen a...@linux.intel.com Open: If the perf test suite is representative enough it may be reasonable to only run that and skip the much longer full test suite. Thoughts? Signed-off-by: Andi Kleen a...@linux.intel.com --- Makefile | 1 + 1 file changed, 1 insertion(+) diff --git a/Makefile b/Makefile index a9770ac..ba64be9 100644 --- a/Makefile +++ b/Makefile @@ -1647,6 +1647,7 @@ ifeq ($(filter all,$(MAKECMDGOALS)),all) all:: profile-clean $(MAKE) PROFILE=GEN all $(MAKE) PROFILE=GEN -j1 test + $(MAKE) PROFILE=GEN -j1 perf endif endif -- 2.0.1 -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Fix the bitrotted profile feedback build
The profile feedback build support had bitrotted. This patchkit fixes it and improves it slightly by adding a new profile-fast target. -Andi -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 4/5] Fix profile feedback with -jN and add profile-fast
From: Andi Kleen a...@linux.intel.com Profile feedback always failed for me with -jN. The problem was that there was no implicit ordering between the profile generate stage and the profile use stage. So some objects in the later stage would be linked with profile generate objects, and fail due to the missing -lgcov. This adds a new profile target that implicitely enforces the correct ordering by using submakes. Plus a profile-install target to also install. This is also nicer to type that PROFILE=... Plus I always run the performance test suite now for the full profile run. In addition I also added a profile-fast / profile-fast-install target the only runs the performance test suite instead of the whole test suite. This significantly speeds up the profile build, which was totally dominated by test suite run time. However it may have less coverage of course. Signed-off-by: Andi Kleen a...@linux.intel.com --- INSTALL | 14 -- Makefile | 21 + 2 files changed, 29 insertions(+), 6 deletions(-) diff --git a/INSTALL b/INSTALL index ba01e74..6ec7a24 100644 --- a/INSTALL +++ b/INSTALL @@ -28,7 +28,7 @@ set up install paths (via config.mak.autogen), so you can write instead If you're willing to trade off (much) longer build time for a later faster git you can also do a profile feedback build with - $ make prefix=/usr PROFILE=BUILD all + $ make prefix=/usr profile # make prefix=/usr PROFILE=BUILD install This will run the complete test suite as training workload and then @@ -36,10 +36,20 @@ rebuild git with the generated profile feedback. This results in a git which is a few percent faster on CPU intensive workloads. This may be a good tradeoff for distribution packagers. +Alternatively you can run profile feedback only with the git benchmark +suite. This runs significantly faster than the full test suite, but +has less coverage: + + $ make prefix=/usr profile-fast + # make prefix=/usr PROFILE=BUILD install + Or if you just want to install a profile-optimized version of git into your home directory, you could run: - $ make PROFILE=BUILD install + $ make profile-install + +or + $ make profile-fast-install As a caveat: a profile-optimized build takes a *lot* longer since the git tree must be built twice, and in order for the profiling diff --git a/Makefile b/Makefile index ba64be9..a760402 100644 --- a/Makefile +++ b/Makefile @@ -1643,13 +1643,20 @@ SHELL = $(SHELL_PATH) all:: shell_compatibility_test ifeq $(PROFILE) BUILD -ifeq ($(filter all,$(MAKECMDGOALS)),all) -all:: profile-clean +all:: profile +endif + +profile:: profile-clean $(MAKE) PROFILE=GEN all $(MAKE) PROFILE=GEN -j1 test $(MAKE) PROFILE=GEN -j1 perf -endif -endif + $(MAKE) PROFILE=USE all + +profile-fast: profile-clean + $(MAKE) PROFILE=GEN all + $(MAKE) PROFILE=GEN -j1 perf + $(MAKE) PROFILE=USE all + all:: $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS) GIT-BUILD-OPTIONS ifneq (,$X) @@ -2336,6 +2343,12 @@ mergetools_instdir_SQ = $(subst ','\'',$(mergetools_instdir)) install_bindir_programs := $(patsubst %,%$X,$(BINDIR_PROGRAMS_NEED_X)) $(BINDIR_PROGRAMS_NO_X) +profile-install: profile + $(MAKE) install + +profile-fast-install: profile-fast + $(MAKE) install + install: all $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(bindir_SQ)' $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(gitexec_instdir_SQ)' -- 2.0.1 -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 5/5] Add a little script to compare two make perf runs
From: Andi Kleen a...@linux.intel.com
Signed-off-by: Andi Kleen a...@linux.intel.com
---
diff-res | 26 ++
1 file changed, 26 insertions(+)
create mode 100755 diff-res
diff --git a/diff-res b/diff-res
new file mode 100755
index 000..90d57be
--- /dev/null
+++ b/diff-res
@@ -0,0 +1,26 @@
+#!/usr/bin/python
+# compare two make perf output file
+# this should be the results only without any header
+import argparse
+import math, operator
+from collections import OrderedDict
+
+ap = argparse.ArgumentParser()
+ap.add_argument('file1', type=argparse.FileType('r'))
+ap.add_argument('file2', type=argparse.FileType('r'))
+args = ap.parse_args()
+
+cmp = (OrderedDict(), OrderedDict())
+for f, k in zip((args.file1, args.file2), cmp):
+for j in f:
+num = j[59:63]
+name = j[:59]
+k[name] = float(num)
+
+for j in cmp[0].keys():
+print j, cmp[1][j] - cmp[0][j]
+
+def geomean(l):
+ return math.pow(reduce(operator.mul, filter(lambda x: x != 0.0, l)), 1.0 /
len(l))
+
+print geomean %.2f - %.2f % (geomean(cmp[0].values()),
geomean(cmp[1].values()))
--
2.0.1
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.12 hangs on boot
Can you please test if this patch fixes it?
-Andi
Don't compare linux processor index with APICID
Fixes boot up lockups on some machines where CPU apic ids
don't start with 0
Signed-off-by: Andi Kleen [EMAIL PROTECTED]
Index: linux/arch/x86_64/kernel/smpboot.c
===
--- linux.orig/arch/x86_64/kernel/smpboot.c
+++ linux/arch/x86_64/kernel/smpboot.c
@@ -211,7 +211,7 @@ static __cpuinit void sync_master(void *
{
unsigned long flags, i;
- if (smp_processor_id() != boot_cpu_id)
+ if (smp_processor_id() != 0)
return;
go[MASTER] = 0;
-
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
