Re: [PATCH] mingw: use OpenSSL's SHA-1 routines
Johannes Sixt writes: > Am 24.02.2017 um 22:54 schrieb Junio C Hamano: >> Johannes Sixt writes: >>> I'll use the patch for daily work for a while to see whether it hurts. >> >> Please ping this thread again when you have something to add. For >> now, I'll demote this patch from 'next' to 'pu' when we rewind and >> rebuild 'next' post 2.12 release. > > ... I've used the patch in > production for some time now and did not notice any slowdowns. Thanks. Dscho obviously thinks this is the right thing for Windows, and you agree. That's more than sufficient votes to make me feel safe ;-) Thanks.
Re: [PATCH] mingw: use OpenSSL's SHA-1 routines
Am 24.02.2017 um 22:54 schrieb Junio C Hamano: Johannes Sixt writes: I'll use the patch for daily work for a while to see whether it hurts. Please ping this thread again when you have something to add. For now, I'll demote this patch from 'next' to 'pu' when we rewind and rebuild 'next' post 2.12 release. As I already said in [1], I have no objection. I've used the patch in production for some time now and did not notice any slowdowns. [1] https://public-inbox.org/git/0080edd6-a515-2fe9-6266-b6f6bbedf...@kdbg.org/ -- Hannes
Re: [PATCH] mingw: use OpenSSL's SHA-1 routines
On Thu, Feb 9, 2017 at 9:02 PM, Jeff King wrote: > > I think this is only half the story. A heavy-sha1 workload is faster, > which is good. But one of the original reasons to prefer blk-sha1 (at > least on Linux) is that resolving libcrypto.so symbols takes a > non-trivial amount of time. I just timed it again, and it seems to be > consistently 1ms slower to run "git rev-parse --git-dir" on my machine > (from the top-level of a repo). Yes. It's also a horrible plain to profile those things. Avoiding openssl was a great thing, because it avoided a lot of crazy overhead. I suspect that most of the openssl win comes from using the actual SHA instructions on modern CPU's. Because last I looked, the hand-coded assembly simply wasn't that much faster. We could easily have some x86-specific library that just does "use SHA instructions if you have them, use blk-sha1 otherwise". Of course, if we end up using the collision checking SHA libraries this is all moot anyway. Linus
Re: [PATCH] mingw: use OpenSSL's SHA-1 routines
Johannes Sixt writes: > It can be argued that in normal interactive use, it is hard to notice > that another DLL is loaded. Don't forget, though, that on Windows it > is not only the pure time to resolve the entry points, but also that > typically virus scanners inspect every executable file that is loaded, > which adds another share of time. > > I'll use the patch for daily work for a while to see whether it hurts. Please ping this thread again when you have something to add. For now, I'll demote this patch from 'next' to 'pu' when we rewind and rebuild 'next' post 2.12 release. Thanks.
Re: [PATCH] mingw: use OpenSSL's SHA-1 routines
Am 13.02.2017 um 18:16 schrieb Johannes Schindelin: On Sat, 11 Feb 2017, Johannes Sixt wrote: Am 10.02.2017 um 00:41 schrieb Junio C Hamano: Johannes Schindelin writes: From: Jeff Hostetler Use OpenSSL's SHA-1 routines rather than builtin block-sha1 routines. This improves performance on SHA1 operations on Intel processors. ... Signed-off-by: Jeff Hostetler Signed-off-by: Johannes Schindelin --- Nice. Will queue as jh/mingw-openssl-sha1 topic; it is a bit too late for today's integration cycle to be merged to 'next', but let's have this by the end of the week in 'master'. Please don't rush this through. I didn't have a chance to cross-check the patch; it will have to wait for Monday. I would like to address Peff's concerns about additional runtime dependencies. I never meant this to be fast-tracked into git.git. We have all the time in our lives to get this in, as Git for Windows already carries this patch for a while, and shall continue to do so. I've been working with this patch for the past few days, and I did not notice any disadvantage during interactive work even though there is a new dependency on libcrypto.dll. Here are some unscientific numbers collected during test suite runs: bash -c "time make -j4 -k test" with this patch: real34m47.242s user9m55.827s sys 25m20.483s without this patch: real34m2.330s user9m56.556s sys 25m5.520s It looks like BLK_SHA1 has some advantage, but I would not count on these figures too much. (I certainly did not sit idly in front of the workstation during these tests, for example. That may have skewed the numbers somewhat.) (And, no, I'm not going to measure best-of-five timings, not even best-of-two. ;) In summary: Interactive response times do not decline noticably. I do not object the patch. -- Hannes
Re: [PATCH] mingw: use OpenSSL's SHA-1 routines
Am 13.02.2017 um 23:38 schrieb Johannes Schindelin: In addition, you build from a custom MINGW/MSys1 setup, correct? Correct. Specifically, I use the build tools from "msysgit" times, but build outside the premanufactured build environement; i.e., the "THIS_IS_MSYSGIT" section in config.mak.uname is not used. -- Hannes
Re: [PATCH] mingw: use OpenSSL's SHA-1 routines
Hi, On Mon, 13 Feb 2017, Johannes Sixt wrote: > Am 13.02.2017 um 20:42 schrieb Junio C Hamano: > > I have been operating under the assumption that everybody on Windows > > who builds Git works off of Dscho's Git for Windows tree, and patches > > that are specific to Windows from Dscho's are sent to me via the list > > only after they have been in Git for Windows and proven to help > > Windows users in the wild. > > > > The consequence of these two assumptions is that I would feel safe to > > treat Windows specific changes that do not touch generic part of the > > codebase from Dscho just like updates from any other subsystem > > maintainers (any git-svn thing from Eric, any gitk thing from Paul, > > any p4 thing Luke and Lars are both happy with, etc.). > > > > You seem to be saying that the first of the two assumptions does not > > hold. Should I change my expectations while queuing Windows specific > > patches from Dscho? > > Your first assumption is incorrect as far as I am concerned. I build > from your tree plus some topics. During -rc period, I build off of > master; after a release, I build off of next. I merge some of the topics > that you carry in pu when I find them interesting or when I suspect them > to regress on Windows. Then I carry around a few additional patches > that the public has never seen, and these days I also merge Dscho's > rebase-i topic. In addition, you build from a custom MINGW/MSys1 setup, correct? Ciao, Johannes
Re: [PATCH] mingw: use OpenSSL's SHA-1 routines
Am 13.02.2017 um 20:42 schrieb Junio C Hamano: I have been operating under the assumption that everybody on Windows who builds Git works off of Dscho's Git for Windows tree, and patches that are specific to Windows from Dscho's are sent to me via the list only after they have been in Git for Windows and proven to help Windows users in the wild. The consequence of these two assumptions is that I would feel safe to treat Windows specific changes that do not touch generic part of the codebase from Dscho just like updates from any other subsystem maintainers (any git-svn thing from Eric, any gitk thing from Paul, any p4 thing Luke and Lars are both happy with, etc.). You seem to be saying that the first of the two assumptions does not hold. Should I change my expectations while queuing Windows specific patches from Dscho? Your first assumption is incorrect as far as I am concerned. I build from your tree plus some topics. During -rc period, I build off of master; after a release, I build off of next. I merge some of the topics that you carry in pu when I find them interesting or when I suspect them to regress on Windows. Then I carry around a few additional patches that the public has never seen, and these days I also merge Dscho's rebase-i topic. -- Hannes
Re: [PATCH] mingw: use OpenSSL's SHA-1 routines
Johannes Sixt writes: > The patch does add a new runtime dependency on libcrypto.dll in my > environment. I would be surprised if it does not also with your modern > build tools. > > I haven't had time to compare test suite runtimes. > >> I'm open to the argument that it doesn't matter in practice for normal >> git users. But it's not _just_ scripting. It depends on the user's >> pattern of invoking git commands (and how expensive the symbol >> resolution is on their system). > > It can be argued that in normal interactive use, it is hard to notice > that another DLL is loaded. Don't forget, though, that on Windows it > is not only the pure time to resolve the entry points, but also that > typically virus scanners inspect every executable file that is loaded, > which adds another share of time. > > I'll use the patch for daily work for a while to see whether it hurts. Thanks. I need to ask an unrelated question at a bit higher level, though. I have been operating under the assumption that everybody on Windows who builds Git works off of Dscho's Git for Windows tree, and patches that are specific to Windows from Dscho's are sent to me via the list only after they have been in Git for Windows and proven to help Windows users in the wild. The consequence of these two assumptions is that I would feel safe to treat Windows specific changes that do not touch generic part of the codebase from Dscho just like updates from any other subsystem maintainers (any git-svn thing from Eric, any gitk thing from Paul, any p4 thing Luke and Lars are both happy with, etc.). You seem to be saying that the first of the two assumptions does not hold. Should I change my expectations while queuing Windows specific patches from Dscho?
Re: [PATCH] mingw: use OpenSSL's SHA-1 routines
Am 10.02.2017 um 17:04 schrieb Jeff King: On Fri, Feb 10, 2017 at 04:49:02PM +0100, Johannes Schindelin wrote: I think this is only half the story. A heavy-sha1 workload is faster, which is good. But one of the original reasons to prefer blk-sha1 (at least on Linux) is that resolving libcrypto.so symbols takes a non-trivial amount of time. I just timed it again, and it seems to be consistently 1ms slower to run "git rev-parse --git-dir" on my machine (from the top-level of a repo). 1ms is mostly irrelevant, but it adds up on scripted workloads that start a lot of git processes. You know my answer to that. If scripting slows things down, we should avoid it in production code. As it is, scripting slows us down. Therefore I work slowly but steadily to get rid of scripting where it hurts most. Well, yes. My question is more "what does it look like on normal Git workloads?". Are you trading off an optimization for your giant 450MB index workload (which _also_ could be fixed by trying do the slow operation less, rather than micro-optimizing it) in a way that hurts people working with more normal sized repos? For instance, "make BLK_SHA1=Yes test" is measurably faster for me than "make BLK_SHA1= test". The patch does add a new runtime dependency on libcrypto.dll in my environment. I would be surprised if it does not also with your modern build tools. I haven't had time to compare test suite runtimes. I'm open to the argument that it doesn't matter in practice for normal git users. But it's not _just_ scripting. It depends on the user's pattern of invoking git commands (and how expensive the symbol resolution is on their system). It can be argued that in normal interactive use, it is hard to notice that another DLL is loaded. Don't forget, though, that on Windows it is not only the pure time to resolve the entry points, but also that typically virus scanners inspect every executable file that is loaded, which adds another share of time. I'll use the patch for daily work for a while to see whether it hurts. -- Hannes
Re: [PATCH] mingw: use OpenSSL's SHA-1 routines
Hi Hannes, On Sat, 11 Feb 2017, Johannes Sixt wrote: > Am 10.02.2017 um 00:41 schrieb Junio C Hamano: > > Johannes Schindelin writes: > > > > > From: Jeff Hostetler > > > > > > Use OpenSSL's SHA-1 routines rather than builtin block-sha1 > > > routines. This improves performance on SHA1 operations on Intel > > > processors. > > > ... > > > > > > Signed-off-by: Jeff Hostetler > > > Signed-off-by: Johannes Schindelin > > > --- > > > > Nice. Will queue as jh/mingw-openssl-sha1 topic; it is a bit too > > late for today's integration cycle to be merged to 'next', but let's > > have this by the end of the week in 'master'. > > Please don't rush this through. I didn't have a chance to cross-check the > patch; it will have to wait for Monday. I would like to address Peff's > concerns about additional runtime dependencies. I never meant this to be fast-tracked into git.git. We have all the time in our lives to get this in, as Git for Windows already carries this patch for a while, and shall continue to do so. Ciao, Dscho
Re: [PATCH] mingw: use OpenSSL's SHA-1 routines
Johannes Sixt writes: > Am 10.02.2017 um 00:41 schrieb Junio C Hamano: >> ... >> Nice. Will queue as jh/mingw-openssl-sha1 topic; it is a bit too >> late for today's integration cycle to be merged to 'next', but let's >> have this by the end of the week in 'master'. > > Please don't rush this through. I didn't have a chance to cross-check > the patch; it will have to wait for Monday. I would like to address > Peff's concerns about additional runtime dependencies. OK. Will mark it as "Will cook in 'next'" for now.
Re: [PATCH] mingw: use OpenSSL's SHA-1 routines
Am 10.02.2017 um 00:41 schrieb Junio C Hamano: Johannes Schindelin writes: From: Jeff Hostetler Use OpenSSL's SHA-1 routines rather than builtin block-sha1 routines. This improves performance on SHA1 operations on Intel processors. ... Signed-off-by: Jeff Hostetler Signed-off-by: Johannes Schindelin --- Nice. Will queue as jh/mingw-openssl-sha1 topic; it is a bit too late for today's integration cycle to be merged to 'next', but let's have this by the end of the week in 'master'. Please don't rush this through. I didn't have a chance to cross-check the patch; it will have to wait for Monday. I would like to address Peff's concerns about additional runtime dependencies. -- Hannes
Re: [PATCH] mingw: use OpenSSL's SHA-1 routines
On Fri, Feb 10, 2017 at 04:49:02PM +0100, Johannes Schindelin wrote: > > I think this is only half the story. A heavy-sha1 workload is faster, > > which is good. But one of the original reasons to prefer blk-sha1 (at > > least on Linux) is that resolving libcrypto.so symbols takes a > > non-trivial amount of time. I just timed it again, and it seems to be > > consistently 1ms slower to run "git rev-parse --git-dir" on my machine > > (from the top-level of a repo). > > > > 1ms is mostly irrelevant, but it adds up on scripted workloads that > > start a lot of git processes. > > You know my answer to that. If scripting slows things down, we should > avoid it in production code. As it is, scripting slows us down. Therefore > I work slowly but steadily to get rid of scripting where it hurts most. Well, yes. My question is more "what does it look like on normal Git workloads?". Are you trading off an optimization for your giant 450MB index workload (which _also_ could be fixed by trying do the slow operation less, rather than micro-optimizing it) in a way that hurts people working with more normal sized repos? For instance, "make BLK_SHA1=Yes test" is measurably faster for me than "make BLK_SHA1= test". I'm open to the argument that it doesn't matter in practice for normal git users. But it's not _just_ scripting. It depends on the user's pattern of invoking git commands (and how expensive the symbol resolution is on their system). -Peff
Re: [PATCH] mingw: use OpenSSL's SHA-1 routines
Hi Peff, On Fri, 10 Feb 2017, Jeff King wrote: > On Thu, Feb 09, 2017 at 11:27:49PM +0100, Johannes Schindelin wrote: > > > From: Jeff Hostetler > > > > Use OpenSSL's SHA-1 routines rather than builtin block-sha1 routines. > > This improves performance on SHA1 operations on Intel processors. > > > > OpenSSL 1.0.2 has made considerable performance improvements and > > support the Intel hardware acceleration features. See: > > https://software.intel.com/en-us/articles/improving-openssl-performance > > https://software.intel.com/en-us/articles/intel-sha-extensions > > > > To test this I added/staged a single file in a gigantic repository > > having a 450MB index file. The code in read-cache.c verifies the > > header SHA as it reads the index and computes a new header SHA as it > > writes out the new index. Therefore, in this test the SHA code must > > process 900MB of data. Testing was done on an Intel I7-4770 CPU @ > > 3.40GHz (Intel64, Family 6, Model 60) CPU. > > I think this is only half the story. A heavy-sha1 workload is faster, > which is good. But one of the original reasons to prefer blk-sha1 (at > least on Linux) is that resolving libcrypto.so symbols takes a > non-trivial amount of time. I just timed it again, and it seems to be > consistently 1ms slower to run "git rev-parse --git-dir" on my machine > (from the top-level of a repo). > > 1ms is mostly irrelevant, but it adds up on scripted workloads that > start a lot of git processes. You know my answer to that. If scripting slows things down, we should avoid it in production code. As it is, scripting slows us down. Therefore I work slowly but steadily to get rid of scripting where it hurts most. Ciao, Dscho
Re: [PATCH] mingw: use OpenSSL's SHA-1 routines
On Thu, Feb 09, 2017 at 11:27:49PM +0100, Johannes Schindelin wrote: > From: Jeff Hostetler > > Use OpenSSL's SHA-1 routines rather than builtin block-sha1 routines. > This improves performance on SHA1 operations on Intel processors. > > OpenSSL 1.0.2 has made considerable performance improvements and > support the Intel hardware acceleration features. See: > https://software.intel.com/en-us/articles/improving-openssl-performance > https://software.intel.com/en-us/articles/intel-sha-extensions > > To test this I added/staged a single file in a gigantic > repository having a 450MB index file. The code in read-cache.c > verifies the header SHA as it reads the index and computes a new > header SHA as it writes out the new index. Therefore, in this test > the SHA code must process 900MB of data. Testing was done on an > Intel I7-4770 CPU @ 3.40GHz (Intel64, Family 6, Model 60) CPU. I think this is only half the story. A heavy-sha1 workload is faster, which is good. But one of the original reasons to prefer blk-sha1 (at least on Linux) is that resolving libcrypto.so symbols takes a non-trivial amount of time. I just timed it again, and it seems to be consistently 1ms slower to run "git rev-parse --git-dir" on my machine (from the top-level of a repo). 1ms is mostly irrelevant, but it adds up on scripted workloads that start a lot of git processes. Whether it's a net win or not depends on how much sha1 computation you do in your workload versus how many processes you start. I don't know what that means for Windows, though. My impression is that process startup is so painfully slow there that the link time may just be lost in the noise. It may just always be a win there. So not really an objection to your patch, but something you may want to consider. (Of course, it would in theory be possible to have the best of both worlds either by static-linking openssl, or by teaching block-sha1 the same optimizations, but both of those are obviously much more complex). -Peff
Re: [PATCH] mingw: use OpenSSL's SHA-1 routines
Johannes Schindelin writes: > From: Jeff Hostetler > > Use OpenSSL's SHA-1 routines rather than builtin block-sha1 routines. > This improves performance on SHA1 operations on Intel processors. > ... > > Signed-off-by: Jeff Hostetler > Signed-off-by: Johannes Schindelin > --- Nice. Will queue as jh/mingw-openssl-sha1 topic; it is a bit too late for today's integration cycle to be merged to 'next', but let's have this by the end of the week in 'master'. Thanks.