Nathan and Tim, Thanks much for helping refocus here. Responses inline.
On Fri, Aug 15, 2014 at 10:29 AM, Nathan Typanski <ntypan...@gmail.com> wrote: > Mike, > > Sorry for contributing to the off-topic discussion. I'll try to make > up for it by posting some interesting data. No worries; I've certainly learned a lot from these tangents! Just wanted to make sure the original signal didn't get lost in the midst. :-) > I wanted to know if your changes improved speed on my system, and ran > my own suite of benchmarks. I didn't actually bench incremental > builds, since I wasted all my time benching the full ones, but I put > together some charts and posted all the data along with them for > inspection. > > <http://goo.gl/zXoniR> > > If I'm correct in saying so, I think the right way to run the current > test suite is to do > > $ make clean && make && make test > > which means when I benchmarked `make test` with the current `master` > tree, I ran /usr/bin/time on both the `make` and `make test` commands, > appended both of those a file, and summed their output. If I was wrong > in how I did that, then the `make test` bench is wrong and you didn't > shave 3-4 seconds off the full build like I thought ;) > > My system software/specs: > > - Arch Linux w/ 3.16.0 kernel > - i7-2620M CPU. During the builds my clock speed averaged 3.2 GHz. > - 8GiB RAM > - Crucial MX100 SSD > - GCC compiler Thanks much for doing this! But I'm really surprised that you're getting 16s full, nonparallel builds from the existing recursive make structure, when my Mac Pro still clocks 65s. What am I missing here? Also, I timed 'make clean && make' and 'make test/test_heartbeat' (and not 'make test') separately on purpose; I'll explain further below. > Also, I read your article. The bits about `ssl/d1_both.c` not being > detected in the recursive make are scary. Ultimately it would probably > end up wasting time more than anything - if the person who merges the > code does a full `make clean` and runs the test suite before pulling > anything, they should catch that. But without a CI server, of course, > that's more time wasted trying to build broken code ... All true. On the one hand, it seems that the single-Makefile experiment is in pursuit of an ideal; but at the same time, I believe this experiment demonstrates that the ideal is well within practical reach. Every little bit of friction and surprising behavior we can remove from the build process improves productivity by improving efficiency and ensuring correctness--and it makes for a *much* improved testing experience. Speaking of which, that's why my final examples focused not on 'make test', which runs the entire test suite, but 'make test/test_heartbeat'. heartbeat_test.c, aside from being the test I wrote, is currently the only example in the code of the type of unit test I'm hoping to encourage: small, focused, and fast. It tests specific functions/one specific feature, and runs quickly enough that it maintains the flow of concentration while changing the code under test, whereas most of the other tests drive end-to-end runs through the command line interface. Those are indeed critical, but as proven, don't catch everything; hence the push to improve unit test coverage. The build time for this single test should prove representative of the edit-compile-build-test cycle that I hope OpenSSL contributors will begin to experience and adopt as a regular development habit. That said, we can also work on improving the isolation of the rest of the test suite so the full 'make test' will also benefit from maximum parallelism. Right now, many tests expect to be run one-at-a-time inside the test/ directory, which means that given enough parallelism, the tests in their current state step on one another. (Yes, I've hit this. :-) On Fri, Aug 15, 2014 at 11:35 AM, Tim Hollebeek <tholleb...@trustwave.com> wrote: > Ask and ye shall receive: > > 1. You are 100% correct that recursive make is completely broken, and > moving to a single makefile is a significant improvement even if something > else is done in the medium/long term. That's what I'm shooting for: Before adopting a new system wholesale, if we ever do, it'd be nice to improve the existing system such that we have a reliable benchmark to compare it against. My particular concern being the "if": There's no clear guarantee at this point that anything else will be adopted, so it'd be nice to cleanup the system we've already got, if possible; and I believe the experiment demonstrates it's quite possible. > 2. If using GMake everywhere is practical, I think it’s a good idea. > I’ve worked on open source projects before that tried to support multiple > makefile systems, and the results were so complicated as to be > unmaintainable. Portability of compilation is the responsibility of the > build system, not the project build built. This is essentially the same > philosophy as CMake. The code and even Makefiles shouldn’t care what > platform they are on unless they REALLY need to. Configure was the old way > to do this, but IMHO it is showing its age. I first used Configure/gmake > like, what, 20 years ago? And even then I had a custom precompiler to edit > the inputs to take into account various stupidities of various > platforms/architectures. I'm strongly leaning on the side of GNU make advocacy here, for purely practical reasons. I'm interested in hearing similarly practical reasons for maintaining BSD (and other?) make compatibility, and how the features that made the GNU make solution feasible can be expressed in those other flavors. But even then, why not just pick one ubiquitous flavor and be done with it? What's to be gained by maintaining compatibility with other flavors when GNU is already available on the same platforms? > 3. FIPS confuses the crap out of our developers every time they try to > build it correctly. Any improvements you can make there will be greatly > appreciated. I'm relatively clueless about FIPS, but I made sure that the "dummy" FIPS builds I was doing maintained parity between the baseline and experimental builds. > 4. I don’t know if you had problems with it, but I’ve had problems for > years with the extensive use of symlinks to source code within the OpenSSL > source. If that could go away, I’d find the OpenSSL source much easier to > understand and navigate. Which symlinks? The only ones I've noticed are those build in include/openssl and test/. Not that I'm attached to them, mind you, but I haven't really been stymied by them--yet. Incidentally, I've posted tips about using cscope and Exuberant Ctags to navigate the source: http://wiki.openssl.org/index.php/Testing_and_Development_Tools_and_Tips > Off-topic #1: your article on goto fail and heartbleed is awesome, and > should be read in its entirety by everyone working on security critical > software. Aww shucks, now you just got me blushing. :-P > Off-topic #2: This would be a HUGE change, but I really wish OpenSSL would > move towards something like Google Test to make it easier to write tests. I chatted with Rich Salz briefly in-person about this, and he indicated this might be feasible. And trust me, I'm the premier champion of Saint Zhanyong, genius and original author of Google Test (and its companion, Google Mock). :-) In the short term, however, I introduced this "pseudo-xUnit pattern" in heartbest_test.c, along with the test/testutil.{h,c} utilities, to mimic much of the structure of Google Test without making GT and the C++ compiler it requires it a build requirement--yet. The idea being, we can still write lots of clearly-structured tests now, even absent a framework, and do so in such a way that migration to a new framework would prove relatively painless. > Off-topic #3: If there is a move towards OpenSSL using CMake, I and possibly > a few people I know would probably be willing to help. Well, I'd certainly be willing to hear your thoughts on Cmake now. I've also noted the concern of Nathan and JJK. Geoff Thorpe (and possibly Ben Laurie, and maybe other core OpenSSL folks) would probably like to jump in too, I'm sure. Hit me up offline, and maybe we can set up a Hangout or something for next week. Mike ______________________________________________________________________ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
