Re: SoC: Distributed Audit Daemon project
On Sat, 26 May 2007, M. Warner Losh wrote: In message: [EMAIL PROTECTED] Benjamin Lutz [EMAIL PROTECTED] writes: : On Friday 25 May 2007 01:22:21 Alexey Mikhailov wrote: : [...] : 2. As I said before initial subject of this project was Distributed : audit daemon. But after some discussions we had decided that this : project can be done in more general maner. We can perform distributed : logging for any user-space app. : [...] : : This sounds very similar to syslogd. Is it feasible to make dlogd a drop-in : replacement for syslogd, at least from a syslog-using-program point of view? I suspect that it is dealing with different data streams. syslog is for programs sending text voluntarily. auditd is for pulling audit trails out of the kernel for which the 'target' programs have no knowledge that the audit trails are being generated, let alone anyway to prevent it. To possibly clarify a few points: (1) A distributed audit daemon wouldn't eliminate the need for local daemons that already manage log streams from various sources -- for example, syslogd for syslog, auditd for audit, Apache generating its own log files, etc. The goal of the distributed audit/log daemon is to manage these log files once log sources (such as auditd) are done with their logs. (2) One of the trickiest parts of the design will be the interaction between log sources and the audit daemon, so that log files can reliably change hands from being managed by the source to the distributed log tool. In the event of a system crash/power loss/network partition/syslogd crash/etc, we still want the log file to be picked up and synchronized. Hence discussion of an explicit hand-off API rather than casually looking in the same directory and hoping we get it right. (3) Unlikely syslogd's network logging support, the goal here is secure, reliable, batched delivery. We've talked a bit about live audit record delivery for IDS, but up front what we actually want to do is push the same sorts of reliability guarantees already present for local trail management to the distributed case. We are looking at pushing existing trail files over the network in a spooled way rather than shipping individual records as they are generated for this reason. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Looking for speed increases in make index and pkg_version for ports
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Stephen Montgomery-Smith wrote: I have been thinking a lot about looking for speed increases for make index and pkg_version and things like that. So for example, in pkg_version, it calls make -V PKGNAME for every installed package. Now make -V PKGNAME should be a speedy operation, but the make has to load in and analyze bsd.port.mk, a quite complicated file with about 200,000 characters in it, when all it is needing to do is to figure out the value of the variable PKGNAME. pkg_version is one thing -- but to build the INDEX you need to extract at least the values of the following variables: PKGNAME .CURDIR PREFIX COMMENT DESCR MAINTAINER CATEGORIES EXTRACT_DEPENDS PATCH_DEPENDS FETCH_DEPENDS BUILD_DEPENDS RUN_DEPENDS LIB_DEPENDS Plus you need to grep in the referenced pkg-descr file for any WWW links. I also extract the values of: MASTER_PORT .MAKEFILE_LIST SUBDIR for my FreeBSD::Portindex stuff. Trouble is, by the time you've extracted all that lot, you have pretty much done the same level of variable processing as you would were you actually going to build the port. One thing that would speed up this process would be a make option to just do parsing of the Makefile and variable expansion, without calling stat(2) on all the various sources and dependencies involved. For instance: happy-idiot-talk:...ports/databases/mysql-connector-java:% truss make -V PKGNAME | grep stat | wc -l 49 It is quite instructive to see what files make(1) touches while doing that. At least half of them are irrelevant if all make(1) is going to do is print out the values of some variables. Multiply that by 17,000 and it adds up to a big waste of effort. Cheers, Matthew - -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.3 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGWpTA8Mjk52CukIwRCI0BAJ0bX5hTAJkMCO6Pl+cA4THv3mKulwCgg+39 kCyAGOTYYz9vEzzM9NRe3no= =MqFV -END PGP SIGNATURE- ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Looking for speed increases in make index and pkg_version for ports
Matthew Seaman wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Stephen Montgomery-Smith wrote: I have been thinking a lot about looking for speed increases for make index and pkg_version and things like that. So for example, in pkg_version, it calls make -V PKGNAME for every installed package. Now make -V PKGNAME should be a speedy operation, but the make has to load in and analyze bsd.port.mk, a quite complicated file with about 200,000 characters in it, when all it is needing to do is to figure out the value of the variable PKGNAME. pkg_version is one thing -- but to build the INDEX you need to extract at least the values of the following variables: PKGNAME .CURDIR PREFIX COMMENT DESCR MAINTAINER CATEGORIES EXTRACT_DEPENDS PATCH_DEPENDS FETCH_DEPENDS BUILD_DEPENDS RUN_DEPENDS LIB_DEPENDS Plus you need to grep in the referenced pkg-descr file for any WWW links. I also extract the values of: MASTER_PORT .MAKEFILE_LIST SUBDIR for my FreeBSD::Portindex stuff. Trouble is, by the time you've extracted all that lot, you have pretty much done the same level of variable processing as you would were you actually going to build the port. One thing that would speed up this process would be a make option to just do parsing of the Makefile and variable expansion, without calling stat(2) on all the various sources and dependencies involved. For instance: happy-idiot-talk:...ports/databases/mysql-connector-java:% truss make -V PKGNAME | grep stat | wc -l 49 It is quite instructive to see what files make(1) touches while doing that. At least half of them are irrelevant if all make(1) is going to do is print out the values of some variables. Multiply that by 17,000 and it adds up to a big waste of effort. Are you sure? A good deal of the stats() is make poking around for the make infrastructure. This should be in the cache. And there are a couple of stats for the *done* files that might be avoided by doing something in the ports infrastructure. But as I already said in my previous mail: numbers, please, no guessing. harti ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Looking for speed increases in make index and pkg_version for ports
Stephen Montgomery-Smith wrote: I have been thinking a lot about looking for speed increases for make index and pkg_version and things like that. So for example, in pkg_version, it calls make -V PKGNAME for every installed package. Now make -V PKGNAME should be a speedy operation, but the make has to load in and analyze bsd.port.mk, a quite complicated file with about 200,000 characters in it, when all it is needing to do is to figure out the value of the variable PKGNAME. I suggest rewriting make so that variables are only evaluated on a need to know basis. So, for example, if all we need to know is PKGNAME, there is no need to evaluate, for example, _RUN_LIB_DEPENDS, unless the writer of that particular port has done something like having PORTNAME depend on the value of _RUN_LIB_DEPENDS. So make should analyze all the code it is given, and only figure it out if it is needed to do so. This would include, for example, figuring out .for and .if directives on a need to know basis as well. I have only poked around a little inside the source for make, but I have a sense that this would be a major undertaking. I certainly have not thought through what it entails in more than a cursory manner. However I am quite excited about the possibility of doing this, albeit I may well put off the whole thing for a year or two or even forever depending upon other priorities in my life. However, in the mean time I want to throw this idea out there to get some feedback, either of the form of this won't work, or of the form I will do it, or I have tried to do this. Having done a great deal of rewriting of make some two years ago I can tell you that even a small change to make is a tough job testing-wise: run all the combinations of !-j and -j N on all architectures and run the change through the port-building cluster. That's a warning to start with. Second I would start with careful profiling to find out where the problem actually is. You might be surprised. As an example: several times the idea came up to use a hash structure instead of linear lists for make variables. I got a patch for this and - it makes absolutely no difference performance-wise (well, there was some indication that performance gets worse, but that was around or below noise level). With careful I mean to find out who takes the time: 1. make and its sub-makes for a) reading the file; b) parsing the file (note that .if and .for processing is done while parsing); c) processing targets. 2. sub-shells executed for executing targets commands (note, that make optimizes the subshells away when there are no special shell symbol in the command line) 3. executed programs (find, sort, ...) Until you have numbers for this everything is rather moot. It might be a good idea to put some performance measurement hooks into make for this to do. If anybody wants to work on make, I would rather recommend to implement %-rules :-) And if anybody wants to recommend gmake over make(1) - look into the code, what mess that is :-/ Regards, harti ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Looking for speed increases in make index and pkg_version for ports
Hartmut Brandt wrote: Having done a great deal of rewriting of make some two years ago I can tell you that even a small change to make is a tough job testing-wise: run all the combinations of !-j and -j N on all architectures and run the change through the port-building cluster. That's a warning to start with. Second I would start with careful profiling to find out where the problem actually is. You might be surprised. As an example: several times the idea came up to use a hash structure instead of linear lists for make variables. I got a patch for this and - it makes absolutely no difference performance-wise (well, there was some indication that performance gets worse, but that was around or below noise level). With careful I mean to find out who takes the time: Yes, I must admit that I thought that a hash structure for the variables would greatly increase the speed of make. I rewrote it using Berkeley databases, and like you said - absolutely no difference!! I even tried btrees. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Looking for speed increases in make index and pkg_version for ports
Stephen Montgomery-Smith wrote: Hartmut Brandt wrote: Having done a great deal of rewriting of make some two years ago I can tell you that even a small change to make is a tough job testing-wise: run all the combinations of !-j and -j N on all architectures and run the change through the port-building cluster. That's a warning to start with. Second I would start with careful profiling to find out where the problem actually is. You might be surprised. As an example: several times the idea came up to use a hash structure instead of linear lists for make variables. I got a patch for this and - it makes absolutely no difference performance-wise (well, there was some indication that performance gets worse, but that was around or below noise level). With careful I mean to find out who takes the time: Yes, I must admit that I thought that a hash structure for the variables would greatly increase the speed of make. I rewrote it using Berkeley databases, and like you said - absolutely no difference!! I even tried btrees. My guess at that time was that because there are actually many variable tables (one per target and the global one) and only a small number of variables in most of the tables the initialisation overhead outweights what you win through the hashing. As for the profiling - I did some profiling on buildworld then. From the several hours a buildworld took only one or two minutes were used by all the makes. At this point I stopped optimizing make :-) (I don't remember the exact numbers - that was two or three years ago). harti ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Looking for speed increases in make index and pkg_version for ports
Stephen Montgomery-Smith wrote: I have been thinking a lot about looking for speed increases for make index and pkg_version and things like that. So for example, in pkg_version, it calls make -V PKGNAME for every installed package. Now make -V PKGNAME should be a speedy operation, but the make has to load in and analyze bsd.port.mk, a quite complicated file with about 200,000 characters in it, when all it is needing to do is to figure out the value of the variable PKGNAME. As long as far-out ideas are being discussed, how about caching such information (including dependenices) in a file (I'd call it a database but then I'd had to start a holy war :) ) so it's calculated only once, preferably on the portsnap / cvsup servers and not at the end-user? signature.asc Description: OpenPGP digital signature
Re: Looking for speed increases in make index and pkg_version for ports
Ivan Voras wrote: Stephen Montgomery-Smith wrote: I have been thinking a lot about looking for speed increases for make index and pkg_version and things like that. So for example, in pkg_version, it calls make -V PKGNAME for every installed package. Now make -V PKGNAME should be a speedy operation, but the make has to load in and analyze bsd.port.mk, a quite complicated file with about 200,000 characters in it, when all it is needing to do is to figure out the value of the variable PKGNAME. As long as far-out ideas are being discussed, how about caching such information (including dependenices) in a file (I'd call it a database but then I'd had to start a holy war :) ) so it's calculated only once, preferably on the portsnap / cvsup servers and not at the end-user? Because the information is not a constant. For example, the mpg123 port changes its PKGNAME as soon as esound is installed. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Looking for speed increases in make index and pkg_version for ports
Jeremy Chadwick wrote: On Sun, May 27, 2007 at 03:52:16PM -0500, Stephen Montgomery-Smith wrote: I have been thinking a lot about looking for speed increases for make index and pkg_version and things like that. So for example, in pkg_version, it calls make -V PKGNAME for every installed package. Now make -V PKGNAME should be a speedy operation, but the make has to load in and analyze bsd.port.mk, a quite complicated file with about 200,000 characters in it, when all it is needing to do is to figure out the value of the variable PKGNAME. I have a related question, pertaining to make all-depends-list and the utter atrocity that is the make variable ALL-DEPENDS-LIST. If you don't know what it is, look for ^ALL-DEPENDS-LIST around line 5175, in bsd.ports.mk. I posted this to [EMAIL PROTECTED], but now I am realizing that it is [EMAIL PROTECTED] that gets more responses. Anyway, here is a multithreaded program all-depends-list that can get you double the speed on dual processor systems, and even some small speed gains on single processor systems. E.g. all-depends-list /usr/ports/x11/xorg http://www.math.missouri.edu/~stephen/all-depends-list.c ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Looking for speed increases in make index and pkg_version for ports
On Monday 28 May 2007 03:43, you wrote: Maybe I should look at the inner workings of cmake and gmake. Maybe they have some good ideas. However having looked through the source code of make, and also looking at the cvs logs, it does seem to be well written. The only possibility I see of making it go a lot faster is a complete redesign, e.g. my just in time idea for processing variables. Stephen Just in time (jit), if I remember correctly, is a term used by java interpreters which compile the byte code into machine code!!! Perhaps this could be developed for makefile's, especially bsd.*.mk. This, I think, could be done in two ways: 1) Develop the bsd.*.mk files in C and link it in with make, or 2) Use the makefiles as source to compile into machine code (passibly via C-ASM). The machine code could be created on demand, or cached and only updated if the source makefile changes. I am not sure if this could work or even if it will have any significant speed increase. However if method 2 does work it has the potential to radically increase the speed of ports _while_ maintaining the flexability. All that will be needed is an API for the machine code and a compiler??? David pgp6Gma7Ai2zD.pgp Description: PGP signature
Re: Looking for speed increases in make index and pkg_version for ports
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Ivan Voras wrote: Stephen Montgomery-Smith wrote: I have been thinking a lot about looking for speed increases for make index and pkg_version and things like that. So for example, in pkg_version, it calls make -V PKGNAME for every installed package. Now make -V PKGNAME should be a speedy operation, but the make has to load in and analyze bsd.port.mk, a quite complicated file with about 200,000 characters in it, when all it is needing to do is to figure out the value of the variable PKGNAME. As long as far-out ideas are being discussed, how about caching such information (including dependenices) in a file (I'd call it a database but then I'd had to start a holy war :) ) so it's calculated only once, preferably on the portsnap / cvsup servers and not at the end-user? Good idea. http://www.infracaninophile.co.uk/portindex/ Been done before though. Cheers, Matthew - -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.3 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGWwnL8Mjk52CukIwRCNDLAJ4jFCpr5y7uAQi97mVRV3Pc4+c99ACeN9vQ tOc6IzTQ90+wObG34KWQzzw= =XuiO -END PGP SIGNATURE- ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Looking for speed increases in make index and pkg_version for ports
On Mon, 28 May 2007, David Naylor wrote: On Monday 28 May 2007 03:43, you wrote: Maybe I should look at the inner workings of cmake and gmake. Maybe they have some good ideas. However having looked through the source code of make, and also looking at the cvs logs, it does seem to be well written. The only possibility I see of making it go a lot faster is a complete redesign, e.g. my just in time idea for processing variables. Stephen Just in time (jit), if I remember correctly, is a term used by java interpreters which compile the byte code into machine code!!! Perhaps this could be developed for makefile's, especially bsd.*.mk. This, I think, could be done in two ways: 1) Develop the bsd.*.mk files in C and link it in with make, or 2) Use the makefiles as source to compile into machine code (passibly via C-ASM). The machine code could be created on demand, or cached and only updated if the source makefile changes. I am not sure if this could work or even if it will have any significant speed increase. However if method 2 does work it has the potential to radically increase the speed of ports _while_ maintaining the flexability. All that will be needed is an API for the machine code and a compiler??? My gut reaction is the same as yours - I doubt that this would bring any speed increases. And the programming effort would be huge. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Looking for speed increases in make index and pkg_version for ports
In [EMAIL PROTECTED], Hartmut Brandt [EMAIL PROTECTED] typed: 1. make and its sub-makes for a) reading the file; b) parsing the file (note that .if and .for processing is done while parsing); c) processing targets. Make and submakes have been gone over already. See URL: http://miller.emu.id.au/pmiller/books/rmch/ . I'm not sure it can be applied to the ports tree, though. I haven't looked into it, but recalled this paper when you mentioned measuring makes and sub-makes. mike -- Mike Meyer [EMAIL PROTECTED] http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Looking for speed increases in make index and pkg_version for ports
Mike Meyer wrote: In [EMAIL PROTECTED], Hartmut Brandt [EMAIL PROTECTED] typed: 1. make and its sub-makes for a) reading the file; b) parsing the file (note that .if and .for processing is done while parsing); c) processing targets. Make and submakes have been gone over already. See URL: http://miller.emu.id.au/pmiller/books/rmch/ . I'm not sure it can be applied to the ports tree, though. I haven't looked into it, but recalled this paper when you mentioned measuring makes and sub-makes. mike Reducing the number of variables will certainly cut down on the amount of overhead in the make/submake context switches by a long shot. Maybe someone should consider running a 'pre-make' using the .mk files, find the variables of interest for all particular sub-ports, and then carry on the 'root make', i.e. make process in each port directory, with just the variables of interest. -Garrett ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Looking for speed increases in make index and pkg_version for ports
Garrett Cooper wrote: Mike Meyer wrote: In [EMAIL PROTECTED], Hartmut Brandt [EMAIL PROTECTED] typed: 1. make and its sub-makes for a) reading the file; b) parsing the file (note that .if and .for processing is done while parsing); c) processing targets. Make and submakes have been gone over already. See URL: http://miller.emu.id.au/pmiller/books/rmch/ . I'm not sure it can be applied to the ports tree, though. I haven't looked into it, but recalled this paper when you mentioned measuring makes and sub-makes. mike Reducing the number of variables will certainly cut down on the amount of overhead in the make/submake context switches by a long shot. Maybe someone should consider running a 'pre-make' using the .mk files, find the variables of interest for all particular sub-ports, and then carry on the 'root make', i.e. make process in each port directory, with just the variables of interest. -Garrett s/long shot/possibly a lot/g Also, I was thinking in particular of the X.Org 7.2 packages, because the bulk majority of the packages are smaller, and compile in a short amount of time. -Garrett ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Looking for speed increases in make index and pkg_version for ports
Stephen Montgomery-Smith wrote: Ivan Voras wrote: As long as far-out ideas are being discussed, how about caching such information (including dependenices) in a file (I'd call it a database but then I'd had to start a holy war :) ) so it's calculated only once, preferably on the portsnap / cvsup servers and not at the end-user? Because the information is not a constant. For example, the mpg123 port changes its PKGNAME as soon as esound is installed. Maybe the time has come to give up on some of the flexibility the ports tree has (and this particular one is confusing to the users) in favour of gaining speed? signature.asc Description: OpenPGP digital signature
Re: Looking for speed increases in make index and pkg_version for ports
Mike Meyer wrote: In [EMAIL PROTECTED], Hartmut Brandt [EMAIL PROTECTED] typed: 1. make and its sub-makes for a) reading the file; b) parsing the file (note that .if and .for processing is done while parsing); c) processing targets. Make and submakes have been gone over already. See URL: http://miller.emu.id.au/pmiller/books/rmch/ . I'm not sure it can be applied to the ports tree, though. I haven't looked into it, but recalled this paper when you mentioned measuring makes and sub-makes. Unfortunately you deleted the sentence before, so I rephrase it: before looking into optimizations find out where the time is actually spend - how many seconds of the hours the process takes, are actually spent in make and sub-makes. If the entire process takes 2 hours of which the makes take 20 seconds then by enhancing performance of make by 50% you win 10 seconds. This is probably not worth a single line of additional code. The paper you point to talks about something entirely different. harti ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Looking for speed increases in make index and pkg_version for ports
On Mon, May 28, 2007 at 11:34:24AM -0500, Stephen Montgomery-Smith wrote: Jeremy Chadwick wrote: On Sun, May 27, 2007 at 03:52:16PM -0500, Stephen Montgomery-Smith wrote: I have been thinking a lot about looking for speed increases for make index and pkg_version and things like that. So for example, in pkg_version, it calls make -V PKGNAME for every installed package. Now make -V PKGNAME should be a speedy operation, but the make has to load in and analyze bsd.port.mk, a quite complicated file with about 200,000 characters in it, when all it is needing to do is to figure out the value of the variable PKGNAME. I have a related question, pertaining to make all-depends-list and the utter atrocity that is the make variable ALL-DEPENDS-LIST. If you don't know what it is, look for ^ALL-DEPENDS-LIST around line 5175, in bsd.ports.mk. I posted this to [EMAIL PROTECTED], but now I am realizing that it is [EMAIL PROTECTED] that gets more responses. Anyway, here is a multithreaded program all-depends-list that can get you double the speed on dual processor systems, and even some small speed gains on single processor systems. E.g. all-depends-list /usr/ports/x11/xorg http://www.math.missouri.edu/~stephen/all-depends-list.c btw.. stehpen, when are you getting a commit bit? :) I certainly hope that soon enough ;) roman ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Looking for speed increases in make index and pkg_version for ports
In [EMAIL PROTECTED], Hartmut Brandt [EMAIL PROTECTED] typed: Mike Meyer wrote: In [EMAIL PROTECTED], Hartmut Brandt [EMAIL PROTECTED] typed: 1. make and its sub-makes for a) reading the file; b) parsing the file (note that .if and .for processing is done while parsing); c) processing targets. Make and submakes have been gone over already. See URL: http://miller.emu.id.au/pmiller/books/rmch/ . I'm not sure it can be applied to the ports tree, though. I haven't looked into it, but recalled this paper when you mentioned measuring makes and sub-makes. Unfortunately you deleted the sentence before, so I rephrase it: before looking into optimizations find out where the time is actually spend - how many seconds of the hours the process takes, are actually spent in make and sub-makes. If the entire process takes 2 hours of which the makes take 20 seconds then by enhancing performance of make by 50% you win 10 seconds. This is probably not worth a single line of additional code. The paper you point to talks about something entirely different. It think we're talking about two different things. You're talking about the efficiency of make, whereas he's talking about the efficiency of make. Um, wait. You're talking about what I'll call the *internal* efficiency of make, defined as how fast it does the things it does. He's talking about what I'll call the *external* efficiency of make, which is how well it does at doing the minimum amount of work it needs to do. I hope you can see where the confusion comes from. In particular, he talks about how recursive makefiles screw up evaluating complex variables, causing them to be executed multiple times. So if you're running a makefile to pull some variables value, as opposed to do real commands, and your entire process takes 2 hours and the Makefile takes 20 seconds, but it evaluates all the variables twice, then by fixing your makefile you win at least 59 minutes and 50 seconds. I think cutting the run time by 50% is worth some work. Benchmarking can help you decide which things it pays to work on if all you're worried about is the internal efficiency. However, the goal is to make the process faster, so we need to worry about the external efficiency as well. The problem here is that the worse it is, the less it looks like you stand to gain by looking at your makefile when you look at the benchmarks. Given that the ports system has both highly complex variables and is very recursive, I believe that it warrants investigation if you're going to work on making make in the ports faster. mike -- Mike Meyer [EMAIL PROTECTED] http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: GPT boot loader?
Resent to freebsd-hackers@: Greetings, all. On 2007.05.22, at 03:57, Ivan Voras wrote: Hi! I've had the opportunity to talk to Adam Martin, Marcel Moolenaar and Peter Wemm about making GPT bootable, but not all of them at the same time, so I'd like this thread to be the meeting point on the subject. I have offered to work on the MBR and some of the lower level assembler nasties to make this thing work. The most problematic issue is that the GPT requires 128 bit fields to identify the partition type. In a typical MBR we have 488 bytes to work with (512 bytes in one sector, 2 bytes for the boot-identifier, and another 32 bytes for the partition table.) If we insert a few valid GPT labels into the MBR to scan for, this takes up 16 bytes per entry. The lucky part is that the MBR really just needs to find a bootable partition, load a few sectors from it, and let go. (Adam and Peter have offered to modify the boot loader chain, but differently.) I can also work on boot2, not just mbr/boot0, to make it understand GPT, but I need to know a bit more about the information it needs to understand. In this situation, /boot/loader still needs GPT support -- according to Marcel, it gets help on the GPT interpretation from the EFI. I do not want to undertake the task of GPT-ifying the loader program. It's not worth it to have someone make the earlier boot stages work with GPT, unless the later stages also will support it. I have written a simple MBR code that will look for the first GPT partition marked with a GUID specified at compile time, and load the first 8k of that partition, to facilitate boot2. However, I do not think I will take this any further unless people take up the task of writing the GPT support for other parts of the boot-chain. Summary: The idea is to replace bsdlabels with GPT. The problem is that GPT is intended to be used with EFI and not to be bootable by regular BIOS machines. For FreeBSD, there are two distinct cases: 1. the machine is GPT-only, there are no other MSDOS and bsdlabel partitions 2. the machine is dual-boot or has some other need to retain MSDOS partitions. The second case is more convoluted as it means the MBR will hold a regular MSDOS partition table, and one of those partitions will hold a GPT - which is trivially done with GEOM but completely non-standard. Marcel and I discussed a hybrid solution, which requires almost no bootloader code to be written -- We could have the MBR point to a legacy boot partition, even down to a really small root filesystem, which the GPT also has an entry for, but the kernel in this root filesystem has a switch set to inform it that it must use GPT and not MBR/BSDlabels. This is not unlike what Apple have done with their EFI on x86 platform. In a related quirk, I have not used MBR partitions for almost 3 years now -- I just raw-BSDlabel my discs. So I know it is possible to do away with the MBR completely. The biggest obstacle to this on x86/amd64 computers is the single-sector bootloader limitations. GPT isn't very PC-friendly, as it also demands that the sectors right after the fake MBR be for the GPT. What I would love to see is some ability to have a smarter PC boot procedure. I'd like to hear more from Adam and Peter (and others!) about their ideas... Optimally, the PC architecture boot procedure (and perhaps the PC itself?) should die and fade away, leaving behind smarter bootloader and firmware code in its place, instead of shoehorning this functionality into an unnecessarily complicated bootloader chain. Since this does not seem to be in the cards, I guess we need to focus on this concept. -- ADAM David Alan Martin FreeBSD Hacker, TCSH Hacker, general UNIX coding Author of AutoFS for FreeBSD 6.x LSD Kernel Lead Developer Filesystem and Storage Lab, SUNY Stony Brook ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Looking for speed increases in make index and pkg_version for ports
Correct me if I wrong. Don't you missed the fact that chdir(2) changes process wide attribute? Though it's easy to fix with -C option. Stephen Montgomery-Smith wrote: Jeremy Chadwick wrote: On Sun, May 27, 2007 at 03:52:16PM -0500, Stephen Montgomery-Smith wrote: I have been thinking a lot about looking for speed increases for make index and pkg_version and things like that. So for example, in pkg_version, it calls make -V PKGNAME for every installed package. Now make -V PKGNAME should be a speedy operation, but the make has to load in and analyze bsd.port.mk, a quite complicated file with about 200,000 characters in it, when all it is needing to do is to figure out the value of the variable PKGNAME. I have a related question, pertaining to make all-depends-list and the utter atrocity that is the make variable ALL-DEPENDS-LIST. If you don't know what it is, look for ^ALL-DEPENDS-LIST around line 5175, in bsd.ports.mk. I posted this to [EMAIL PROTECTED], but now I am realizing that it is [EMAIL PROTECTED] that gets more responses. Anyway, here is a multithreaded program all-depends-list that can get you double the speed on dual processor systems, and even some small speed gains on single processor systems. E.g. all-depends-list /usr/ports/x11/xorg http://www.math.missouri.edu/~stephen/all-depends-list.c ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Looking for speed increases in make index and pkg_version for ports
Roman Divacky wrote: On Mon, May 28, 2007 at 11:34:24AM -0500, Stephen Montgomery-Smith wrote: Jeremy Chadwick wrote: On Sun, May 27, 2007 at 03:52:16PM -0500, Stephen Montgomery-Smith wrote: I have been thinking a lot about looking for speed increases for make index and pkg_version and things like that. So for example, in pkg_version, it calls make -V PKGNAME for every installed package. Now make -V PKGNAME should be a speedy operation, but the make has to load in and analyze bsd.port.mk, a quite complicated file with about 200,000 characters in it, when all it is needing to do is to figure out the value of the variable PKGNAME. I have a related question, pertaining to make all-depends-list and the utter atrocity that is the make variable ALL-DEPENDS-LIST. If you don't know what it is, look for ^ALL-DEPENDS-LIST around line 5175, in bsd.ports.mk. I posted this to [EMAIL PROTECTED], but now I am realizing that it is [EMAIL PROTECTED] that gets more responses. Anyway, here is a multithreaded program all-depends-list that can get you double the speed on dual processor systems, and even some small speed gains on single processor systems. E.g. all-depends-list /usr/ports/x11/xorg http://www.math.missouri.edu/~stephen/all-depends-list.c btw.. stehpen, when are you getting a commit bit? :) I certainly hope that soon enough ;) Probably not. The program seems to have a bug in it. In particular, I didn't read the fgetln man page sufficiently well. So think of it as a proof of concept rather than a finished product. I'm going to rest from this stuff for a while, but I enjoyed the exchanges and it has given me encouragement to work on it again in the future sometime. Stephen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Looking for speed increases in make index and pkg_version for ports
Stephen Montgomery-Smith wrote: Roman Divacky wrote: On Mon, May 28, 2007 at 11:34:24AM -0500, Stephen Montgomery-Smith wrote: Jeremy Chadwick wrote: On Sun, May 27, 2007 at 03:52:16PM -0500, Stephen Montgomery-Smith wrote: I have been thinking a lot about looking for speed increases for make index and pkg_version and things like that. So for example, in pkg_version, it calls make -V PKGNAME for every installed package. Now make -V PKGNAME should be a speedy operation, but the make has to load in and analyze bsd.port.mk, a quite complicated file with about 200,000 characters in it, when all it is needing to do is to figure out the value of the variable PKGNAME. I have a related question, pertaining to make all-depends-list and the utter atrocity that is the make variable ALL-DEPENDS-LIST. If you don't know what it is, look for ^ALL-DEPENDS-LIST around line 5175, in bsd.ports.mk. I posted this to [EMAIL PROTECTED], but now I am realizing that it is [EMAIL PROTECTED] that gets more responses. Anyway, here is a multithreaded program all-depends-list that can get you double the speed on dual processor systems, and even some small speed gains on single processor systems. E.g. all-depends-list /usr/ports/x11/xorg http://www.math.missouri.edu/~stephen/all-depends-list.c btw.. stehpen, when are you getting a commit bit? :) I certainly hope that soon enough ;) Probably not. The program seems to have a bug in it. In particular, I didn't read the fgetln man page sufficiently well. So think of it as a proof of concept rather than a finished product. I'm going to rest from this stuff for a while, but I enjoyed the exchanges and it has given me encouragement to work on it again in the future sometime. Stephen fgetln(2) just scans ahead to the next newline, so the pointer to the next line is returned and the length of the string (with newline char included) is stored in the len variable (2nd parameter to function). -Garrett ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Looking for speed increases in make index and pkg_version for ports
On Sun, May 27, 2007 at 03:30:48PM -0700, Jeremy Chadwick wrote: That said, I'll ask this out in the open: am I the only one who sees the benefit of GNU make in regards to this? There's a lot of built-in functions in GNU make which could help in regards to ports. I have no qualms with PMake per se, but if another tool gives us what we need, then maybe we should consider the pros and cons of adapting that. There's also CMake, which is incredibly fast. Yes, you are. What gmake benefits? Gmake does not provide the flexibility and power that pmake provides. Off the top of my head: gmake does not have .for loops, variable expansion modifiers, or even the != shell command variable assigment. I use these in almost every Makefile I write, and the ports uses these things quite a bit. Also, gmake syntax is horrendous compared to pmake. People are already complaining about how ugly the ports makefiles are-- they'd be worse under gmake. Might as well rewrite the whole infrastructure in /bin/sh ... Also, there's the licensing issues. Remember-- any significant changes to this infrastructure has to work with the core utilities.. this leaves out gmake, python, ruby, etc. I doubt anyone will find anything as powerful as pmake without sacrificing the much-used flexibility it provides. -- Rick C. Petty ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
