Re: Looking for speed increases in "make index"
On Sun, May 27, 2007 at 10:51:56PM -0400, Mike Meyer wrote: > In <[EMAIL PROTECTED]>, Edwin Groothuis <[EMAIL PROTECTED]> typed: > > On Sun, May 27, 2007 at 08:44:56PM -0500, Mark Linimon wrote: > > > In summary, the ports infrastructure is really complicated because it's > > > trying to deal with all kinds of constraints and conditions. I challenge > > Reading this, I was wondering what the ports infrastructure has > > ever done for us? > > See http://www.epicure.demon.co.uk/whattheromans.html > > While that's funny, it makes me wonder if you're serious when you ask > the question. No I wasn't serious, I was sarcastic. I've spend enough time working in ports/Mk to know what is has done for us. Edwin -- Edwin Groothuis |Personal website: http://www.mavetju.org [EMAIL PROTECTED]| Weblog: http://www.mavetju.org/weblog/ ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Looking for speed increases in "make index"
In <[EMAIL PROTECTED]>, Edwin Groothuis <[EMAIL PROTECTED]> typed: > On Sun, May 27, 2007 at 08:44:56PM -0500, Mark Linimon wrote: > > In summary, the ports infrastructure is really complicated because it's > > trying to deal with all kinds of constraints and conditions. I challenge > Reading this, I was wondering what the ports infrastructure has > ever done for us? > See http://www.epicure.demon.co.uk/whattheromans.html While that's funny, it makes me wonder if you're serious when you ask the question. The ports system is a wonder. If you've ever tried installing software off the net without such a thing to help, you'll know what I mean. If you haven't, you should thank jkh for your state of blessedness. That said, it's now a decade old, and I'm sure doing far more than jkh ever expected of it. It's also tightly integarted with the package system, which is in a similar state. Both are suffering in comparison to newer systems, many of which have less ambitious goals. It seems like in the last month or so a lot of people have popped up with an interest in reworking one or both of them. Hopefully, some of them with time to do so will get good advice. http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Looking for speed increases in "make index"
On Mon, May 28, 2007 at 12:15:28AM +0200, Michel Talon wrote: > To gain some performance, a first idea would be to simplify > bsd.ports.mk. I am convinced that a substantial part of the 4000 lines > are historical crap which serve no useful purpose. 11272 of LOC in bsd.*.mk, but who's counting. > There are tons of variables who have probably purely anecdotical > interest. There are targets which could be happily suppressed. Please let us know which functionality you think is extra. You should use the individual port Makefiles as well as ports/Makefile to figure out what is unused. For extra credit, please include ports/Tools/portbuild/scripts so the build cluster will continue to work. Please don't think I am picking on you specifically; however, about every 6 months or so someone decides that "the ports framework is too complicated" without saying exactly what needs to be removed. Since I look at all the portmgr PRs as they come in, and participate in rejecting in some of the (by our determination) more marginal features, I can assure you that not every single new proposal makes it in there, nor has in the past. Every- thing that's in there is because there was some specific justification for it (at least at the time). Given that we had no install base, a significant rewrite would not be a burden, but that's not the case. Please note, I've agreed for several years that a great deal of the code could be factored out into some kind of C library for speed and reduction of code duplication. Some work is going towards that in the Summer of Code. But the hard part is making it work, in a backwards compatible fashion, and doing the exhaustive regression testing to prove that it solves more problems that it creates. (portmgr spends a _lot_ of time on regression testing, behind the scenes.) In summary, the ports infrastructure is really complicated because it's trying to deal with all kinds of constraints and conditions. I challenge anyone who thinks things can be removed to roll up their sleeves and make a good case for it. I'd be happy to have something easier to read. mcl ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Looking for speed increases in "make index"
On Sun, May 27, 2007 at 08:44:56PM -0500, Mark Linimon wrote: > In summary, the ports infrastructure is really complicated because it's > trying to deal with all kinds of constraints and conditions. I challenge Reading this, I was wondering what the ports infrastructure has ever done for us? See http://www.epicure.demon.co.uk/whattheromans.html Edwin -- Edwin Groothuis |Personal website: http://www.mavetju.org [EMAIL PROTECTED]| Weblog: http://www.mavetju.org/weblog/ ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Looking for speed increases in "make index" and pkg_version for ports
I'm looking for something that will work with the existing framework. But yes, I get the feeling that maybe using "make" to process the ports might be the source of the problem. Make is a program primarily designed for figuring out which was made first, the target or the source, but in the ports what we really want is a scripting language that presides over "cd WKSRC; make". (P.S. sorry for top-posting, but I am following your lead.) Bakul Shah wrote: Not quite what you asked for but... Given the size and complexity of the port system I have long felt that rather than do everything via more and more complex Mk/*.mk what is is needed is a ports server and a thin CLI frontend to it. This server can store dependency data in an efficient manner, deal with conditional dependencies, port renames, security and what not. It can build or fetch or serve packages, handle updates etc. Things mentioned in UPDATING file can instead be done by the server. In general it can automate a lot of stuff, remove error prone redundancies etc. If it is small enough and written in C, it can even be shipped with the base system instead of various pkg_* programs. It can provide two interfaces, one for normal users (with commands like add, check, config, delete, info, search, update, which) and one for port developers (command for adding/remove/renaming ports, etc.). Initially it must work with existing Makefiles. I have been thinking a lot about looking for speed increases for "make index" and pkg_version and things like that. So for example, in pkg_version, it calls "make -V PKGNAME" for every installed package. Now "make -V PKGNAME" should be a speedy operation, but the make has to load in and analyze bsd.port.mk, a quite complicated file with about 200,000 characters in it, when all it is needing to do is to figure out the value of the variable PKGNAME. I suggest rewriting "make" so that variables are only evaluated on a "need to know" basis. So, for example, if all we need to know is PKGNAME, there is no need to evaluate, for example, _RUN_LIB_DEPENDS, unless the writer of that particular port has done something like having PORTNAME depend on the value of _RUN_LIB_DEPENDS. So "make" should analyze all the code it is given, and only figure it out if it is needed to do so. This would include, for example, figuring out .for and .if directives on a need to know basis as well. I have only poked around a little inside the source for make, but I have a sense that this would be a major undertaking. I certainly have not thought through what it entails in more than a cursory manner. However I am quite excited about the possibility of doing this, albeit I may well put off the whole thing for a year or two or even forever depending upon other priorities in my life. However, in the mean time I want to throw this idea out there to get some feedback, either of the form of "this won't work," or of the form "I will do it," or "I have tried to do this." Best regards, Stephen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]" ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Looking for speed increases in "make index" and pkg_version for ports
Jeremy Chadwick wrote:
On Sun, May 27, 2007 at 03:52:16PM -0500, Stephen Montgomery-Smith wrote:
I have been thinking a lot about looking for speed increases for "make
index" and pkg_version and things like that. So for example, in
pkg_version, it calls "make -V PKGNAME" for every installed package. Now
"make -V PKGNAME" should be a speedy operation, but the make has to load in
and analyze bsd.port.mk, a quite complicated file with about 200,000
characters in it, when all it is needing to do is to figure out the value of
the variable PKGNAME.
I have a related question, pertaining to "make all-depends-list" and the
utter atrocity that is the make variable ALL-DEPENDS-LIST. If you don't
know what it is, look for ^ALL-DEPENDS-LIST around line 5175, in
bsd.ports.mk.
I call it an atrocity because it's a mix of make variable expansion
combined with sh scripting, and it's nearly impossible to read. It's
not commented either, so folks like myself are left thinking "What IS
this mess?!". It's expanded via $$(${ALL-DEPENDS-LIST}) in for loops,
throughout several places in bsd.port.mk.
I do not entirely understand what ALL-DEPENDS-LIST is about (that should
be apparent), but upon performing some of my benchmarks, I found this to
be a very slow piece of bsd.port.mk. make -V _DEPEND_DIRS is incredibly
fast, but ALL-DEPENDS-LIST is not.
Does it need to be done this way? Can we just iterate through all of
the ports, call make -V _DEPEND_DIRS, then sort | uniq the results? I
suppose that depends on the operation (make vs. make clean vs.
others)...
The port I used for testing some of the benchmarks was net/gacxtool. It
seems to be a good example of a "hefty" port.
I looked very hard at this particular piece of code. Once you
understand how it works, you realize that it is rather well written. It
basically does what you suggest, except it keeps a list of ports it has
already checked, so that it doesn't do them over again. This piece of
code is as efficient as it can possibly be, given that the program has
to recursively call make on every dependency port at least once (and as
it happens only once).
I suggest rewriting "make" so that variables are only evaluated on a "need
to know" basis. So, for example, if all we need to know is PKGNAME, there
is no need to evaluate, for example, _RUN_LIB_DEPENDS, unless the writer of
that particular port has done something like having PORTNAME depend on the
value of _RUN_LIB_DEPENDS. So "make" should analyze all the code it is
given, and only figure it out if it is needed to do so. This would include,
for example, figuring out .for and .if directives on a need to know basis as
well.
This sounds like a good solution. In fact, I'm lead to believe that
heavy reliance on /bin/sh is part of why the ports collection is slow.
No, it's not the sole reason, but it's one of many. I'm of the belief
that anything we can do to migrate portions into native make would be
benefitial.
I have done profiling tests on make, and in its current form,
bsd.ports.mk actually spends rather little time inside of bin/sh. Thw
writers of bsd.ports.mk have done a very good job of minimizing the
bin/sh calls.
That said, I'll ask this out in the open: am I the only one who sees the
benefit of GNU make in regards to this? There's a lot of built-in
functions in GNU make which could help in regards to ports. I have no
qualms with PMake per se, but if another tool gives us what we need,
then maybe we should consider the pros and cons of adapting that.
There's also CMake, which is incredibly fast.
Maybe I should look at the inner workings of cmake and gmake. Maybe
they have some good ideas. However having looked through the source
code of make, and also looking at the cvs logs, it does seem to be well
written. The only possibility I see of making it go a lot faster is a
complete redesign, e.g. my just in time idea for processing variables.
Stephen
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Looking for speed increases in "make index"
On Mon, 28 May 2007, Michel Talon wrote: Stephen Montgomery-Smith said: I suggest rewriting "make" so that variables are only evaluated on a "need to know" basis. or "I have tried to do this." Of course a lot of people have thinked about it, and quickly realized that it was not going to work. In the bsd.ports.mk, evaluation of one variable may be dependent on some conditional, which may itself be dependant on some other variable, appearing as some target. This constantly happens in bsd.ports.mk. If you think about that, you convince yourself that a "reduced make" needs to understand targets, needs to understand conditionals, and needs to evaluate not only the variable under consideration, but but possibly any other. In other words, you need a full blown make. Actually I have thought quite a lot about this over the last couple of weeks. You are correct in that the "make" really does have to be a full blown make. What I am suggesting is something rather different - a rather sophisticated make that works hard to only that which it really has to do. And it certainly would require a lot of sophistication, precisely for the reasons you state. To gain some performance, a first idea would be to simplify bsd.ports.mk. I am convinced that a substantial part of the 4000 lines are historical crap which serve no useful purpose. There are tons of variables who have probably purely anecdotical interest. There are targets which could be happily suppressed. Of course, due to the complexity of bsd.ports.mk, rewriting it is certainly not an easy task. There is a freebsd port whose aim is to rewrite it, i don't know how far they are. The NetBSD people have completely rewritten the system, i have no idea if the new one is faster. The OpenBSD people have also rewritten bsd.ports.mk, with apparently much faster makefile. I rewrote a bsd.ports.mk in which ALL targets were removed, and all it did was evalute variables. (I wrote a perl script that did this automatically.) The net effect was no speed gain. In other words, processing unnecessary targets is not the problem. I really have looked at bsd.ports.mk a lot, and given that it is written in the context of what make requires, it seems to be rather well and efficiently written. I did find one speed increase in bsd.gnome.mk, which I am kind of proud of, but even that only gained about a 5-10% speed increase in "make index." A second idea would be to multithread make, since modern machines will have a lot of cores. At present make -j forks submakes and waits for their completion. This doesn't help for the problem at hand. However, multithreading the execution of a single makefile is certainly not trivial due to the interdependencies. I don't think multithreaded make will help in this situation. Anyways, one of the things which cannot be a big factor is reading bsd.ports.mk from hard disk. It is certainly cached in memory when you run make index. On the other hand it is so big that it probably doesn't fit in cache, or probably only fits in caches of machines generously endowed. I have remarked that the difference of execution speed of make index between machines of similar speed but very different cache size (i am thinking Pentium 4 versus Core 2 Duo) is striking. It is less than 10 minutes on the big cache machine versus 30 minutes on the small cache one. If the cache effect is indeed dominant, then reducing the size of bsd.ports.mk by all possible means would be very beneficial. My suspicion is that the timne taken to read bsd.ports.mk from the hard disk is a rather small part of what takes it so long. Compare doing a "make" with a simple "cat" and the time differences are very great. I think make spends far more time processing the file than reading it. By the way an alternative system would be to use something other than make to do the job. This is what the Gentoo people are doing, using first a python based system, and now a C++ one (paludis). Here also i don't have any idea if it is faster. It would be nice to find a solution that doesn't involve redesigning the whole ports process from scratch. -- Michel TALON ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]" ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Looking for speed increases in "make index"
Stephen Montgomery-Smith said: > I suggest rewriting "make" so that variables are only evaluated on a > "need to know" basis. > or "I have tried to do this." Of course a lot of people have thinked about it, and quickly realized that it was not going to work. In the bsd.ports.mk, evaluation of one variable may be dependent on some conditional, which may itself be dependant on some other variable, appearing as some target. This constantly happens in bsd.ports.mk. If you think about that, you convince yourself that a "reduced make" needs to understand targets, needs to understand conditionals, and needs to evaluate not only the variable under consideration, but but possibly any other. In other words, you need a full blown make. To gain some performance, a first idea would be to simplify bsd.ports.mk. I am convinced that a substantial part of the 4000 lines are historical crap which serve no useful purpose. There are tons of variables who have probably purely anecdotical interest. There are targets which could be happily suppressed. Of course, due to the complexity of bsd.ports.mk, rewriting it is certainly not an easy task. There is a freebsd port whose aim is to rewrite it, i don't know how far they are. The NetBSD people have completely rewritten the system, i have no idea if the new one is faster. The OpenBSD people have also rewritten bsd.ports.mk, with apparently much faster makefile. A second idea would be to multithread make, since modern machines will have a lot of cores. At present make -j forks submakes and waits for their completion. This doesn't help for the problem at hand. However, multithreading the execution of a single makefile is certainly not trivial due to the interdependencies. Anyways, one of the things which cannot be a big factor is reading bsd.ports.mk from hard disk. It is certainly cached in memory when you run make index. On the other hand it is so big that it probably doesn't fit in cache, or probably only fits in caches of machines generously endowed. I have remarked that the difference of execution speed of make index between machines of similar speed but very different cache size (i am thinking Pentium 4 versus Core 2 Duo) is striking. It is less than 10 minutes on the big cache machine versus 30 minutes on the small cache one. If the cache effect is indeed dominant, then reducing the size of bsd.ports.mk by all possible means would be very beneficial. By the way an alternative system would be to use something other than make to do the job. This is what the Gentoo people are doing, using first a python based system, and now a C++ one (paludis). Here also i don't have any idea if it is faster. -- Michel TALON ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Looking for speed increases in "make index" and pkg_version for ports
Not quite what you asked for but... Given the size and complexity of the port system I have long felt that rather than do everything via more and more complex Mk/*.mk what is is needed is a ports server and a thin CLI frontend to it. This server can store dependency data in an efficient manner, deal with conditional dependencies, port renames, security and what not. It can build or fetch or serve packages, handle updates etc. Things mentioned in UPDATING file can instead be done by the server. In general it can automate a lot of stuff, remove error prone redundancies etc. If it is small enough and written in C, it can even be shipped with the base system instead of various pkg_* programs. It can provide two interfaces, one for normal users (with commands like add, check, config, delete, info, search, update, which) and one for port developers (command for adding/remove/renaming ports, etc.). Initially it must work with existing Makefiles. > I have been thinking a lot about looking for speed increases for "make > index" and pkg_version and things like that. So for example, in > pkg_version, it calls "make -V PKGNAME" for every installed package. > Now "make -V PKGNAME" should be a speedy operation, but the make has to > load in and analyze bsd.port.mk, a quite complicated file with about > 200,000 characters in it, when all it is needing to do is to figure out > the value of the variable PKGNAME. > > I suggest rewriting "make" so that variables are only evaluated on a > "need to know" basis. So, for example, if all we need to know is > PKGNAME, there is no need to evaluate, for example, _RUN_LIB_DEPENDS, > unless the writer of that particular port has done something like having > PORTNAME depend on the value of _RUN_LIB_DEPENDS. So "make" should > analyze all the code it is given, and only figure it out if it is needed > to do so. This would include, for example, figuring out .for and .if > directives on a need to know basis as well. > > I have only poked around a little inside the source for make, but I have > a sense that this would be a major undertaking. I certainly have not > thought through what it entails in more than a cursory manner. However > I am quite excited about the possibility of doing this, albeit I may > well put off the whole thing for a year or two or even forever depending > upon other priorities in my life. > > However, in the mean time I want to throw this idea out there to get > some feedback, either of the form of "this won't work," or of the form > "I will do it," or "I have tried to do this." > > Best regards, Stephen > ___ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "[EMAIL PROTECTED]" ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Looking for speed increases in "make index" and pkg_version for ports
On Sun, May 27, 2007 at 03:52:16PM -0500, Stephen Montgomery-Smith wrote:
> I have been thinking a lot about looking for speed increases for "make
> index" and pkg_version and things like that. So for example, in
> pkg_version, it calls "make -V PKGNAME" for every installed package. Now
> "make -V PKGNAME" should be a speedy operation, but the make has to load in
> and analyze bsd.port.mk, a quite complicated file with about 200,000
> characters in it, when all it is needing to do is to figure out the value of
> the variable PKGNAME.
I have a related question, pertaining to "make all-depends-list" and the
utter atrocity that is the make variable ALL-DEPENDS-LIST. If you don't
know what it is, look for ^ALL-DEPENDS-LIST around line 5175, in
bsd.ports.mk.
I call it an atrocity because it's a mix of make variable expansion
combined with sh scripting, and it's nearly impossible to read. It's
not commented either, so folks like myself are left thinking "What IS
this mess?!". It's expanded via $$(${ALL-DEPENDS-LIST}) in for loops,
throughout several places in bsd.port.mk.
I do not entirely understand what ALL-DEPENDS-LIST is about (that should
be apparent), but upon performing some of my benchmarks, I found this to
be a very slow piece of bsd.port.mk. make -V _DEPEND_DIRS is incredibly
fast, but ALL-DEPENDS-LIST is not.
Does it need to be done this way? Can we just iterate through all of
the ports, call make -V _DEPEND_DIRS, then sort | uniq the results? I
suppose that depends on the operation (make vs. make clean vs.
others)...
The port I used for testing some of the benchmarks was net/gacxtool. It
seems to be a good example of a "hefty" port.
> I suggest rewriting "make" so that variables are only evaluated on a "need
> to know" basis. So, for example, if all we need to know is PKGNAME, there
> is no need to evaluate, for example, _RUN_LIB_DEPENDS, unless the writer of
> that particular port has done something like having PORTNAME depend on the
> value of _RUN_LIB_DEPENDS. So "make" should analyze all the code it is
> given, and only figure it out if it is needed to do so. This would include,
> for example, figuring out .for and .if directives on a need to know basis as
> well.
This sounds like a good solution. In fact, I'm lead to believe that
heavy reliance on /bin/sh is part of why the ports collection is slow.
No, it's not the sole reason, but it's one of many. I'm of the belief
that anything we can do to migrate portions into native make would be
benefitial.
That said, I'll ask this out in the open: am I the only one who sees the
benefit of GNU make in regards to this? There's a lot of built-in
functions in GNU make which could help in regards to ports. I have no
qualms with PMake per se, but if another tool gives us what we need,
then maybe we should consider the pros and cons of adapting that.
There's also CMake, which is incredibly fast.
--
| Jeremy Chadwickjdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Looking for speed increases in "make index" and pkg_version for ports
I have been thinking a lot about looking for speed increases for "make index" and pkg_version and things like that. So for example, in pkg_version, it calls "make -V PKGNAME" for every installed package. Now "make -V PKGNAME" should be a speedy operation, but the make has to load in and analyze bsd.port.mk, a quite complicated file with about 200,000 characters in it, when all it is needing to do is to figure out the value of the variable PKGNAME. I suggest rewriting "make" so that variables are only evaluated on a "need to know" basis. So, for example, if all we need to know is PKGNAME, there is no need to evaluate, for example, _RUN_LIB_DEPENDS, unless the writer of that particular port has done something like having PORTNAME depend on the value of _RUN_LIB_DEPENDS. So "make" should analyze all the code it is given, and only figure it out if it is needed to do so. This would include, for example, figuring out .for and .if directives on a need to know basis as well. I have only poked around a little inside the source for make, but I have a sense that this would be a major undertaking. I certainly have not thought through what it entails in more than a cursory manner. However I am quite excited about the possibility of doing this, albeit I may well put off the whole thing for a year or two or even forever depending upon other priorities in my life. However, in the mean time I want to throw this idea out there to get some feedback, either of the form of "this won't work," or of the form "I will do it," or "I have tried to do this." Best regards, Stephen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: SoC: Distributed Audit Daemon project
On 27/05/07, Bjoern A. Zeeb <[EMAIL PROTECTED]> wrote: Having a generic, more secure and reliable (local) logging mechnism should be discussed in at least another thread. You may as well think of taking this idea to IETF as RFC 3164 lives there as a Memo these days and it might be a general enough thing for everyone. I wonder if there are no lightwight but more secure and reliable implementations out there already. Maybe time for some research. http://cr.yp.to/daemontools/multilog.html Obviously the license would require somebody to re-implement it under a BSD license and add networking capabilities, but the design is sound. I use it everywhere in place of the stock syslog (takes some re-architecturing - there's no longer a centralised syslog daemon). MC ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: SoC: Distributed Audit Daemon project
On Sun, 27 May 2007, Benjamin Lutz wrote: Hi, [ log shippping daemon for audit and other other logs ] [ syslog problems ] sorry I have to cut this short;) What Alexey said - this will be about log shipping not about writing single log records etc. Your syslog problems are outside the scope of this topic as long as it does not come to 'shipping' syslogged log files, i.e. after newsyslog rotated them, to another machine and for that hooking into the distributed log (shipping) daemon. Having a generic, more secure and reliable (local) logging mechnism should be discussed in at least another thread. You may as well think of taking this idea to IETF as RFC 3164 lives there as a Memo these days and it might be a general enough thing for everyone. I wonder if there are no lightwight but more secure and reliable implementations out there already. Maybe time for some research. /bz -- Bjoern A. Zeeb bzeeb at Zabbadoz dot NeT ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: SoC: Distributed Audit Daemon project
On Saturday 26 May 2007 09:49, Alexey Mikhailov wrote: > On Friday 25 May 2007 22:04:34 Benjamin Lutz wrote: > > On Friday 25 May 2007 01:22:21 Alexey Mikhailov wrote: > > > [...] > > > 2. As I said before initial subject of this project was > > > "Distributed audit daemon". But after some discussions we had > > > decided that this project can be done in more general maner. We > > > can perform distributed logging for any user-space app. > > > [...] > > > > This sounds very similar to syslogd. Is it feasible to make dlogd a > > drop-in replacement for syslogd, at least from a > > syslog-using-program point of view? > > Our project concentrates on log shipping. We're paying most attention > to securely and reliable log ships. So our project differs from > syslogd in major way. > > But actually it could be possible to be dlogd used by > syslogd\syslog-ng for logs shipping, as I see it. The thing that bugs me most about syslog is not even the transport to remote syslogd instances; that's relatively easy to fix (put some SSL between the daemons, or use encrypted tunnels, etc). It's that when a process logs a syslog event, it can claim to be anything at all. Iirc, it can even give a bogus timestamp. So what I was hoping for here is for auditd to come with a hook that intercepts syslog(3) calls, adds/validates pid, process name and timestamp, and then puts that information somewhere (some local log, a remote log, a lineprinter). It doesn't even have to give the information back to a syslogd daemon; whatever auditd uses for itself would be fine too. What I'm hoping for here is some way to get a guarantee that the information in a log is actually correct. The way it is at the moment, syslog messages are way too trivial to spoof. Anyway, this is just a feature wish :) I'm happy to see you work on auditd, whether or not it contains these syslog bits. Cheers Benjamin pgpawCcXyt3p8.pgp Description: PGP signature
