Re: [CVS] RPM: rpm/lib/ fsm.c
On Sat, Dec 20, 2008, Per Øyvind Karlsen wrote: [...] fsm.c: In function 'fsmMapFContext': fsm.c:772: warning: unused variable 'st' This one? Yes, exactly! I get it on linux as well, but I never figured out why it complained about st being unused since it seemingly did get used two lines below, but my ignorance prolly' can't really debate much with technical details.. Could you clarify on why? :) The point is that the variable is used on the matchpathcon() function call only. But matchpathcon() is a _macro_ which is defined only for Linux's SELINUX situation. On non-Linux (or as in your case, Linux but not SELINUX-enabled platform) the matchpathcon() macro expands to nothing which in turn leads to the unused variable situation. Ralf S. Engelschall r...@engelschall.com www.engelschall.com __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: [CVS] RPM: rpm/lib/ fsm.c
2008/12/20 Ralf S. Engelschall rse+rpm-de...@rpm5.orgrse%2brpm-de...@rpm5.org On Sat, Dec 20, 2008, Per Øyvind Karlsen wrote: [...] fsm.c: In function 'fsmMapFContext': fsm.c:772: warning: unused variable 'st' This one? Yes, exactly! I get it on linux as well, but I never figured out why it complained about st being unused since it seemingly did get used two lines below, but my ignorance prolly' can't really debate much with technical details.. Could you clarify on why? :) The point is that the variable is used on the matchpathcon() function call only. But matchpathcon() is a _macro_ which is defined only for Linux's SELINUX situation. On non-Linux (or as in your case, Linux but not SELINUX-enabled platform) the matchpathcon() macro expands to nothing which in turn leads to the unused variable situation. Ah, no SELINUX here, so I guess that explains things then. :)
Re: [CVS] RPM: rpm/lib/ fsm.c
On Wed, Jul 18, 2007, Jeff Johnson wrote: On Jul 18, 2007, at 10:14 AM, Ralf S. Engelschall wrote: On Wed, Jul 18, 2007, Jeff Johnson wrote: Ripping the need for -lpthread is on my todo list. Easier to rip than discuss. Well, a simple --without-pthreads now does the trick just fine and allows one to build RPM without any needs for an existing POSIX pthreads API. That's all I personally needed... Is --without-pthreads sufficient in all possibl;e cases? Well, I use it in %standalone and it seems to at least build and run just fine. But I've not done any in-depth testing until now as I'm still integrating RPM 5 into OpenPKG CURRENT (I have no real non-OpenPKG .spec files around for playing under non-Linux ;-) [...] I have been linking -lpthreads in rpm because beecrypt uses threads to gather entropy, and because of posix-shared-mutexes. BeeCrypt when built with --disable-threads uses a /dev/random just fine AFAIK. And the POSIX mutexes in Berkeley-DB I usually avoid and use the more conservative and portable UNIX/fcntl locking What I don't know generally is whether an executable needs -lpthread if a library uses -lpthread. The general answer (of course) needs to investigate code paths deeply. E.g. there is much compexity added to the signal handler so that rpmlib (through rpm-python) can be run on a thread. Well, first, a standalone libpthread is not a generic thing. On some platforms POSIX pthreads is directly in libc or magically activated otherwise. The most portable way is currently to use gcc and its -pthread option: a small abstraction which does the right thing on the various platforms. Second, if a library is using the POSIX pthread API, the application actually has to link against it, too. Because POSIX pthreads is a lot more than just a few #defines, so one usually always has to link against the same library. On smart platforms the inter-library dependency from libfoo to a libpthread is enough. Then it is not really necessary that the application also links against libpthread explicitly or even includes pthread.h. The hint is: POSIX pthread doesn't have an init or setup call, so _ALL_ implementations have to do an _implicit_ initialization (without an explicit call by the application's main) anyway. That's why one usually doesn't have to do anything if a library is using POSIX pthreads. There are two approaches to sanity without general answers that are not forthcoming. My previous answer was -lpthreads always, the alternative flexible answer is expunge every use of thread functions, the path I am now pursuing. We don't have to really get rid of the thread functions. Being able to disable them on-demand is sufficient IMHO... Ralf S. Engelschall [EMAIL PROTECTED] www.engelschall.com __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: [CVS] RPM: rpm/lib/ fsm.c
On Wed, Jul 18, 2007, Jeff Johnson wrote: [...] If I read correctly, the conservative/portable approach is -lpthread in executable so that external libraries linked with -lpthread function correctly. That kinda forces mandatory -lpthread for rpm when external libraries may (or may not) have -lpthread. This is for what pkg-config(1) or a foo-config(1) of the third-library is for: that RPM doesn't have to know this or _always_ try to link against pthreads. And for the same reason why created our rpm.pc, btw. But this pthread mess is a general problem, yes. [...] The path forward is adding per-object thread locks, the path backwards is removing -lpthread, the status quo needs to change no matter what. I'm always a little bit surprised that people use POSIX pthread mutex even in non-multithreaded applications. Sure, can be done. But that it really should be 10 times faster than some other mutex approaches I'm sceptical about -- perhaps for one single OS, ok. What is the purpose of RPM locking its database operations? That no second RPM instance (either rpm(1) or an RPM API based application) trashes it while one RPM is already operating, right? At least for this one doesn't need POSIX pthread mutexes. Here an fcntl(3) based approach even within RPM itself should be both sufficient and fast enough... Or are you planning that RPM can be really internally run multi-threaded? Then the case is different, of course. Then you _have_ to use POSIX pthreads, of course. Ralf S. Engelschall [EMAIL PROTECTED] www.engelschall.com __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: [CVS] RPM: rpm/lib/ fsm.c
On Jul 18, 2007, at 11:11 AM, Ralf S. Engelschall wrote: On Wed, Jul 18, 2007, Jeff Johnson wrote: [...] If I read correctly, the conservative/portable approach is - lpthread in executable so that external libraries linked with -lpthread function correctly. That kinda forces mandatory -lpthread for rpm when external libraries may (or may not) have -lpthread. This is for what pkg-config(1) or a foo-config(1) of the third-library is for: that RPM doesn't have to know this or _always_ try to link against pthreads. And for the same reason why created our rpm.pc, btw. But this pthread mess is a general problem, yes. Mess indeed. [...] The path forward is adding per-object thread locks, the path backwards is removing -lpthread, the status quo needs to change no matter what. I'm always a little bit surprised that people use POSIX pthread mutex even in non-multithreaded applications. Sure, can be done. But that it really should be 10 times faster than some other mutex approaches I'm sceptical about -- perhaps for one single OS, ok. The private report of 10x performance increase was from Keith Bostic. ;-) The benchmark context is what is confusing: locking overhead is 10x faster, the wall clock continues to click along at exactly the same rate. The real benefit is unifying process - thread locks to reduce the impediment to attempting multi-threaded. Changing rpm's locking scheme in order to pursue a surely controversial multi-threaded rpmlib implementation Chimera Just Ain't Going To Happen. What is the purpose of RPM locking its database operations? That no second RPM instance (either rpm(1) or an RPM API based application) trashes it while one RPM is already operating, right? At least for this one doesn't need POSIX pthread mutexes. Here an fcntl(3) based approach even within RPM itself should be both sufficient and fast enough... See lib/rpmlock.c for a perfectly sane shared/exclusive lock used (optionally but flexibly! ;-) by rpmlib. Work done by Gustavo Niemeyer. The killer is that there is too much flexibility in choice of the path to the fcntl lock, and the locking scope is imperfectly specified as well. E.g. the RedHat beehive build system used to have all rpmdb's in all chroot's open, thereby having outer - inner chroot lock contention that no path to file with a fcntl lock can ever solve. Well a fd can be passed from parent - child in the usal ways, let's not go there. The confusion in rpm comes when the Berkeley DB, not the transaction lock, is abused for rpm inter-process locking. This was a design flaw in the original rpm implementation. Which reminds me: the lockdbfd option needs to be fixed in rpmdb/db3.c. What is there now does not honor the lock, only warns. That was the correct behavior when rpm4 - legacy rpm3 applications were competing for an rpm lock, but otherwise is entirely the wrong thing to do is lockdbfd is going to be abused for rpm transaction locking purposes. Or are you planning that RPM can be really internally run multi-threaded? Then the case is different, of course. Then you _have_ to use POSIX pthreads, of course. rpm is already multi-threaded many years now: rpm -Uvh --psmthreads *.rpm or rpm -Uvh --fsmthreads *.rpm The current rpm implementation is rather uselessly frozen in amber atm, and needs to change. I quote myself: The path forward is adding per-object thread locks, the path backwards is removing -lpthread, the status quo needs to change no matter what. 73 de Jeff __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: [CVS] RPM: rpm/lib/ fsm.c
On Jul 18, 2007, at 12:39 PM, Ralf S. Engelschall wrote: On Wed, Jul 18, 2007, Jeff Johnson wrote: [...] Or are you planning that RPM can be really internally run multi-threaded? Then the case is different, of course. Then you _have_ to use POSIX pthreads, of course. rpm is already multi-threaded many years now: rpm -Uvh --psmthreads *.rpm or rpm -Uvh --fsmthreads *.rpm Oh, this you mean. Yes, for this we already need POSIX pthread mutexes. There using UNIX/fcntl would be bogus, of course. Sorry, I overlooked. The implementation is feeble, was used to assess how many hacks need to be removed (alternatively how many locks need to be added). Look for UNSAFE wrapper. The results were quite promising. Meanwhile, ripping the wrapper is easier than making rpmlib thread-safe. 73 de Jeff __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: [CVS] RPM: rpm/lib/ fsm.c
On Tue, Jun 19, 2007, Jeff Johnson wrote: On Jun 19, 2007, at 2:42 AM, Ralf S. Engelschall wrote: RPM Package Manager, CVS Repository http://rpm5.org/cvs/ Server: rpm5.org Name: Ralf S. Engelschall Root: /v/rpm/cvs Email: [EMAIL PROTECTED] Module: rpm Date: 19-Jun-2007 08:42:10 Branch: HEAD Handle: 2007061907421000 Modified files: rpm/lib fsm.c Log: fix an 'unused variable' warning and fix usage of HAVE_MMAP (actually unused as no Autoconf glue exists anyway) So *that's* why rpm installs survived the 2.6.19 mmap(2) borkenness. I've been wondering. Getting mmap enabled is likely a large performance win. Maybe, yes. I've now added the missing Autoconf glue AC_FUNC_MMAP for checking whether the system supports a reasonable mmap(2). Let's see what happens... Ralf S. Engelschall [EMAIL PROTECTED] www.engelschall.com __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: [CVS] RPM: rpm/lib/ fsm.c
On Jun 19, 2007, at 12:20 PM, Ralf S. Engelschall wrote: On Tue, Jun 19, 2007, Jeff Johnson wrote: Getting mmap enabled is likely a large performance win. Maybe, yes. I've now added the missing Autoconf glue AC_FUNC_MMAP for checking whether the system supports a reasonable mmap(2). Let's see what happens... Profiled with -pg 4+ years ago, mmap was 20% faster than straight I/O loops. But yes, will be interesting to see what happens. On the roadmap somewhere is sparse file support, reestablishing holes in installed files precisely, a request from the linux ELF cabal. 73 de Jeff __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
