Re: [CVS] RPM: rpm/lib/ fsm.c

2008-12-20 Thread Ralf S. Engelschall
On Sat, Dec 20, 2008, Per Øyvind Karlsen wrote:

 [...]
 fsm.c: In function 'fsmMapFContext':
 fsm.c:772: warning: unused variable
 'st'

 This one?

Yes, exactly!

 I get it on linux as well, but I never figured out why it complained
 about st being unused since it seemingly did get used two lines below,
 but my ignorance prolly' can't really debate much with technical
 details..

 Could you clarify on why? :)

The point is that the variable is used on the matchpathcon() function
call only. But matchpathcon() is a _macro_ which is defined only for
Linux's SELINUX situation. On non-Linux (or as in your case, Linux
but not SELINUX-enabled platform) the matchpathcon() macro expands to
nothing which in turn leads to the unused variable situation.

   Ralf S. Engelschall
   r...@engelschall.com
   www.engelschall.com

__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org


Re: [CVS] RPM: rpm/lib/ fsm.c

2008-12-20 Thread Per Øyvind Karlsen
2008/12/20 Ralf S. Engelschall rse+rpm-de...@rpm5.orgrse%2brpm-de...@rpm5.org


 On Sat, Dec 20, 2008, Per Øyvind Karlsen wrote:

  [...]
  fsm.c: In function 'fsmMapFContext':
  fsm.c:772: warning: unused variable
  'st'
 
  This one?

 Yes, exactly!

  I get it on linux as well, but I never figured out why it complained
  about st being unused since it seemingly did get used two lines below,
  but my ignorance prolly' can't really debate much with technical
  details..
 
  Could you clarify on why? :)

 The point is that the variable is used on the matchpathcon() function
 call only. But matchpathcon() is a _macro_ which is defined only for
 Linux's SELINUX situation. On non-Linux (or as in your case, Linux
 but not SELINUX-enabled platform) the matchpathcon() macro expands to
 nothing which in turn leads to the unused variable situation.

 Ah, no SELINUX here, so I guess that explains things then. :)


Re: [CVS] RPM: rpm/lib/ fsm.c

2007-07-18 Thread Ralf S. Engelschall
On Wed, Jul 18, 2007, Jeff Johnson wrote:

 On Jul 18, 2007, at 10:14 AM, Ralf S. Engelschall wrote:

 On Wed, Jul 18, 2007, Jeff Johnson wrote:

 Ripping the need for -lpthread is on my todo list. Easier to rip than
 discuss.

 Well, a simple --without-pthreads now does the trick just fine and
 allows one to build RPM without any needs for an existing POSIX pthreads
 API. That's all I personally needed...

 Is --without-pthreads sufficient in all possibl;e cases?

Well, I use it in %standalone and it seems to at least build and run
just fine. But I've not done any in-depth testing until now as I'm still
integrating RPM 5 into OpenPKG CURRENT (I have no real non-OpenPKG .spec
files around for playing under non-Linux ;-)

 [...]
 I have been linking -lpthreads in rpm because beecrypt uses threads to
 gather entropy, and because of posix-shared-mutexes.

BeeCrypt when built with --disable-threads uses a /dev/random just fine
AFAIK. And the POSIX mutexes in Berkeley-DB I usually avoid and use the
more conservative and portable UNIX/fcntl locking

 What I don't know generally is whether an executable needs -lpthread if
 a library uses -lpthread. The general answer (of course) needs to
 investigate code paths deeply. E.g. there is much compexity added
 to the signal handler so that rpmlib (through rpm-python) can be run
 on a thread.

Well, first, a standalone libpthread is not a generic thing. On some
platforms POSIX pthreads is directly in libc or magically activated
otherwise. The most portable way is currently to use gcc and its
-pthread option: a small abstraction which does the right thing on the
various platforms.

Second, if a library is using the POSIX pthread API, the application
actually has to link against it, too. Because POSIX pthreads is a lot
more than just a few #defines, so one usually always has to link against
the same library. On smart platforms the inter-library dependency
from libfoo to a libpthread is enough. Then it is not really necessary
that the application also links against libpthread explicitly or even
includes pthread.h. The hint is: POSIX pthread doesn't have an init
or setup call, so _ALL_ implementations have to do an _implicit_
initialization (without an explicit call by the application's main)
anyway. That's why one usually doesn't have to do anything if a library
is using POSIX pthreads.

 There are two approaches to sanity without general answers that are
 not forthcoming. My previous answer was -lpthreads always, the alternative
 flexible answer is expunge every use of thread functions, the path I am
 now pursuing.

We don't have to really get rid of the thread functions. Being able
to disable them on-demand is sufficient IMHO...

   Ralf S. Engelschall
   [EMAIL PROTECTED]
   www.engelschall.com

__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org


Re: [CVS] RPM: rpm/lib/ fsm.c

2007-07-18 Thread Ralf S. Engelschall
On Wed, Jul 18, 2007, Jeff Johnson wrote:

 [...]
 If I read correctly, the conservative/portable approach is -lpthread in
 executable so that external libraries linked with -lpthread function
 correctly. That kinda forces mandatory -lpthread for rpm when
 external libraries may (or may not) have -lpthread.

This is for what pkg-config(1) or a foo-config(1) of the third-library
is for: that RPM doesn't have to know this or _always_ try to link
against pthreads. And for the same reason why created our rpm.pc, btw.
But this pthread mess is a general problem, yes.

 [...]
 The path forward is adding per-object thread locks, the path backwards is
 removing -lpthread, the status quo needs to change no matter what.

I'm always a little bit surprised that people use POSIX pthread mutex
even in non-multithreaded applications. Sure, can be done. But that it
really should be 10 times faster than some other mutex approaches I'm
sceptical about -- perhaps for one single OS, ok.

What is the purpose of RPM locking its database operations? That no
second RPM instance (either rpm(1) or an RPM API based application)
trashes it while one RPM is already operating, right? At least for this
one doesn't need POSIX pthread mutexes. Here an fcntl(3) based approach
even within RPM itself should be both sufficient and fast enough...

Or are you planning that RPM can be really internally run
multi-threaded? Then the case is different, of course. Then you _have_
to use POSIX pthreads, of course.

   Ralf S. Engelschall
   [EMAIL PROTECTED]
   www.engelschall.com

__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org


Re: [CVS] RPM: rpm/lib/ fsm.c

2007-07-18 Thread Jeff Johnson


On Jul 18, 2007, at 11:11 AM, Ralf S. Engelschall wrote:


On Wed, Jul 18, 2007, Jeff Johnson wrote:


[...]
If I read correctly, the conservative/portable approach is - 
lpthread in

executable so that external libraries linked with -lpthread function
correctly. That kinda forces mandatory -lpthread for rpm when
external libraries may (or may not) have -lpthread.


This is for what pkg-config(1) or a foo-config(1) of the third-library
is for: that RPM doesn't have to know this or _always_ try to link
against pthreads. And for the same reason why created our rpm.pc, btw.
But this pthread mess is a general problem, yes.



Mess indeed.


[...]
The path forward is adding per-object thread locks, the path  
backwards is

removing -lpthread, the status quo needs to change no matter what.


I'm always a little bit surprised that people use POSIX pthread mutex
even in non-multithreaded applications. Sure, can be done. But that it
really should be 10 times faster than some other mutex approaches I'm
sceptical about -- perhaps for one single OS, ok.



The private report of 10x performance increase was from Keith  
Bostic. ;-)


The benchmark context is what is confusing: locking overhead is 10x  
faster, the wall clock

continues to click along at exactly the same rate.

The real benefit is unifying process - thread locks to reduce the
impediment to attempting multi-threaded. Changing rpm's locking
scheme in order to pursue a surely controversial multi-threaded
rpmlib implementation Chimera Just Ain't Going To Happen.


What is the purpose of RPM locking its database operations? That no
second RPM instance (either rpm(1) or an RPM API based application)
trashes it while one RPM is already operating, right? At least for  
this
one doesn't need POSIX pthread mutexes. Here an fcntl(3) based  
approach

even within RPM itself should be both sufficient and fast enough...



See lib/rpmlock.c for a perfectly sane shared/exclusive lock used  
(optionally

but flexibly! ;-) by rpmlib. Work done by Gustavo Niemeyer. The killer
is that there is too much flexibility in choice of the path to the fcntl
lock, and the locking scope is imperfectly specified as well. E.g.
the RedHat beehive build system used to have all rpmdb's in
all chroot's open, thereby having outer - inner chroot lock contention
that no path to file with a fcntl lock can ever solve. Well a fd can be
passed from parent - child in the usal ways, let's not go there.

The confusion in rpm comes when the Berkeley DB, not the transaction
lock, is abused for rpm inter-process locking. This was a design flaw
in the original rpm implementation.

Which reminds me: the lockdbfd option needs to be fixed in
rpmdb/db3.c. What is there now does not honor the lock, only
warns. That was the correct behavior when rpm4 - legacy rpm3
applications were competing for an rpm lock, but otherwise is
entirely the wrong thing to do is lockdbfd is going to be abused
for rpm transaction locking purposes.


Or are you planning that RPM can be really internally run
multi-threaded? Then the case is different, of course. Then you _have_
to use POSIX pthreads, of course.



rpm is already multi-threaded many years now:
 rpm -Uvh --psmthreads *.rpm
or
rpm -Uvh --fsmthreads *.rpm

The current rpm implementation is rather uselessly frozen in amber  
atm, and

needs to change. I quote myself:

The path forward is adding per-object thread locks, the path  
backwards is

removing -lpthread, the status quo needs to change no matter what.

73 de Jeff
__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org


Re: [CVS] RPM: rpm/lib/ fsm.c

2007-07-18 Thread Jeff Johnson


On Jul 18, 2007, at 12:39 PM, Ralf S. Engelschall wrote:


On Wed, Jul 18, 2007, Jeff Johnson wrote:


[...]

Or are you planning that RPM can be really internally run
multi-threaded? Then the case is different, of course. Then you  
_have_

to use POSIX pthreads, of course.


rpm is already multi-threaded many years now:
 rpm -Uvh --psmthreads *.rpm
or
rpm -Uvh --fsmthreads *.rpm


Oh, this you mean. Yes, for this we already need POSIX pthread  
mutexes.

There using UNIX/fcntl would be bogus, of course. Sorry, I overlooked.



The implementation is feeble, was used to assess how many hacks need
to be removed (alternatively how many locks need to be added).

Look for UNSAFE wrapper. The results were quite promising.

Meanwhile, ripping the wrapper is easier than making rpmlib thread-safe.

73 de Jeff
__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org


Re: [CVS] RPM: rpm/lib/ fsm.c

2007-06-19 Thread Ralf S. Engelschall
On Tue, Jun 19, 2007, Jeff Johnson wrote:

 On Jun 19, 2007, at 2:42 AM, Ralf S. Engelschall wrote:

   RPM Package Manager, CVS Repository
   http://rpm5.org/cvs/

 

   Server: rpm5.org Name:   Ralf S. Engelschall
   Root:   /v/rpm/cvs   Email:  [EMAIL PROTECTED]
   Module: rpm  Date:   19-Jun-2007 08:42:10
   Branch: HEAD Handle: 2007061907421000

   Modified files:
 rpm/lib fsm.c

   Log:
 fix an 'unused variable' warning and fix usage of HAVE_MMAP (actually
 unused as no Autoconf glue exists anyway)


 So *that's* why rpm installs survived the 2.6.19 mmap(2) borkenness.
 I've been wondering.

 Getting mmap enabled is likely a large performance win.

Maybe, yes. I've now added the missing Autoconf glue AC_FUNC_MMAP for
checking whether the system supports a reasonable mmap(2). Let's see
what happens...
   Ralf S. Engelschall
   [EMAIL PROTECTED]
   www.engelschall.com

__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org


Re: [CVS] RPM: rpm/lib/ fsm.c

2007-06-19 Thread Jeff Johnson


On Jun 19, 2007, at 12:20 PM, Ralf S. Engelschall wrote:


On Tue, Jun 19, 2007, Jeff Johnson wrote:



Getting mmap enabled is likely a large performance win.


Maybe, yes. I've now added the missing Autoconf glue AC_FUNC_MMAP for
checking whether the system supports a reasonable mmap(2). Let's see
what happens...


Profiled with -pg 4+ years ago, mmap was 20% faster than straight I/O  
loops.


But yes, will be interesting to see what happens.

On the roadmap somewhere is sparse file support, reestablishing holes
in installed files precisely, a request from the linux ELF cabal.

73 de Jeff
__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org