Package type in rpm lead wrong?
Hi guys, the "type" in a rpm package created with rpm5 always seems to be "0" (binary rpm), even if it's a source rpm. This confuses tools like 'file' that determine the rpm type from the lead. Is that an intended change or did I stumble over a bug? Thanks, Michael. -- Michael Schroeder m...@suse.de SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: EVR issues: set:versions, epoch-as-string, now twiddle-in-version
On Fri, Jun 22, 2012 at 09:39:22PM +0400, Alexey Tourbin wrote: > What is a probability? (That's the second stunning questions after > what is a number.) If numbers were unlimited, they could have > represented symbols exactly. The only reason the numbers "clash" is > because they "sit" within a limited range. What you have to do to > control probabilities is to select the appropriate range, which is a > trade-off between how many clashes are possible and how much bits per > number you are allowed to take. The basic idea is that, to encode a > Provides version, you first need to know how many symbols you are > going to encode, say 1024. You then ask yourself, how many bits per > symbol should I take. If you take e.g. only 10 bits, that's plain > stupid, because 10-bit numbers can address only 1024 values (so you > end up with most of the bits set, or most of the numbers taken, that > is to say). If you take 20 bits per symbol, though, things get more > interesting. 20-bit numbers can address 2^{20} different symbols. > Since the probability is due to range limitations, you get > 2^{10}/2^{20}=0.1% false positive rate, which is basically what you > want. When encoding a Requires version, it's another matter: you > should use the same number of bits as the corresponding Requires > versions have. You mean the corresponding Provides, i.e. the symbols in the "target" library, right? Basically you should never need to downsample when doing the set comparison, as downsampling kills your probablilities. Cheers, Michael. -- Michael Schroeder m...@suse.de SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: EVR issues: set:versions, epoch-as-string, now twiddle-in-version
On Wed, May 30, 2012 at 02:56:20PM -0400, Jeffrey Johnson wrote: > > On Apr 23, 2012, at 10:32 AM, Jeff Johnson wrote: > > > I should point out that writing the attached > > message (and sending from the wrong e-mail address) has instantly > > led to a different -- and perhaps more natural -- syntax like > > > > Requires: set(libfoo.so.1) >= whatever > > > > After a month of quietly muddling, I like this namespace like syntax > better because it doesn't lead to magic tokens like "set:" in the EVR > string and is more consistent with existing namespaces in RPM dependencies. Is that just syntax? I.e., will rpm -q --whatprovides libfoo.so.1 still find the package? Cheers, Michael. -- Michael Schroeder m...@suse.de SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: Twiddle in Version (was Re: [CVS] RPM: rpm/ CHANGES rpm/rpmdb/ rpmdb.c)
On Mon, Nov 01, 2010 at 04:50:51PM -0400, Jeff Johnson wrote: > Consider this perfectly acceptable (if twiddle-in-version is implemented) > Provides: foo = ~~~ > > If that Provides: is installed in an rpmdb, the existing RPM EVR comparison > will see explicit existing members of the {E,V,R} triple that WILL start > matching with constructs like > Requires: foo = 1 > and (more importantly, yo added the fix in the past week) > Conflicts: foo = 1 Do you mean this matches now, or this will match with the changed algorithm? I don't see how there can be a match, the provides has E = 0 V = ~~~ R = missing The requires has E = 0 V = 1 R = missing "1" doesn't match "~~~". > The open question for a twiddle-in-version implementation that MUST be > answered is > > How should missing values be compared to "~" placeholders meaningfully? > > (aside) IMHO the ~ support should just affect rpmEVRcmp(), i.e. the function used to compare the E/V/R parts. Cheers, Michael. -- Michael Schroeder m...@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: Implementing EVR comparisons using *RE's ?
On Fri, May 08, 2009 at 10:21:48AM +0200, Ralf S. Engelschall wrote: > This way the whole version comparing stuff is fully pluggable, as one > can define how a version is parsed into its chunks and second how those > chunks are compared. Good luck with providing rpms for multiple distributions that use different version comparision settings... Having a fixed algorithm in place also has some benefits. Cheers, Michael. -- Michael Schroeder m...@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: Should Obsoletes: become persistent?
On Wed, May 06, 2009 at 01:43:36PM -0400, Jeff Johnson wrote: > (aside) > I ask this question (and other questions like whether SRPM > installs should be tracked) periodically, have for years. > This bug report triggers the duty/obligation to ask Yet Again: > > https://bugzilla.redhat.com/show_bug.cgi?id=499239 > > Obsoletes: are applied once, when the package that carries > the Obsoletes: is upgraded. > > Basically that leaves a back-door where a later install/upgrade > of the Obsolete:'d package(s) is permitted. The issue is periodically > re-discovered and invariably leads to > RPM sux! > conclusions. > > The proper "fix" is to make Obsoletes: persistent like Conflicts:, > with a semantic that any Obsolete:'d package added to a transaction > is automagically discarded (with or without warning with all the usual > enablers/disablers for the warm/fuzzy bikeshed discusssions). Uh, that's the wrong direction. Instead, you should add an erase transaction element to get rid of the installed package that contains the obsoletes. That way, going back to the old name is still possible. I.e. A: installed package B: new name of A, contains "Obsoletes: A" $ rpm -U B A is uninstalled, B gets installed $ rpm -U A B is uninstalled, A gets installed Cheers, Michael. -- Michael Schroeder m...@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: %post-script prerequisites
On Thu, Sep 25, 2008 at 02:37:27PM +0200, Pixel wrote: > Michael Schroeder <[EMAIL PROTECTED]> writes: > > > So A requires B and B has a requires(post) to A? This isn't installable > > at all, thus it's a packaging bug. > > i don't agree. > > not knowing exactly what rpmlib does, here is how i understand the pb: > > "Requires" is same as "Requires(postrans)", ie if A requires B, it > doesn't mean B must be installed before A. It means that B must be installed before A *works* (easiest case: B is a library). Thus, if C has a "Requires(post): A" (e.g. C calls A in a post scriptlet), A must work, so B and A have to be installed before C. Cheers, Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: %post-script prerequisites
On Thu, Sep 25, 2008 at 03:12:15PM +0400, Dmitry V. Levin wrote: > On Thu, Sep 25, 2008 at 11:08:31AM +0200, Michael Schroeder wrote: > [...] > > So A requires B and B as a requires(post) to A? This isn't installable > > at all, thus it's a packaging bug. > > Sorry, you cannot avoid the problem of resolving dependency loops > by just saying that dependency loops are evil, packaging bug, etc. I didn't say that. I said that dependency loops that include prereqs are evil. Please show me one example where they are needed. Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: %post-script prerequisites
On Wed, Sep 24, 2008 at 07:27:15PM +, Alexey Tourbin wrote: > On Wed, Sep 24, 2008 at 07:08:54PM +, Alexey Tourbin wrote: > > In package foo, program /usr/bin/foo is both packaged *and* called > > in its %post script. The program /usr/bin/foo runs /usr/bin/bar, > > for which we have the dependency "Requires: /usr/bin/bar". > > Here is similar example that does not require --noorder option > to demonstrate the problem. The difference is that packages A > and B have circular dependencies, so, unless we have "Requires(post)", > rpm choose to install A first, and its %post script fails. > > Name: A > Version: 1.0 > Release: 1 > Summary: A > License: GPL > Group: Development/Other > Requires: /usr/bin/B > #Requires(post): /usr/bin/B > BuildArch: noarch > AutoReqProv: no > %package -n B > Summary: B > Group: Development/Other > Requires: A > AutoReqProv: no > %description > %description -n B > %install > mkdir -p %buildroot/usr/bin > cat >%buildroot/usr/bin/A < /usr/bin/B > EOF > cat >%buildroot/usr/bin/B < : > EOF > chmod +x %buildroot/usr/bin/* > %files > /usr/bin/A > %files -n B > /usr/bin/B > %post > /usr/bin/A So A requires B and B as a requires(post) to A? This isn't installable at all, thus it's a packaging bug. A has B in its requires, so B must be completely installed before A works (maybe /usr/bin/A calls /usr/bin/B). But B needs /usr/bin/A in the postinstall. Cheers, Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: Conflicts on files not symmetric
On Fri, Sep 19, 2008 at 04:21:50PM +, Alexey Tourbin wrote: > Technically there's no piping, only a file duplicated on stdin. And > "filetriggers" are run only once, at the end of transaction (they're > actually "posttrans filetriggers"), which saves consecutive ldconfig, > gtk-update-icon-cache, or whatever calls. Uh, but can that work? A Prereq to another package basically says that the package must be fully configured before installation, so all triggers must be run. Post-transaction is a bit late... Cheers, Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: Conflicts on files not symmetric
On Fri, Sep 19, 2008 at 03:33:20PM +, Alexey Tourbin wrote: > On Fri, Sep 19, 2008 at 11:26:23AM -0400, Jeff Johnson wrote: > > Likely the 1st thing to get into place is the ability to trigger from > > adding > > a file to a directory, i.e. trigger if RPMTAG_DIRNAMES matches a trigger > > pattern, add trailing / to pattern to differentiate a dirname trigger. > > Triggers based on DIRNAMES are not enough. Here is an example: > when e.g. /usr/share/icons/hicolor/32x32/apps/kpdf.png is installed, > upgraded, or removed, "gtk-update-icon-cache /usr/share/icons/hicolor" > must be triggered. I'm thinking more in a different direction: change find-provides that it generates a "Provide: trigger:gtk_update_icon_cache" and add a trigger to the package containing gtk-update-icon-cache. The trigger should get the hdrid of the triggering package as argument, so that it can query the file list (rpm --querybynumber). It then can do it's magic with the matching files. My main goal is to get rid of most of the scriptlets, as packages tend to scrw up writing them. There could be a "trigger:useradd = foo" trigger, or a "trigger:service" trigger for /etc/init.d/ services. Dunno if it's really doable, though. Cheers, Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: Conflicts on files not symmetric
On Fri, Sep 19, 2008 at 02:52:21PM +, Alexey Tourbin wrote: > On Fri, Sep 19, 2008 at 11:07:04AM +0200, Michael Schroeder wrote: > > while implementing virtual triggers > [...] > May I perhaps take a look at where you are? > I'm implementing some sort of triggers, too. Oh, I'm not implementing new triggers, I'm just changing the current implementation so that they also trigger on package provides (and maybe the file list) and not just package NEVR. What are you working on? Cheers, Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Conflicts on files not symmetric
Hi rpm-4 and rpm-5, while implementing virtual triggers I stumbled over the fact that conflicts on files only work in one direction. Example: Package A contains /foo Package B has "Conflicts: /foo" If A is already installed and one tries to install B, a dependency conflict is reported: error: Failed dependencies: /foo conflicts with B-1-1.i586 If B is already installed and one tries to install A, no error is displayed and the package is installed. So, is this a known bug? Is it worth fixing? If this needs fixing, here's an idea: As it probably costs too much time to test each file in the file list for conflicts, how about creating a index of the conflicts first and only test the files that match the index? I.e. add a function char *rpmdbKeyIterator(rpmdbMatchIterator mi) { int rc, xx; dbiIndex dbi; dbi = dbiOpen(mi->mi_db, mi->mi_rpmtag, 0); if (mi->mi_dbc == NULL) xx = dbiCopen(dbi, dbi->dbi_txnid, &mi->mi_dbc, mi->mi_cflags); rc = dbiGet(dbi, mi->mi_dbc, &mi->mi_key, &mi->mi_data, DB_NEXT); if (rc) return 0; if (mi->mi_keyp) free((char *)mi->mi_keyp); mi->mi_keyp = (void *)xmalloc(mi->mi_key.size + 1); memcpy((char *)mi->mi_keyp, mi->mi_key.data, mi->mi_key.size); ((char *)mi->mi_keyp)[mi->mi_key.size] = 0; mi->mi_keylen = mi->mi_key.size; return (char *)mi->mi_keyp; } to rpmdb.c, then use mi = rpmtsInitIterator(ts, RPMTAG_CONFLICTNAME, 0, 0); while ((key = rpmdbKeyIterator(mi)) != 0) ... to create an array of basenames, sort it and then go through the filelist and match only if bsearch for the basename returns true. Thoughts? Cheers, Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: mktemp/mkstemp usage
On Wed, Jul 18, 2007 at 12:37:41PM +0400, Dmitry V. Levin wrote: > On Wed, Jul 18, 2007 at 10:07:27AM +0200, Michael Schroeder wrote: > > On Wed, Jul 18, 2007 at 02:04:11AM +0400, Dmitry V. Levin wrote: > > > No, please do not apply this change. > > > The problem is real, so let linker warnings remind us about this issue. > > The problem is *not* real, close(mkstemp()) looks a bit strange, but > > is sane. > > The issue is not only how secure temporary files are created, but also > whether environment is secure enough to pass temp files by their names > to external executables. %_tmppath aka /var/tmp has the sticky bit set so it is safe. Regarding %_specdir aka %_topdir/SPECS, SUSE also has a t-bit on it, but I'm not so sure about other distros. Cheers, Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: mktemp/mkstemp usage
On Wed, Jul 18, 2007 at 02:04:11AM +0400, Dmitry V. Levin wrote: > No, please do not apply this change. > The problem is real, so let linker warnings remind us about this issue. The problem is *not* real, close(mkstemp()) looks a bit strange, but is sane. Cheers, Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: mktemp/mkstemp usage
On Tue, Jul 17, 2007 at 10:19:48PM +0200, Ralf S. Engelschall wrote: > I'm currently investigating on the mktemp/mkstemp(3) front > and stumbled over IMHO totally broken uses in e.g. build.c: > > | tmpSpecFile = (char *) rpmGetPath("%{_specdir}/", > "rpm-spec.XX", NULL); > | #if defined(HAVE_MKSTEMP) > | (void) close(mkstemp(tmpSpecFile)); > | #else > | (void) mktemp(tmpSpecFile); > | #endif > | > | (void) isCompressed(arg, &res); > | > | cmd = alloca(strlen(arg) + 50 + strlen(tmpSpecFile)); > | sprintf(cmd, "%s < %s | tar xOvf - Specfile 2>&1 > %s", > | zcmds[res & 0x3], arg, tmpSpecFile); > > First, the mktemp/mkstemp _replace_ the "X" in the template argument > with random stuff, so it is bogus to first let a file be created by > mktemp/mkstemp(3) under e.g. ".../rpm-spec.NMNOPQ" and then actually > using the file ".../rpm-spec.XX". Or have I completely > missed some important point here? mkstemp modifies it's argument, so the right filename is used later on. > Second, the use of close(mkstemp()) instead of mktemp() serves just one > purpose, of course: to get rid of a linker warning that mktemp(3) us > unsafe. But using mkstemp this way is _AS UNSAFE_ as using mktemp. That's not true, mkstemp() really creates a file with O_EXCL, so you are sure that you own the file afterwards. mktemp() doesn't create a file, so there is a race if you create it later on. Cheers, Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: Removing the rpm dependency on beecrypt
On Mon, Jul 16, 2007 at 08:14:42AM -0400, Jeff Johnson wrote: > rpm is a mechanism, not policy, implementation of signature checking. > > There has never been any attempt at permitting/denying installation > of packages signed with pubkey X. > > In fact, quite the opposite is the normal policy signature/digest > policy has > been implemented by yum, anaconda and apt-rpm. > > One can argue that a tool should do one thing well. In the case > of rpm that is installing files on files systems. > > Furthermore, the only mechanism to "allow" installation of signed > packages is to deny > all non-signed packages. So the mechanism feature you ask of a package > manager is a policy guarantee that the package manager will not install > unsigned packages? Quite backwards imho. My point was that a standard place for policies would be a good thing to have. I don't know about apt or yum, but the SUSE installer checks if the signature is present in the rpm database (by calling rpm --checksig) to decide if the package may be installed or not. Cheers, Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: Removing the rpm dependency on beecrypt
On Sun, Jul 15, 2007 at 01:44:00PM -0400, Jeff Johnson wrote: > pubkeys are left to external helpers. Lord knows that the P in PKI > leaves per-system "public" keys as implementation specified. There > are no reasonably widespread convention(s) for shared per-system > public keys other than /etc/pki in Fedora and similar per-vendor > (or per-distro) peculier conventional choices for paths to public key > storage. Ok, but keep in mind that it's not about just storing known pubkeys, but also telling the system that you allow the installation of packages signed with pubkey X. Cheers, Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: Should rpm call umask(2)?
On Tue, Jul 10, 2007 at 03:14:11PM -0400, Jeff Johnson wrote: > This ancient bug > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=83006 > keeps resurfacing. > > It's trivial to add to main() > mode_t mask = 002; Uh, not 002 please, 022 is the standard. Make it configurable if you really need to do something like that. And while you're at it, don't forget to reset the environment and the process limits (setrlimit). ;-) Cheers, Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: Manifest not working
On Wed, Jul 04, 2007 at 06:27:02AM -0400, Jeff Johnson wrote: > > On Jul 4, 2007, at 3:56 AM, Michael Schroeder wrote: > >A manifest file needs to be at least 96 bytes long (the size of > >the rpm lead). Maybe yours is too short? > > > > Not true: > > [EMAIL PROTECTED] rpm44]$ cat q > time-1.7-27.i386.rpm > [EMAIL PROTECTED] rpm44]$ sudo rpm -Uvh q > Preparing... > ### [100%] >1:time > ### [100%] Ah, so this bug is already fixed. Good to hear, sorry for the noise... Cheers, Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: Manifest not working
On Tue, Jul 03, 2007 at 04:04:22PM -0500, Mark Hatle wrote: > The manifest files don't seem to be working for me. But I can't debug > rpm due to it being stripped. A manifest file needs to be at least 96 bytes long (the size of the rpm lead). Maybe yours is too short? Cheers, Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: fsync()
On Tue, Jul 03, 2007 at 12:49:14PM -0400, Jeff Johnson wrote: > > On Jul 3, 2007, at 12:16 PM, Michael Schroeder wrote: > > > > >No, as one needs a patch to Berkeleydb to make it support > >individual syncing. > > > > What's the patch? Overloading the sync vector or something deeper in > db/*? Something deeper. (Hmm, didn't you have some good connections to the berkeley db folks? Maybe you can push the patch upstream...) --- db/db/db.c.orig 2004-11-11 15:58:46.0 + +++ db/db/db.c 2005-12-15 16:17:45.0 + @@ -591,6 +591,8 @@ __db_dbenv_mpool(dbp, fname, flags) (F_ISSET(dbp, DB_AM_NOT_DURABLE) ? DB_TXN_NOT_DURABLE : 0), 0, dbp->pgsize)) != 0) return (ret); + if (LF_ISSET(DB_NOFSYNC) && mpf->mfp) + F_SET(mpf->mfp, MP_NOFSYNC); return (0); } --- db/db/db_iface.c.orig 2004-10-16 01:31:54.0 + +++ db/db/db_iface.c2005-12-15 16:17:45.0 + @@ -1068,7 +1068,7 @@ __db_open_arg(dbp, txn, fname, dname, ty #defineOKFLAGS \ (DB_AUTO_COMMIT | DB_CREATE | DB_DIRTY_READ | DB_EXCL |\ DB_FCNTL_LOCKING | DB_NO_AUTO_COMMIT | DB_NOMMAP | DB_RDONLY |\ - DB_RDWRMASTER | DB_THREAD | DB_TRUNCATE | DB_WRITEOPEN) + DB_RDWRMASTER | DB_THREAD | DB_TRUNCATE | DB_WRITEOPEN | DB_NOFSYNC) if ((ret = __db_fchk(dbenv, "DB->open", flags, OKFLAGS)) != 0) return (ret); if (LF_ISSET(DB_EXCL) && !LF_ISSET(DB_CREATE)) --- db/dbinc/db.in.orig 2004-10-16 01:31:54.0 + +++ db/dbinc/db.in 2005-12-15 16:17:45.0 + @@ -260,6 +260,7 @@ struct __db_dbt { #defineDB_FCNTL_LOCKING 0x0002000 /* UNDOC: fcntl(2) locking. */ #defineDB_RDWRMASTER 0x0004000 /* UNDOC: allow subdb master open R/W */ #defineDB_WRITEOPEN 0x0008000 /* UNDOC: open with write lock. */ +#defineDB_NOFSYNC0x001 /* UNDOC: don't fsync */ /* * Flags private to DB_ENV->txn_begin. --- db/dbinc/mp.h.orig 2004-10-16 01:31:54.0 + +++ db/dbinc/mp.h 2005-12-15 16:25:56.0 + @@ -309,6 +309,7 @@ struct __mpoolfile { #defineMP_FAKE_UOC 0x080 /* Unlink_on_close field: fake flag. */ #defineMP_NOT_DURABLE 0x100 /* File is not durable. */ #defineMP_TEMP 0x200 /* Backing file is a temporary. */ +#define MP_NOFSYNC 0x400 /* Don't fsync */ u_int32_t flags; }; --- db/mp/mp_sync.c.orig2004-11-11 15:58:48.0 + +++ db/mp/mp_sync.c 2005-12-15 16:23:57.0 + @@ -553,7 +553,7 @@ done: /* if (ret == 0 && (op == DB_SYNC_CACHE || op == DB_SYNC_FILE)) { if (dbmfp == NULL) ret = __memp_sync_files(dbenv, dbmp); - else + else if (!dbmfp->mfp || !F_ISSET(dbmfp->mfp, MP_NOFSYNC)) ret = __os_fsync(dbenv, dbmfp->fhp); } @@ -600,7 +600,7 @@ int __memp_sync_files(dbenv, dbmp) MUTEX_THREAD_LOCK(dbenv, dbmp->mutexp); for (dbmfp = TAILQ_FIRST(&dbmp->dbmfq); dbmfp != NULL; dbmfp = TAILQ_NEXT(dbmfp, q)) { - if (dbmfp->mfp != mfp || F_ISSET(dbmfp, MP_READONLY)) + if (dbmfp->mfp != mfp || F_ISSET(dbmfp, MP_READONLY | MP_NOFSYNC)) continue; ret = __os_fsync(dbenv, dbmfp->fhp); break; @@ -662,6 +662,9 @@ __memp_mf_sync(dbmp, mfp) dbenv = dbmp->dbenv; + if (F_ISSET(mfp, MP_NOFSYNC)) + return 0; + /* * Expects caller to be holding the region lock: we're using the path * name and __memp_nameop might try and rename the file. Cheers, Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: fsync()
On Tue, Jul 03, 2007 at 11:38:15AM -0400, Jeff Johnson wrote: > We also disagree on the importance of attempting to sync > data to disk in spite of modest cost. I believe that rpm (and rpm5.org) > should sync to disk where appropriate rather than disabling rpmdb > sync's (and fsync for file contents here) as SuSE Just to clarify things: SUSE is just disabling the syncing of the index databases, the Packages database is still fsync()ed. Index databases can be easily regenerated by 'rpm --rebuilddb', but if Packages is corrupt you have hosed your system. > and perhaps rpm.org are choosing to do. No, as one needs a patch to Berkeleydb to make it support individual syncing. Cheers, Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: fsync()
On Tue, Jul 03, 2007 at 10:39:10AM -0400, Jeff Johnson wrote: > But this isn't every program, rpm is an installer, and installers are > expected to try harder. > > Slowing down is likely unmeasureable, Not from my experience... > and can certainly be conditioned > on file system type or configuration. I wouldn't mind a rpm macro check so that people can turn it on or off according to their needs. Thanks, Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: fsync()
On Tue, Jul 03, 2007 at 08:10:04PM +1000, Russell Coker wrote: > Do you consider systems that lose data, have internal databases that don't > match their own state, and generally cause down-time for people to be > correct? You might as well argue that *every* program that writes to disk should do fsync() calls. If you crash the system while rpm is running the system is broken nevertheless. Your fsyncs just make the critical window a bit smaller while slowing down rpm. Cheers, Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org
Re: fsync()
On Sun, Jul 01, 2007 at 08:02:40PM +1000, Russell Coker wrote: > It's really a pity that people can't just write correct programs and that it > needs to cost someone time and money and then more time fighting with the > developers to get the fix applied. You have a very strange definition of "correct"... Michael. -- Michael Schroeder [EMAIL PROTECTED] SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} __ RPM Package Managerhttp://rpm5.org Developer Communication Listrpm-devel@rpm5.org