Re: Human readable sort
Pádraig Brady wrote: Latest version attached. ... * NEWS: Document the new option * doc/coreutils.texi (sort invocation): ditto * src/sort.c (main): handle the new --human-numeric-sort option (-h). (human_numcompare): A new function to compare SI and IEC suffixes before falling back to the standard --numeric comparison. (find_unit_order): A new helper function to find the order of magnitude of a number string as determined by its suffix. (check_mixed_SI_IEC): A new helper function to exit with error if both SI and IEC suffixes are presented. Looks fine. Thank you, Pádraig and Michael. ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: [PATCH] chroot specify user/group feature
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 According to Jim Meyering on 5/26/2009 7:21 AM: Merged and pushed. Thanks again. Would it be worth starting to patch the testsuite to replace 'setuidgid -g list usr cmd arg' with 'chroot --user usr --groups=list / cmd arg' in order to give this feature more exposure and reduce our dependence on uninstalled apps? - -- Don't work too hard, make some time for fun as well! Eric Blake e...@byu.net -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Cygwin) Comment: Public key at home.comcast.net/~ericblake/eblake.gpg Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkodI0IACgkQ84KuGfSFAYAwzACgu9wm2OjNjseMJuSz2LE8Um0P 0isAnj+mM2OacXUeDuFVf+mGlI+OC86l =NaUv -END PGP SIGNATURE- ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: coreutils 7.4 build: previous declaration of '__u32' was here
From: Mike Frysinger vap...@gentoo.org Sent: 2009-05-26 21:17 On Tuesday 26 May 2009 17:05:29 Kamil Dudka wrote: On Tuesday 26 of May 2009 22:55:37 Poor Yorick wrote: Building coreutils to install in an alternate location which has its own libcap and glibc (self-compiled): CC ls.o In file included from ls.c:43: /path/to/include/sys/capability.h:34: error: redefinition of typedef '__u32' /path/to/glibc/include/asm-generic/int-ll64.h:24: note: previous declaration of '__u32' was here Any suggestions on the root cause or workaround for this? It might be caused by this bug: https://bugzilla.redhat.com/show_bug.cgi?id=483548 ... just look at sys/capability.h itself. the bug is right -- it's a mess. remove all the crap from that header and everything magically works. http://sources.gentoo.org/sys-libs/libcap/files/libcap-2.16-drop-linux- workarounds.patch -mike After reading the bug report, I was able to build coreutils by invoking make like so (running configure as normal): make LIB_SELINUX= SELINUX_CONTEXT_H=selinux/context.h SELINUX_SELINUX_H=selinux/selinux.h This tip just got me the rest of my week back. Many thanks! -- Yorick ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: [PATCH] New fallocate module
Take 2 attached.
Still outstanding from the gnulib portion is support for solaris.
I may get time to set this up, but I've very little time lately,
so I would like not to require this for merging.
Also I didn't yet implement the optimization to coreutils
that Paul mentioned, to cache whether the filesystem will
just return ENOSYS all the time.
cheers,
Pádraig.
From bd55931dd6286ff952fdcadd25d0bef263be4de2 Mon Sep 17 00:00:00 2001
From: =?utf-8?q?P=C3=A1draig=20Brady?= p...@draigbrady.com
Date: Thu, 21 May 2009 08:03:00 +0100
Subject: [PATCH] fallocate: New module to ensure this interface is available
fallocate() allows one to associate a unit of space
with a (portion of a) file before writing. This info can
then be used to immediately determine if enough space is available,
and also allow efficient allocation on the storage device.
* m4/fallocate.m4: check we can link to fallocate()
* lib/fcntl.in.h: replacement stub if missing
* modules/fallocate: dependencies for new module
* MODULES.html.sh (File system functions): add it
---
MODULES.html.sh |1 +
lib/fcntl.in.h| 17 +
m4/fallocate.m4 | 26 ++
modules/fallocate | 23 +++
4 files changed, 67 insertions(+), 0 deletions(-)
create mode 100644 m4/fallocate.m4
create mode 100644 modules/fallocate
diff --git a/MODULES.html.sh b/MODULES.html.sh
index 06afa2d..6a1e058 100755
--- a/MODULES.html.sh
+++ b/MODULES.html.sh
@@ -2438,6 +2438,7 @@ func_all_modules ()
func_module dirfd
func_module double-slash-root
func_module euidaccess
+ func_module fallocate
func_module file-type
func_module fileblocks
func_module filemode
diff --git a/lib/fcntl.in.h b/lib/fcntl.in.h
index fd7520e..709db88 100644
--- a/lib/fcntl.in.h
+++ b/lib/fcntl.in.h
@@ -63,6 +63,23 @@ extern int open (const char *filename, int flags, ...);
extern void _gl_register_fd (int fd, const char *filename);
#endif
+#ifdef REPLACE_FALLOCATE
+# undef fallocate
+# define fallocate rpl_fallocate
+#include sys/types.h
+#include errno.h
+#undef FALLOC_FL_KEEP_SIZE /* Ensure this name is available. */
+#define FALLOC_FL_KEEP_SIZE 0x01
+static inline int fallocate (int fd, int mode, off_t offset, off_t len)
+{
+ /* This is a valid replacement for missing glibc fallocate(),
+ because code calling fallocate() must also handle this error
+ in the case that the kernel or filesystem don't support this. */
+ return ENOSYS;
+ /* FIXME: support fcntl(fd, F_ALLOCSP, ...) on solaris. */
+}
+#endif
+
#ifdef __cplusplus
}
#endif
diff --git a/m4/fallocate.m4 b/m4/fallocate.m4
new file mode 100644
index 000..b218253
--- /dev/null
+++ b/m4/fallocate.m4
@@ -0,0 +1,26 @@
+# fallocate.m4 serial 1
+dnl Copyright (C) 2009 Free Software Foundation, Inc.
+dnl This file is free software; the Free Software Foundation
+dnl gives unlimited permission to copy and/or distribute it,
+dnl with or without modifications, as long as this notice is preserved.
+
+AC_DEFUN([gl_FUNC_FALLOCATE],
+[
+ AC_CHECK_HEADERS_ONCE([linux/falloc.h])
+
+ dnl Persuade glibc fcntl.h to declare fallocate().
+ AC_REQUIRE([AC_USE_SYSTEM_EXTENSIONS])
+
+ AC_CACHE_CHECK([for fallocate], [gl_cv_func_fallocate], [
+AC_TRY_LINK([#include fcntl.h/* fallocate() declaration */
+ #include linux/falloc.h /* FALLOC_FL_KEEP_SIZE define */],
+ [fallocate(-1, FALLOC_FL_KEEP_SIZE, 0, 0);],
+ gl_cv_func_fallocate=yes, gl_cv_func_fallocate=no)])
+
+if test $gl_cv_func_fallocate = yes; then
+ AC_DEFINE([HAVE_FALLOCATE], [1], [Defined if fallocate() exists])
+else
+ AC_REQUIRE([AC_C_INLINE])
+ AC_DEFINE([REPLACE_FALLOCATE], [1], [Include replacement])
+fi
+])
diff --git a/modules/fallocate b/modules/fallocate
new file mode 100644
index 000..bbcb5db
--- /dev/null
+++ b/modules/fallocate
@@ -0,0 +1,23 @@
+Description:
+Ensure fallocate() is available
+
+Files:
+m4/fallocate.m4
+
+Depends-on:
+errno
+extensions
+fcntl
+
+configure.ac:
+gl_FUNC_FALLOCATE
+
+Makefile.am:
+
+Include:
+
+License:
+LGPL
+
+Maintainer:
+Pádraig Brady
--
1.5.3.6
From 7202ea62e6f6358931c50327df4a7b70b84332b7 Mon Sep 17 00:00:00 2001
From: =?utf-8?q?P=C3=A1draig=20Brady?= p...@draigbrady.com
Date: Mon, 18 May 2009 08:38:18 +0100
Subject: [PATCH] cp,mv,install: Try to fallocate() the destination file
This will allocate the file efficiently and also
give immediate indication of insufficient space for the copy.
* bootstrap.conf: include fallocate module from gnulib.
* src/copy.c (copy_reg): call fallocate() for non sparse files.
* NEWS: mention the change.
---
NEWS |6 ++
bootstrap.conf |1 +
src/copy.c | 56 +++-
3 files changed, 62 insertions(+), 1 deletions(-)
diff --git a/NEWS b/NEWS
index 31f1b1a..80bc22b 100644
--- a/NEWS
+++ b/NEWS
@@ -7,6 +7,12 @@ GNU coreutils NEWS-*- outline -*-
Re: [PATCH] chroot specify user/group feature
Hi Eric, Eric Blake e...@byu.net writes: Would it be worth starting to patch the testsuite to replace 'setuidgid -g list usr cmd arg' with 'chroot --user usr --groups=list / cmd arg' in order to give this feature more exposure and reduce our dependence on uninstalled apps? IMHO, since chroot now allows to specify users and groups by their names too, maybe it worths to move these functionalities in a gnulib module and share them with setuidgid. What all of you think? Giuseppe ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
two new chroot bugs
Eric Blake wrote:
According to Jim Meyering on 5/26/2009 7:21 AM:
Merged and pushed.
Thanks again.
Would it be worth starting to patch the testsuite to replace 'setuidgid -g
list usr cmd arg' with 'chroot --user usr --groups=list / cmd arg' in
order to give this feature more exposure and reduce our dependence on
uninstalled apps?
Yes. It would be good to get more coverage for installed tools.
However, I've just noticed that the new code in chroot.c needs work.
Two problems:
- set*ID failure did not evoke non-zero exit
- chroot --u=N / id -a currently uses gid uninitialized
i.e., when --userspec=USER (without :GROUP)
Same for --u=:1 (i.e, group ID with no user-ID)
$ sudo ./chroot --u=0 / id -g
3496015512
Here are patches, for review.
I'll add tests tomorrow.
From 63a1039e21fbf127e052f1b4d80176e3a2386d2a Mon Sep 17 00:00:00 2001
From: Jim Meyering meyer...@redhat.com
Date: Wed, 27 May 2009 22:06:04 +0200
Subject: [PATCH] chroot: set-*-ID failure must provoke nonzero exit before
execvp
* src/chroot.c (main): Exit upon set-group-ID or set-user-ID failure.
---
src/chroot.c | 21 ++---
1 files changed, 18 insertions(+), 3 deletions(-)
diff --git a/src/chroot.c b/src/chroot.c
index 788a1fc..ab35944 100644
--- a/src/chroot.c
+++ b/src/chroot.c
@@ -207,6 +207,7 @@ main (int argc, char **argv)
char *user;
char *group;
char const *err = parse_user_spec (userspec, uid, gid, user, group);
+ unsigned int fail = 0;
if (err)
error (EXIT_FAILURE, errno, %s, err);
@@ -214,14 +215,28 @@ main (int argc, char **argv)
free (user);
free (group);
+ /* Attempt to set all three: supplementary groups, group ID, user ID.
+ Diagnose any failures. If any have failed, exit before execvp. */
if (groups set_additional_groups (groups))
-error (0, errno, _(failed to set additional groups));
+{
+ error (0, errno, _(failed to set additional groups));
+ ++fail;
+}
if (gid setgid (gid))
-error (0, errno, _(failed to set group-ID));
+{
+ error (0, errno, _(failed to set group-ID));
+ ++fail;
+}
if (uid setuid (uid))
-error (0, errno, _(failed to set user-ID));
+{
+ error (0, errno, _(failed to set user-ID));
+ ++fail;
+}
+
+ if (fail)
+exit (EXIT_FAILURE);
}
/* Execute the given command. */
--
1.6.3.1.268.g94d6d1
From 6752c02be6bc760e92687e5ae71e93513f5b84da Mon Sep 17 00:00:00 2001
From: Jim Meyering meyer...@redhat.com
Date: Wed, 27 May 2009 23:06:15 +0200
Subject: [PATCH] chroot: don't set bogus user-ID or group-ID for --u=U or --u=:G
* src/chroot.c (main): Initialize both uid and gid. To -1.
This also allows one to set the primary group-ID to 0, in case
it's not that already.
---
src/chroot.c |8
1 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/src/chroot.c b/src/chroot.c
index ab35944..8bc4e59 100644
--- a/src/chroot.c
+++ b/src/chroot.c
@@ -202,8 +202,8 @@ main (int argc, char **argv)
if (userspec)
{
- uid_t uid;
- gid_t gid;
+ uid_t uid = -1;
+ gid_t gid = -1;
char *user;
char *group;
char const *err = parse_user_spec (userspec, uid, gid, user, group);
@@ -223,13 +223,13 @@ main (int argc, char **argv)
++fail;
}
- if (gid setgid (gid))
+ if (gid != (gid_t) -1 setgid (gid))
{
error (0, errno, _(failed to set group-ID));
++fail;
}
- if (uid setuid (uid))
+ if (uid != (uid_t) -1 setuid (uid))
{
error (0, errno, _(failed to set user-ID));
++fail;
--
1.6.3.1.268.g94d6d1
___
Bug-coreutils mailing list
Bug-coreutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: two new chroot bugs
Jim Meyering wrote: ... Here are patches, for review. I'll add tests tomorrow. That latter log wasn't complete. Subject: [PATCH] chroot: don't set bogus user-ID or group-ID for --u=U or --u=:G * src/chroot.c (main): Initialize both uid and gid. To -1. This also allows one to set the primary group-ID to 0, in case it's not that already. Here's what I have, now: chroot: don't set bogus user-ID or group-ID for --u=U: or --u=:G * src/chroot.c (main): Initialize both uid and gid. To -1. This also allows one to set the user-ID or primary group-ID to 0, in case it's not that already. ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: [PATCH] chroot specify user/group feature
Giuseppe Scrivano wrote: Eric Blake e...@byu.net writes: Would it be worth starting to patch the testsuite to replace 'setuidgid -g list usr cmd arg' with 'chroot --user usr --groups=list / cmd arg' in order to give this feature more exposure and reduce our dependence on uninstalled apps? IMHO, since chroot now allows to specify users and groups by their names too, maybe it worths to move these functionalities in a gnulib module and share them with setuidgid. What all of you think? setuidgid appears to be subsumed by chroot with the new options. If we can remove setuidgid.c, that code is no longer duplicated, so there's less (no?) motivation to move it into gnulib. ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: [PATCH] sort: Add --threads option, which parallelizes internal sort.
On Thu, Mar 26, 2009 at 09:50:08PM +, Ralf Wildenhues wrote: Hi Paul, all, Paul Eggert writes: This patch is by Glen Lenker, Matt Pham, Benjamin Nuernberger, Sky Lin, TaeSung Roh, and Paul Eggert. It adds support for parallelism within an internal sort. On our simple tests on a 2-core desktop x86, overall performance improved by roughly a factor of 1.6. This is too interesting to pass up. Example run, on an 8-way, and with cat'ed instances of the dictionary, on tmpfs, timings best of three: Hey Ralf, did you happen to specify the amount of RAM sort should use. Not specifying enough RAM for sort would force break what would be a single internal sort into multiple internal sort passes and then an external sort. As it is external sort is still sequential. runtime [s] threads file size [MB] 1 2 4 8 1 0.060.040.030.04 2 0.130.090.070.07 4 0.280.200.160.15 8 0.610.430.340.32 16 1.340.940.740.72 32 3.002.061.631.57 64 6.364.383.443.32 12813.499.307.137.24 25628.62 19.49 15.17 15.18 I ran these tests on a 256MB instance of the dictionary in tmpfs, on a 8-core machine specifying 2G of RAM. runtime [s] threads file size [MB] 1 2 4 8 256 2m41.219 1m27.357 52.53 36.429 Here's the abbreviated 'time' output for the last row: 26.95user 1.67system 0:28.62elapsed 100%CPU 30.78user 1.98system 0:19.49elapsed 168%CPU 37.41user 2.04system 0:15.17elapsed 260%CPU 60.68user 2.79system 0:15.18elapsed 417%CPU I forgot to use your time format in the test above, this is from a seperate test run. 160.22user 1.34system 2:41.61elapsed 99%CPU 159.83user 1.45system 1:27.12elapsed 185%CPU 159.84user 1.56system 0:52.26elapsed 308%CPU 160.67user 1.53system 0:36.26elapsed 447%CPU This seems to be what I would expect from a good implementation. It suggests to me that too much time is spent busy-waiting in pthread_join, or that sort is computing too much (I haven't looked at the patch in detail). Also, I'd have expected the rate going from 1 to 2 threads to get at least a bit better with bigger file size, but it remains remarkably constant, around 1.45 for this setup. What am I missing? ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: [PATCH] New fallocate module
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 According to Pádraig Brady on 5/27/2009 10:20 AM: Take 2 attached. Still outstanding from the gnulib portion is support for solaris. I may get time to set this up, but I've very little time lately, so I would like not to require this for merging. That seems like a reasonable approach for a first cut, especially with your FIXME comment. + dnl Persuade glibc fcntl.h to declare fallocate(). + AC_REQUIRE([AC_USE_SYSTEM_EXTENSIONS]) Use gl_USE_SYSTEM_EXTENSIONS instead; it avoids some warnings from newer autoconf. - -- Don't work too hard, make some time for fun as well! Eric Blake e...@byu.net -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Cygwin) Comment: Public key at home.comcast.net/~ericblake/eblake.gpg Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkoeD6AACgkQ84KuGfSFAYC3BACaAuF+a/BiNeXUSU/wI3Xbjuqc ejYAni3Rw3U5EnS+M91jgj22w5O+JO7H =aQJI -END PGP SIGNATURE- ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
