Re: [bug #25538] excluded files are still stat()ed
Kevin Pulo invalid.nore...@gnu.org wrote:
URL:
http://savannah.gnu.org/bugs/?25538
Summary: excluded files are still stat()ed
...
Details:
My problem is that files I have excluded from du using -X or --exclude still
have stat() run on them. In particular, this is a problem when trying to
exclude some fuse filesystems on Linux, eg. sshfs and encfs, which deny all
access to other users (including root).
For example:
r...@bebique:~# du -axk /home/kev/mnt/sf
du: cannot access `/home/kev/mnt/sf/home': Permission denied
4 /home/kev/mnt/sf
r...@bebique:~# du -axk --exclude=/home/kev/mnt/sf/home /home/kev/mnt/sf
du: cannot access `/home/kev/mnt/sf/home': Permission denied
4 /home/kev/mnt/sf
r...@bebique:~# echo $?
1
r...@bebique:~#
The non-zero exit status is particularly troubling, since it means I cannot
chain other commands after du using '' whenever it's operating on a tree
that has these sorts of fuse fs's in it.
The only workaround at the moment is to exclude the parent directory, eg.
--exclude=~kev/mnt/sf in the example above. This unfortunately means that
everything else in that directory is also excluded.
r...@bebique:~# du -axk --exclude=/home/kev/mnt/sf /home/kev/mnt/sf
r...@bebique:~# echo $?
0
r...@bebique:~#
Having looked at the code, I'm not sure how this would best be fixed. The
list of excluded files is only used in process_file(), which is too late. I
presume that the stat() is happening inside fts_read(), which populates
end-fts_statp with the results of the stat() call. I suppose that extending
fts_read() to also respect the exclusion list would be fairly invasive.
Alternatively, the EPERM could persist during the fts_read(), but then be
somehow forgotten about later for excluded files, allowing the exit status
to return to being 0 (assuming no other genuine errors).
Thanks for the report.
You're right that accommodating this would be invasive.
However, I'm beginning to think that it's necessary.
fts needs stat information on directories for a few reasons:
- dev/inode, for cycle detection
- dev to support the FTS_XDEV option
- stat.st_mode or dirent.d_type to know when something is a directory
so obviously we can't skip it.
However, one approach would be to extend fts by giving it an
option whereby fts_read would no longer call fts_stat before returning.
With this new option, it would be the responsibility of the caller
to set information before the next fts_read call.
Here's the relevant, just-before-return code from fts_read:
check_for_dir:
sp-fts_cur = p;
if (p-fts_info == FTS_NSOK)
{
if (p-fts_statp-st_size == FTS_STAT_REQUIRED)
p-fts_info = fts_stat(sp, p, false);
else
fts_assert (p-fts_statp-st_size ==
FTS_NO_STAT_REQUIRED);
}
if (p-fts_info == FTS_D)
{
/* Now that P-fts_statp is guaranteed to be valid,
if this is a command-line directory, record its
device number, to be used for FTS_XDEV. */
if (p-fts_level == FTS_ROOTLEVEL)
sp-fts_dev = p-fts_statp-st_dev;
Dprintf (( entering: %s\n, p-fts_path));
if (! enter_dir (sp, p))
{
__set_errno (ENOMEM);
return NULL;
}
}
return p;
By the way, while looking at this, I noticed
fts_stat was being called earlier than I expected (it was
probably in fts_build), so I made this change to du.c:
diff --git a/src/du.c b/src/du.c
index 860e8fe..0749097 100644
--- a/src/du.c
+++ b/src/du.c
@@ -660,7 +660,7 @@ main (int argc, char **argv)
char *files_from = NULL;
/* Bit flags that control how fts works. */
- int bit_flags = FTS_TIGHT_CYCLE_CHECK;
+ int bit_flags = FTS_TIGHT_CYCLE_CHECK | FTS_DEFER_STAT;
/* Select one of the three FTS_ options that control if/when
to follow a symlink. */
I'll probably commit that and similar for the other fts clients
today.
___
Bug-coreutils mailing list
Bug-coreutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: [PATCH] install: add -C option to install file only when necessary
Kamil Dudka kdu...@redhat.com wrote:
it was discussed here 5 years ago (and considered good idea) to add -C option
to install:
http://lists.gnu.org/archive/html/bug-coreutils/2003-11/msg00017.html
With this option install checks an existing destination file and if it is not
different (by content, owner, group and mode) from source, the file is not
installed. Preserving destination's original mtime can significantly decrease
time of building when a system library is reinstalled but the header files
are not changed at all.
This option was already introduced in RHEL-4, but it was never accepted by
upstream. There were some issues which should be solved by this patch.
Hi Kamil,
Does any other install implementation provide -C?
The precedent of RHEL4 is probably enough to justify burning the
short -C option, but finding at least one more would be better.
No long option?
How about using --compare?
Thanks for all the tests.
Please adjust according to the comments below,
and it should be good to go.
From 9d4ae524b93e3bb2f8cb2c99e22f3f192e8dfae8 Mon Sep 17 00:00:00 2001
From: Kamil Dudka kdu...@redhat.com
Date: Fri, 16 Jan 2009 11:58:26 +0100
Subject: [PATCH] install: add -C option to install file only when necessary
* src/install.c (have_same_content): New function to compare files
content.
(need_copy): New function to check if copy is necessary.
(main): Handle new option -C.
(copy_file): Skip file copying if not necessary.
(usage): Show new option -C in --help.
* tests/install/install-C: Basic tests for install -C.
* tests/install/install-C-root: Tests requiring root privileges.
* tests/install/install-C-selinux: Tests requiring SELinux.
* tests/Makefile.am: Add new tests for install -C.
* doc/coreutils.texi: Document new install option -C.
* NEWS: Mention the change.
---
NEWS|3 +
doc/coreutils.texi |5 ++
src/install.c | 112
++-
tests/Makefile.am |3 +
tests/install/install-C | 75 ++
tests/install/install-C-root| 64 ++
tests/install/install-C-selinux | 56 +++
7 files changed, 317 insertions(+), 1 deletions(-)
create mode 100755 tests/install/install-C
create mode 100755 tests/install/install-C-root
create mode 100755 tests/install/install-C-selinux
diff --git a/NEWS b/NEWS
index f1b383e..05d6644 100644
--- a/NEWS
+++ b/NEWS
@@ -10,6 +10,9 @@ GNU coreutils NEWS-*-
outline -*-
dd accepts iflag=cio and oflag=cio to open the file in CIO (concurrent I/O)
mode where this feature is available.
+ install -C installs file, unless target already exists and is the same
file,
+ in which case the modification time is not changed
+
ls --color now highlights hard linked files, too
stat -f recognizes the Lustre file system type
diff --git a/doc/coreutils.texi b/doc/coreutils.texi
index d8df107..b1c22e0 100644
--- a/doc/coreutils.texi
+++ b/doc/coreutils.texi
@@ -2123,6 +2123,11 @@ The program accepts the following options. Also see
@ref{Common options}.
@table @samp
+...@item -C
+...@opindex -C
+Install file, unless target already exists and is the same file, in which
+case the modification time is not changed.
+
@item -c
@itemx --crown-margin
@opindex -c
diff --git a/src/install.c b/src/install.c
index 9dda05a..03c849a 100644
--- a/src/install.c
+++ b/src/install.c
@@ -31,6 +31,7 @@
#include cp-hash.h
#include copy.h
#include filenamecat.h
+#include full-read.h
#include mkancesdirs.h
#include mkdir-p.h
#include modechange.h
@@ -125,6 +126,9 @@ static mode_t dir_mode = DEFAULT_MODE;
or S_ISGID bits. */
static mode_t dir_mode_bits = CHMOD_MODE_BITS;
+/* Compare files before installing (-C) */
+static bool copy_only_if_needed;
+
/* If true, strip executable files after copying them. */
static bool strip_files;
@@ -167,6 +171,90 @@ static struct option const long_options[] =
{NULL, 0, NULL, 0}
};
+/* Compare content of opened files using file descriptors A_FD and B_FD.
Return
+ true if files are equal. */
+static bool
+have_same_content (int a_fd, int b_fd)
+{
+#define CMP_BLOCK_SIZE 65536
+ char a_buff[CMP_BLOCK_SIZE];
+ char b_buff[CMP_BLOCK_SIZE];
+
+ size_t size;
+ while (0 (size = full_read (a_fd, a_buff, CMP_BLOCK_SIZE))) {
+if (size != full_read (b_fd, b_buff, CMP_BLOCK_SIZE))
+ return false;
+
+if (memcmp (a_buff, b_buff, size) != 0)
+ return false;
+ }
+
+ return size == 0;
+#undef CMP_BLOCK_SIZE
+}
+
+/* Return true if copy of file FILE to destination TO is necessary. */
+static bool
+need_copy (const char *file, const char *to, const struct cp_options *x)
Please rename:
s/file/src_name/
s/to/dest_name/
+{
+ struct stat file_s, to_s;
please rename: src_sb, dest_sb
+ int
Re: Bugs in unexpand(1) version 6.10
On Wednesday 11 February 2009 02:04:11 Jim Meyering wrote: Mike Frysinger vap...@gentoo.org wrote: ... i was thinking a common change to the version-etc module to add a packager field rather than having every package out there allow people to tweak PACKAGE_NAME. what do you think of that ? Sounds sensible. The question then becomes whether to change version_etc (probably not), or to add a new interface that takes the additional parameter. Does anyone prefer to add a parameter to version_etc? i prefer modifying version_etc as this would go a long way in acknowledging that end users are not the main consumer of software. they get it by way of distro packagers. however, i dont think it needs to modify the function prototype ? if the m4 set up a PACKAGE_PACKAGER define, the version_etc module could use that. if the person running configure doesnt specify the --with-packager=... option, then it wont show up in the output. That sounds even better. Post the patch! one last question: where to put the string ? $ ls --version ls (GNU coreutils) 6.12 Packaged by ... some distro string here ... Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Written by Richard Stallman and David MacKenzie. and if no packager option was used, then that Packaged by line will be entirely omitted ... -mike ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: Bugs in unexpand(1) version 6.10
$ ls --version ls (GNU coreutils) 6.12 Packaged by ... some distro string here ... That looks like the right place to me. I'd be tempted to put a blank line between Packaged by ... and Copyright Please include a url to your downstream coreutils package in the distro string. This seems fine, but can we also include the information in --help, which is what we did when we amended the coding standards a few days ago? For instance, grep 2.5.4 --help ends with: Report bugs to: bug-g...@gnu.org GNU Grep home page: http://www.gnu.org/software/grep/ General help using GNU software: http://www.gnu.org/gethelp/ I suggest having another line such as Gentoo Grep home page and bug reports: http://whatever between Report bugs to and GNU Grep home page. Perhaps even the address in Report bugs to: should be changed also. Perhaps that same line should be used in --version and --help. I assume we're trying to get across the same information. karl ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: Bugs in unexpand(1) version 6.10
On Wednesday 11 February 2009 13:31:57 Karl Berry wrote: $ ls --version ls (GNU coreutils) 6.12 Packaged by ... some distro string here ... That looks like the right place to me. I'd be tempted to put a blank line between Packaged by ... and Copyright the attached patch does this Please include a url to your downstream coreutils package in the distro string. This seems fine, but can we also include the information in --help, which is what we did when we amended the coding standards a few days ago? For instance, grep 2.5.4 --help ends with: Report bugs to: bug-g...@gnu.org GNU Grep home page: http://www.gnu.org/software/grep/ General help using GNU software: http://www.gnu.org/gethelp/ I suggest having another line such as Gentoo Grep home page and bug reports: http://whatever between Report bugs to and GNU Grep home page. Perhaps even the address in Report bugs to: should be changed also. Perhaps that same line should be used in --version and --help. I assume we're trying to get across the same information. so should this be one or two new options ? --with-packager=Gentoo --with-packager-version=some patchset version info --with-packager-bugurl=http://bugs.gentoo.org/; $ ls --version ls (GNU coreutils) 6.12 Packaged by Gentoo (some patchset version info) .. $ ls --help ... Report bugs to bug-coreutils@gnu.org. GNU coreutils home page: http://www.gnu.org/software/coreutils/ Gentoo bug reporting page: http://bugs.gentoo.org/ with the last line looking something like: %s bug reporting page: %s\n, PACKAGE_PACKAGER, PACKAGE_PACKAGER_BUGURL, -mike diff --git a/lib/version-etc.c b/lib/version-etc.c index 2258c2e..e2daa0d 100644 --- a/lib/version-etc.c +++ b/lib/version-etc.c @@ -59,6 +59,10 @@ version_etc_va (FILE *stream, else fprintf (stream, %s %s\n, package, version); +#ifdef PACKAGE_PACKAGER + fprintf (stream, Packaged by %s\n, PACKAGE_PACKAGER); +#endif + /* TRANSLATORS: Translate (C) to the copyright symbol (C-in-a-circle), if this symbol is available in the user's locale. Otherwise, do not translate (C); leave it as-is. */ diff --git a/m4/version-etc.m4 b/m4/version-etc.m4 new file mode 100644 index 000..eecbc47 --- /dev/null +++ b/m4/version-etc.m4 @@ -0,0 +1,19 @@ +# version-etc.m4 serial 1 +# Copyright (C) 2009 Free Software Foundation, Inc. +# This file is free software; the Free Software Foundation +# gives unlimited permission to copy and/or distribute it, +# with or without modifications, as long as this notice is preserved. + +AC_DEFUN([gl_VERSION_ETC], +[dnl + AC_ARG_WITH([packager], +[AS_HELP_STRING([--with-packager], + [String identifying the packager of this software])], +[dnl + case $withval in + yes|no) ;; + *) AC_DEFINE_UNQUOTED([PACKAGE_PACKAGER], [$withval], + [Packager identification]);; + esac +]) +]) diff --git a/modules/version-etc b/modules/version-etc index aac2311..589c8c9 100644 --- a/modules/version-etc +++ b/modules/version-etc @@ -4,12 +4,14 @@ Print --version and bug-reporting information in a consistent format. Files: lib/version-etc.h lib/version-etc.c +m4/version-etc.m4 Depends-on: gettext-h stdarg configure.ac: +gl_VERSION_ETC Makefile.am: lib_SOURCES += version-etc.h version-etc.c ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: [PATCH] tests: Avoid skipping stty-* tests.
On 10 Feb 2009, Jim Meyering verbalised: FWIW, when I run make check on a F10 system, the 3 stty* tests are still skipped. However, when I run them individually, with your correction, they now _do_ PASS. Interesting. They're run here (not an F10 system :) ). FWIW I don't think pwd-unreadable-parent should be skipped on Linux either, but on every Linux/glibc system I've run it on it is skipped: REPLACE_GETCWD is set, but so is HAVE_PARTLY_WORKING_GETCWD, and pwd-unreadable-parent won't fail merely because glibc's getcwd fails for enormously deep directories. (I haven't looked into why this is yet.) (But this is a gnulib thing, not a coreutils thing. More later when I've tested it on more systems and proved it's not my own stupidity.) ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: [bug #25538] excluded files are still stat()ed
Kevin Pulo wrote: r...@bebique:~# du -axk /home/kev/mnt/sf du: cannot access `/home/kev/mnt/sf/home': Permission denied 4 /home/kev/mnt/sf r...@bebique:~# du -axk --exclude=/home/kev/mnt/sf/home /home/kev/mnt/sf du: cannot access `/home/kev/mnt/sf/home': Permission denied 4 /home/kev/mnt/sf r...@bebique:~# echo $? 1 r...@bebique:~# The fuse mount point is /home/kev/mnt/sf/home right? I believe this is a bug in the fuse kernel filesystem driver. While it can refuse access to the contents of the filesystem, a stat on the mount point itself should work and probably return a mode indicating you do not have read or execute access to the directory so tools know not to try traversing it. ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: bug with ls
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 [please keep the list in the loop; and on this list, it is customary to not top-post] According to jason on 2/11/2009 8:14 PM: Meanwhile, any patch to the documentation to make it more obvious that -R is a no-op in the presence of -d is both simpler to write and also more likely to be applied immediately. Understood, a good explanation. I don't know of other distros/un*x implementations that behave this way, mostly because I work with Linux almost exclusively these days. I agree updating or clarifying the man page condition would be best. Do you have any suggested wording which would have helped your situation? - -- Don't work too hard, make some time for fun as well! Eric Blake e...@byu.net -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Cygwin) Comment: Public key at home.comcast.net/~ericblake/eblake.gpg Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmTlNMACgkQ84KuGfSFAYALPACfWJAQhkpjGkxgbCnr9svt5wUO UDoAn0wCMoKPODWMrl2SlqAjJgikTO++ =5RlI -END PGP SIGNATURE- ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: Difference in ln/ln -s semantics
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
According to Vitali Lovich on 2/3/2009 12:30 PM:
How about for symlinks: When creating symbolic links, each TARGET path
will be resolved relative to the parent directory of the symbolic link
when it is accessed below the When creating hard links, each TARGET
must exist. text.
Is it too sutble that this may be a problem for relative paths? ln
man page already contains a link to the symlink man page - but it's
probably unlikely that someone using the tool is going to bother looking
at the system call behind it.
How about the following patch? I've pushed it to the next branch, to make
it easier to review.
- --
Don't work too hard, make some time for fun as well!
Eric Blake e...@byu.net
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkmTpbUACgkQ84KuGfSFAYC9RQCgy2yyUt5mgfu3sHVIuHUSfJmG
no4An0uZPww/cgLMZpuZkp0fcIc2sYSx
=CMlD
-END PGP SIGNATURE-
From 022889a4575390b656abf89b0e8f43ef03c1cc00 Mon Sep 17 00:00:00 2001
From: Eric Blake e...@byu.net
Date: Wed, 11 Feb 2009 21:09:09 -0700
Subject: [PATCH] ln: add details to --help text
* src/ln.c (usage): Mention more about symlink properties.
* doc/coreutils.texi (ln invocation): Likewise.
* THANKS: Update.
Reported by Vitali Lovich.
Signed-off-by: Eric Blake e...@byu.net
---
THANKS |1 +
doc/coreutils.texi | 25 -
src/ln.c |6 --
3 files changed, 29 insertions(+), 3 deletions(-)
diff --git a/THANKS b/THANKS
index da1fc15..a3f9801 100644
--- a/THANKS
+++ b/THANKS
diff --git a/doc/coreutils.texi b/doc/coreutils.texi
index c3a1164..57497e9 100644
--- a/doc/coreutils.texi
+++ b/doc/coreutils.texi
@@ -8689,9 +8689,32 @@ ln invocation
reading, writing, and so on) are passed the symbolic link file, the
kernel automatically @dfn{dereferences} the link and operates on the
target of the link. But some operations (e.g., removing) work on the
-link file itself, rather than on its target. @xref{Symbolic Links,,,
+link file itself, rather than on its target. The owner, group, and
+mode of a symlink are not significant to file access performed through
+the link. @xref{Symbolic Links,,,
libc, The GNU C Library Reference Manual}.
+Symbolic links can contain arbitrary strings; a @dfn{dangling symlink}
+occurs when the string in the symlink does not resolve to a file.
+There are no restrictions against creating dangling symbolic links.
+There are trade-offs to using absolute or relative symlinks. An
+absolute symlink always points to the same file, even if the directory
+containing the link is moved. However, if the symlink is visible from
+more than one machine (such as on a networked file system), the file
+pointed to might not always be the same. A relative symbolic link is
+resolved in relation to the directory that contains the link, and is
+often useful in referring to files on the same device without regards
+to what name that device is mounted on when accessed via networked
+machines.
+
+When creating a relative symlink in a different location than the
+current directory, the resolution of the symlink will be different
+than the resolution of the same string from the current directory.
+Therefore, many users prefer to first change directories to the
+location where the relative symlink will be created, so that
+tab-completion or other file resolution will find the same target as
+what will be placed in the symlink.
+
The program accepts the following options. Also see @ref{Common options}.
@table @samp
diff --git a/src/ln.c b/src/ln.c
index 2dc5628..d0f9266 100644
--- a/src/ln.c
+++ b/src/ln.c
@@ -1,5 +1,5 @@
/* `ln' program to create links between files.
- Copyright (C) 1986, 1989-1991, 1995-2008 Free Software Foundation, Inc.
+ Copyright (C) 1986, 1989-1991, 1995-2009 Free Software Foundation, Inc.
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
@@ -359,7 +359,9 @@ In the 1st form, create a link to TARGET with the name
LINK_NAME.\n\
In the 2nd form, create a link to TARGET in the current directory.\n\
In the 3rd and 4th forms, create links to each TARGET in DIRECTORY.\n\
Create hard links by default, symbolic links with --symbolic.\n\
-When creating hard links, each TARGET must exist.\n\
+When creating hard links, each TARGET must exist. Symbolic links\n\
+can hold arbitrary text; if later resolved, a relative link is\n\
+interpreted in relation to its parent directory.\n\
\n\
), stdout);
fputs (_(\
--
1.6.1.2
___
Bug-coreutils mailing list
Bug-coreutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: [PATCH] install: add -C option to install file only when necessary
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 According to Jim Meyering on 2/11/2009 9:41 AM: Does any other install implementation provide -C? Some BSD versions provide it. http://www.ss64.com/bash/install.html And automake's install-sh, the replacement for systems without install(1), added it according to this thread: http://lists.gnu.org/archive/html/automake-patches/2006-10/msg00070.html - -- Don't work too hard, make some time for fun as well! Eric Blake e...@byu.net -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Cygwin) Comment: Public key at home.comcast.net/~ericblake/eblake.gpg Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmTsqcACgkQ84KuGfSFAYARWgCePLr/jTqqWqgtLGwEDhHrmGJs m2IAn39v1TxSZ1WxNWOEqnt7msO7e83g =rN2y -END PGP SIGNATURE- ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: Difference in ln/ln -s semantics
Eric Blake e...@byu.net wrote: According to Vitali Lovich on 2/3/2009 12:30 PM: How about for symlinks: When creating symbolic links, each TARGET path will be resolved relative to the parent directory of the symbolic link when it is accessed below the When creating hard links, each TARGET must exist. text. Is it too sutble that this may be a problem for relative paths? ln man page already contains a link to the symlink man page - but it's probably unlikely that someone using the tool is going to bother looking at the system call behind it. How about the following patch? I've pushed it to the next branch, to make it easier to review. Thanks! Don't work too hard, make some time for fun as well! Subject: [PATCH] ln: add details to --help text * src/ln.c (usage): Mention more about symlink properties. * doc/coreutils.texi (ln invocation): Likewise. * THANKS: Update. Reported by Vitali Lovich. Nice additions. You're welcome to push them to master. ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: Parallelizing sort
On Wed, Feb 11, 2009 at 10:12:42PM -0800, Nima Nikzad wrote: Hello coreutils community, I am new to the exciting world of open source development and thought I should introduce my self and get some feedback on what I am working on. My name is Nima Nikzad and I, along with a small group of students, am working on changes to coreutils as part of a project for Prof. Paul Eggert's software engineering course at UCLA. Specifically, we are looking at how we can better utilize modern multi-core processors when sorting and merging files. My group is first taking a look at how to parallelize the external sort phase of sort's operation. Another group of students is taking a look at in-memory sort while we are working on this. Has anyone already tried their hand at parallelizing this code? Are there any special considerations we should keep in mind while moving forward? My group was looking to tackle the problem by using OpenMP as we have some experience working with it in the past and like that we can do quite a bit with it while (hopefully) having little impact on the structure of the existing code. Does anyone have any feedback on threading technologies that would be appropriate for this project or does anyone think OpenMP is a poor choice? I look forward to hearing your suggestions! We are looking to have something implemented and benchmarked soon and I will be sure to keep in contact with the community. Please feel free to reach me at nnikzad at ucla.edu. Thank you! - Nima Nikzad Hi everyone, My name is Glen Lenker. I am the project leader for the second group working to parallelize sort under Paul Eggert. As mentioned above, my group is primarily focused on parallelizing the in-memory sort, but at the moment we are still considering working to parallelize other aspects of the current sort as well. At the moment we are considering using the Gnulib thread module as our threading library. Jim: We heard from Paul Eggert today that you have looked into how sort would benefit from parallelization. I am particularly interested in your approach. If you don't mind, perhaps you could start the discussion on this. -- Glen Lenker ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
