bug#17600: Maybe a bug of `date`

[HoHo Zhao]

Thank you Bob. I got it clear.

On 2014/5/27 10:55, Bob Proulx wrote:
> HoHo Zhao wrote:
>> Wrong:
>>  $ TZ=UTC date -d "15:00 CST"   (China Standard Time)
>>  Mon May 26 21:00:00 UTC 2014
>>
>> So the problem is with "CST" in the date STRING.
> 
> CST in the above is being interpreted as US Central Standard Time.
> For Central Standard Time it is correct.
> 
> CST is one of the ambiguous timezones that Pádraig referred to.  This
> is one of the reasons "date -R" with the unambiguous numeric timezones
> is the better output format.
> 
> Bob
> 
> 

-- 
HoHo Zhao
Regional IT Support
Red Hat Greater China

bug#17600: Maybe a bug of `date`

[Bob Proulx]

HoHo Zhao wrote:
> Wrong:
>   $ TZ=UTC date -d "15:00 CST"   (China Standard Time)
>   Mon May 26 21:00:00 UTC 2014
> 
> So the problem is with "CST" in the date STRING.

CST in the above is being interpreted as US Central Standard Time.
For Central Standard Time it is correct.

CST is one of the ambiguous timezones that Pádraig referred to.  This
is one of the reasons "date -R" with the unambiguous numeric timezones
is the better output format.

Bob

bug#14752: [PATCH] sort: print warning when fork() failed for --compress-program

[Pádraig Brady]

On 05/26/2014 10:00 PM, Azat Khuzhin wrote:
>> So the issue here is that sort is allocating
>> a large buffer up front thus impacting the fork().
>> Really sort(1) should be trying to avoid this issue
>> in the first place, and the issue is already logged at:
>> http://bugs.gnu.org/14752
> 
> Yes this is the same as I linked above.
> Does any body have a patch for this, or should I start working on this?

I was waiting for a patch that didn't materialize.
I'll have a look myself now.

thanks,
Pádraig.

bug#14752: [PATCH] sort: print warning when fork() failed for --compress-program

[Azat Khuzhin]

On Mon, May 26, 2014 at 09:43:23PM +0100, Pádraig Brady wrote:
> On 05/26/2014 09:13 PM, Azat Khuzhin wrote:
> > sort already have one when it trying to create decompressor, it is
> > obvious why it is really required in this case, since sort will read
> > compressed data as plain otherwise.
> > But sometimes it is really usefull to know whether sort failed to create
> > compressor or not, since some users may rely on available free space and
> > compressor.
> > 
> > * src/sort.c (create_temp_file): Add a warning when creating of
> > compressor failed.
> > ---
> > There is some old discussion about this
> > http://osdir.com/ml/bug-coreutils-gnu/2013-07/msg00010.html, but before this
> > will be fixed(?) we could print a warning on fail at least.
> > Thanks.
> > 
> >  src/sort.c | 7 +++
> >  1 file changed, 7 insertions(+)
> > 
> > diff --git a/src/sort.c b/src/sort.c
> > index 49caae5..eb1b1f3 100644
> > --- a/src/sort.c
> > +++ b/src/sort.c
> > @@ -1133,6 +1133,13 @@ maybe_create_temp (FILE **pfp, bool 
> > survive_fd_exhaustion)
> >  
> >async_safe_die (errno, "couldn't execute compress program");
> >  }
> > +  else
> > +{
> > +  error (0, errno,
> > + _("warning: couldn't create process for %s "
> > +   "(try to install overcommit always)"),
> > + compress_program);
> > +}
> >  }
> >  
> >*pfp = fdopen (tempfd, "w");
> 
> Thanks for the patch.
> 
> Note POSIX says that programs shouldn't output to stderr
> unless they're exiting with a failure code,

Hm, didn't know that, Thanks!
(Sometimes POSIX is not so good for human, I suppose.)

> I guess to avoid gradual accretion of warnings etc.
> which could impair general usage.
> 
> So the issue here is that sort is allocating
> a large buffer up front thus impacting the fork().
> Really sort(1) should be trying to avoid this issue
> in the first place, and the issue is already logged at:
> http://bugs.gnu.org/14752

Yes this is the same as I linked above.
Does any body have a patch for this, or should I start working on this?

Thanks.

> 
> thanks,
> Pádraig.

-- 
Respectfully
Azat Khuzhin

bug#17470: [PATCH] sort: rotate on ENOSPC while creating tmp files

[Azat Khuzhin]

On Mon, May 26, 2014 at 01:44:14PM -0700, Paul Eggert wrote:
> Azat Khuzhin wrote:
> >The current patch only look while files is created, but this is not
> >enough `I agree with you, it must check write(2) and fallback to creating
> >when write(2) will fail with ENOSPC.
> >This is what you mean?
> 
> Yes.
> 
> >when we have ENOSPC it
> >means that we already won't create any more files there, and one file
> >with relatively small size is not a big deal.
> 
> OK.  The point is that 'sort' shouldn't lose the data (including the
> possibly-incomplete trailing line) that's already in the temporary file when
> a write to that file fails.

Thanks for the explanation, I see what you mean.

> 
> Also, the code could treat EIO like ENOSPC, I suppose, to be more robust in
> the presence of bad temporary devices.
> 
> But beware file systems that report ENOSPC and EIO in a delayed fashion,
> i.e., not immediately upon the failing write, but somewhat later, typically
> when closing the output file.

Yeah, that's a good catch, I will keep this in mind when I will start
working on this.

Thanks,
Azat.

bug#14752: [PATCH] sort: print warning when fork() failed for --compress-program

[Pádraig Brady]

On 05/26/2014 09:13 PM, Azat Khuzhin wrote:
> sort already have one when it trying to create decompressor, it is
> obvious why it is really required in this case, since sort will read
> compressed data as plain otherwise.
> But sometimes it is really usefull to know whether sort failed to create
> compressor or not, since some users may rely on available free space and
> compressor.
> 
> * src/sort.c (create_temp_file): Add a warning when creating of
> compressor failed.
> ---
> There is some old discussion about this
> http://osdir.com/ml/bug-coreutils-gnu/2013-07/msg00010.html, but before this
> will be fixed(?) we could print a warning on fail at least.
> Thanks.
> 
>  src/sort.c | 7 +++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/src/sort.c b/src/sort.c
> index 49caae5..eb1b1f3 100644
> --- a/src/sort.c
> +++ b/src/sort.c
> @@ -1133,6 +1133,13 @@ maybe_create_temp (FILE **pfp, bool 
> survive_fd_exhaustion)
>  
>async_safe_die (errno, "couldn't execute compress program");
>  }
> +  else
> +{
> +  error (0, errno,
> + _("warning: couldn't create process for %s "
> +   "(try to install overcommit always)"),
> + compress_program);
> +}
>  }
>  
>*pfp = fdopen (tempfd, "w");

Thanks for the patch.

Note POSIX says that programs shouldn't output to stderr
unless they're exiting with a failure code,
I guess to avoid gradual accretion of warnings etc.
which could impair general usage.

So the issue here is that sort is allocating
a large buffer up front thus impacting the fork().
Really sort(1) should be trying to avoid this issue
in the first place, and the issue is already logged at:
http://bugs.gnu.org/14752

thanks,
Pádraig.

bug#17470: [PATCH] sort: rotate on ENOSPC while creating tmp files

[Paul Eggert]

Azat Khuzhin wrote:

The current patch only look while files is created, but this is not
enough `I agree with you, it must check write(2) and fallback to creating
when write(2) will fail with ENOSPC.
This is what you mean?


Yes.


when we have ENOSPC it
means that we already won't create any more files there, and one file
with relatively small size is not a big deal.


OK.  The point is that 'sort' shouldn't lose the data (including the 
possibly-incomplete trailing line) that's already in the temporary file 
when a write to that file fails.


Also, the code could treat EIO like ENOSPC, I suppose, to be more robust 
in the presence of bad temporary devices.


But beware file systems that report ENOSPC and EIO in a delayed fashion, 
i.e., not immediately upon the failing write, but somewhat later, 
typically when closing the output file.

bug#17470: [PATCH] sort: rotate on ENOSPC while creating tmp files

[Azat Khuzhin]

On Wed, May 14, 2014 at 07:48:39AM -0700, Paul Eggert wrote:
> Pádraig Brady wrote:
> >Also do we always know how much to fallocate?
> 
> Not if we're using compression on the temporaries, no.
> 
> I think a patch along these lines could be worthwhile, if it was simple and
> if it actually worked (the current one doesn't).  

The current patch only look while files is created, but this is not
enough `I agree with you, it must check write(2) and fallback to creating
when write(2) will fail with ENOSPC.
This is what you mean?

> Something along the
> following lines, say.  When multiple -T options are specified (-T FOO, -T
> FOP, -T FOQ, ...) and one of them runs out of disk space when creating a
> temporary file FOO/BAR, 'sort' stops creating files in FOO (effectively
> removing FOO from the option list) creates a file FOP/BAR instead, and
> redoes the process (whatever it was) that sent output to FOO/BAR, sending
> the output to FOP/BAR this time.

I don't think that redoes is worth it, since when we have ENOSPC it
means that we already won't create any more files there, and one file
with relatively small size is not a big deal.
I also think that dropping direcotory from list is a good idea, user can
notice this (using some monitorings) and clean it.

For example recently I need to sort relatively huge amount of data, and
I don't have enough space for writing all tmp files (why
--compress-program not works for me is another story), so I wrote
script, that did next:
When free space < %3 on some of partitions, I run pkill -STOP sort, and
then move some files (that not currently opened by sort(1)) to partition
that have more free space, and create symlinks for old locations.
When all the free space was eliminated on all available partitions, I
archived existed tmp files (that also not opened by sort(1)) and create
a pipes, that redirects output of $(gzip -d) into it, and using this
hacks sort finished successfully for me.


> 
> I don't have the energy right now to write that, but if someone else wrote
> it I'd review it.

Thanks for you notes Pádraig.

-- 
Respectfully
Azat Khuzhin

bug#17601: printf not interpreting hex escapes

[Phillip Susi]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On 05/26/2014 12:50 PM, Pádraig Brady wrote:
> $ type printf printf is a shell builtin

Of course, darn builtins!  Sorry for the noise.


-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCgAGBQJTg3sMAAoJEI5FoCIzSKrwP80H+QH46DNMtO7DJalEa1UQrpXL
xadU4ahdEtD4VUvqEcOX6LFDmmKgqAHAOnvQyT9rojAysv+EzTvxZ9Ni4ebI7pqE
BFiE0j3k55qsTxN92ZyhKE5WZt+CYEp2vaHSyrRsWbztOss2Mh0xCI0eGJfIEGSa
fmJFIBxz92ZU0xR6NPC+obA/8GJb6St6pJLQGomRskbNrpB5GaJOn5Jdrg3dMCQI
XVpSJp42acZ0Ivb+aplQXoZrujfzlGzhcUOoRtx8FQdCmshWMYNeV8+HV0zqNoVd
TYpBnqbdJXg2eS7caL7Wd0HxwGXAheYDjr9qebb8nDLnhzG+KXpZoYLlfinKcXs=
=dH7q
-END PGP SIGNATURE-

bug#17601: printf not interpreting hex escapes

[Pádraig Brady]

tag 17601 notabug
close 17601
stop

On 05/26/2014 05:41 PM, Phillip Susi wrote:
> printf '\x41' prints "x41" instead of "A".
> 
> Also it has no --help or --version switch, but I seem to be running
> version 8.21-1ubuntu5 ( obviously on ubuntu ).

$ type printf
printf is a shell builtin

$ dash -c "printf '\x41\n'"
\x41

$ bash -c "printf '\x41\n'"
A

$ dash -c "env printf '\x41\n'"
A

bug#17601: printf not interpreting hex escapes

[Phillip Susi]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

printf '\x41' prints "x41" instead of "A".

Also it has no --help or --version switch, but I seem to be running
version 8.21-1ubuntu5 ( obviously on ubuntu ).

-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCgAGBQJTg27TAAoJEI5FoCIzSKrwcTAH/1zFzoplzkWtkhNFilJr+2aX
pwHqEhtTI1V1zolTE72R8fKn52XoXaR2LXTBUZdf170r4vmlSKCbb/dzZSbaLRLr
VHsDGn4lhr/kgQfQKx1hX5/K90w/VwnKTxmJiQ02by6kF/c78wT1x0J7BGP1i+34
jJWOkqRwckFoGyYXJP4s0W+Mr7leNMU9KWXG4NdjKBUQepBJxdK06c9DRxrUy/+M
ZkqUwcfQpdx7S6UASjD4c15Eg/fb3hEXLOYsmIzz1G+iS4Zac6u9PRRnhgJj/dlW
NH6YZcdaSrqSQ6P13fDODE9nU8X8Fwy5nQx3MpBT36+ptb2ewL35DaKDIr1eSw8=
=Cr+I
-END PGP SIGNATURE-

bug#17600: Maybe a bug of `date`

[Pádraig Brady]

tag 17600 notabug
close 17600
stop

On 05/26/2014 09:45 AM, HoHo Zhao wrote:
> Hi team,
> 
> I found this "bug" maybe I am lazy to look through the info document.
> 
> Correct:
>   $ TZ=UTC date -d "15:00 BST"
>   Mon May 26 14:00:00 UTC 2014
> 
> Correct:
>   $ TZ=UTC date -d "15:00 EST"
>   Mon May 26 20:00:00 UTC 2014
> 
> Correct:
>   $ TZ=UTC date -d "15:00 JST"   (Japan Standard Time)
>   Mon May 26 06:00:00 UTC 2014
> 
> Wrong:
>   $ TZ=UTC date -d "15:00 CST"   (China Standard Time)
>   Mon May 26 21:00:00 UTC 2014
> 
> So the problem is with "CST" in the date STRING.
> 
> Thanks,

I think this is due to the ambiguity of CST.
You're better using unambiguous location based zones

  TZ=UTC date -d 'TZ="Asia/Chongqing" 15:00'

More details at http://www.pixelbeat.org/docs/linux_timezones/

thanks,
Pádraig.

bug#17600: Maybe a bug of `date`

[HoHo Zhao]

Hi team,

I found this "bug" maybe I am lazy to look through the info document.

Correct:
$ TZ=UTC date -d "15:00 BST"
Mon May 26 14:00:00 UTC 2014

Correct:
$ TZ=UTC date -d "15:00 EST"
Mon May 26 20:00:00 UTC 2014

Correct:
$ TZ=UTC date -d "15:00 JST"   (Japan Standard Time)
Mon May 26 06:00:00 UTC 2014

Wrong:
$ TZ=UTC date -d "15:00 CST"   (China Standard Time)
Mon May 26 21:00:00 UTC 2014

So the problem is with "CST" in the date STRING.

Thanks,
-- 
HoHo Zhao
Regional IT Support
Red Hat Greater China

bug#17590: [PATCH] build: libstdbuf.so: avoid new OS X link failure

[Jim Meyering]

On Mon, May 26, 2014 at 1:25 AM, Pádraig Brady  wrote:
> On 05/25/2014 11:19 PM, Jim Meyering wrote:
>> On Sun, May 25, 2014 at 1:31 PM, Pádraig Brady  wrote:
>>> On 05/25/2014 08:48 PM, Jim Meyering wrote:
 Without the attached patch, I'd get this new link failure on OS X:

 Undefined symbols for architecture x86_64:
   "_libintl_gettext", referenced from:
   _apply_mode in src_libstdbuf_so-libstdbuf.o
 ld: symbol(s) not found for architecture x86_64
 collect2: error: ld returned 1 exit status
 make[2]: *** [src/libstdbuf.so] Error 1
>>>
>>> Oh cool, I presume that's since I generalized the
>>> stdbuf enablement check that stdbuf is now built
>>> on Mac OS X. I presume it works too or you would
>>> have seen the test failure.
>>>
>>> Change looks good.
>>
>> Pushed.  Unfortunately, once past that link failure,
>> the test of new-to-OSX stdbuf fails.  Here's stdbuf.log:
>
> That shows that the test is correct,
> and indicates that the buffering settings were ignored.
>
> I did a very quick search which suggests something
> like the attached might work (assuming the build params
> we hardcode for building the shared lib are OK).

You appear to have nailed it.  The patch looks fine, and
with that, the test now passes on OS X 10.8.5.

bug#17578: cp(1) man page: behavior of "cp -R dir target" on symbolic links is unclear

[Vincent Lefevre]

On 2014-05-26 09:59:04 +0100, Pádraig Brady wrote:
> Yes we shouldn't assume users would know that -H and -L would override -P.
> Also --link is recently wound up in this. Hence it's not easy to describe.

It could be described like in the info file.

> We could accurately describe using something like the following, but given
> that the info docs are correct and complete here

Not all users use the info docs (I find man / --help easier to use).

> and that most users use -a anyway which already explicitly considers
> --no-dereference,

That's the point of this bug report: I usually use -a, which implies
--no-dereference, so that I thought that -R alone would avoid this
--no-dereference, but this is not the way cp behaves.

> it's probably best to err on the side of conciseness in the
> man/--help and let users refer to info for complete docs.

That would just be one sentence below the list of options (a bit like
in the info doc).

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

bug#17578: cp(1) man page: behavior of "cp -R dir target" on symbolic links is unclear

[Pádraig Brady]

tag 17578 wontfix
close 17578
stop

On 05/25/2014 09:27 PM, Paul Eggert wrote:
> Pádraig Brady wrote:
>>-  -R, -r, --recursive  copy directories recursively
>>+  -R, -r, --recursive  copy directories recursively (implies -P)
> 
> Unfortunately it's not that simple, as -R doesn't always imply -P.

Yes we shouldn't assume users would know that -H and -L would override -P.
Also --link is recently wound up in this. Hence it's not easy to describe.
We could accurately describe using something like the following, but given
that the info docs are correct and complete here and that most users use
-a anyway which already explicitly considers --no-dereference, it's probably
best to err on the side of conciseness in the man/--help and let users
refer to info for complete docs.

thanks,
Pádraig.

diff --git a/src/cp.c b/src/cp.c
index 243d6af..878266a 100644
--- a/src/cp.c
+++ b/src/cp.c
@@ -206,7 +206,8 @@ Copy SOURCE to DEST, or multiple SOURCE(s) to DIRECTORY.\n\
   --parentsuse full source file name under DIRECTORY\n\
 "), stdout);
   fputs (_("\
-  -R, -r, --recursive  copy directories recursively (implies -P)\n\
+  -R, -r, --recursive  copy directories recursively. Without -H,-L 
or\n\
+ --link, implies --no-dereference\n\
   --reflink[=WHEN] control clone/CoW copies. See below\n\
   --remove-destination remove each existing destination file before\n\
  attempting to open it (contrast with --force)\

bug#17590: [PATCH] build: libstdbuf.so: avoid new OS X link failure

[Pádraig Brady]

On 05/25/2014 11:19 PM, Jim Meyering wrote:
> On Sun, May 25, 2014 at 1:31 PM, Pádraig Brady  wrote:
>> On 05/25/2014 08:48 PM, Jim Meyering wrote:
>>> Without the attached patch, I'd get this new link failure on OS X:
>>>
>>> Undefined symbols for architecture x86_64:
>>>   "_libintl_gettext", referenced from:
>>>   _apply_mode in src_libstdbuf_so-libstdbuf.o
>>> ld: symbol(s) not found for architecture x86_64
>>> collect2: error: ld returned 1 exit status
>>> make[2]: *** [src/libstdbuf.so] Error 1
>>
>> Oh cool, I presume that's since I generalized the
>> stdbuf enablement check that stdbuf is now built
>> on Mac OS X. I presume it works too or you would
>> have seen the test failure.
>>
>> Change looks good.
> 
> Pushed.  Unfortunately, once past that link failure,
> the test of new-to-OSX stdbuf fails.  Here's stdbuf.log:

That shows that the test is correct,
and indicates that the buffering settings were ignored.

I did a very quick search which suggests something
like the attached might work (assuming the build params
we hardcode for building the shared lib are OK).

thanks,
Pádraig.
>From 47b39387fd18039c24325fdffd5f99435b5ee978 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= 
Date: Mon, 26 May 2014 09:19:16 +0100
Subject: [PATCH] stdbuf: support OS X

* src/stdbuf.c (set_LD_PRELOAD): Adjust to use Mac OS X
specific environment variables in __APPLE__ platforms.
Fixes http://bugs.gnu.org/17590
---
 src/stdbuf.c |   15 ---
 1 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/src/stdbuf.c b/src/stdbuf.c
index c648fc5..92ee282 100644
--- a/src/stdbuf.c
+++ b/src/stdbuf.c
@@ -187,7 +187,12 @@ static void
 set_LD_PRELOAD (void)
 {
   int ret;
-  char *old_libs = getenv ("LD_PRELOAD");
+#ifdef __APPLE__
+  char const *preload_env = "DYLD_INSERT_LIBRARIES";
+#else
+  char const *preload_env = "LD_PRELOAD";
+#endif
+  char *old_libs = getenv (preload_env);
   char *LD_PRELOAD;
 
   /* Note this would auto add the appropriate search path for "libstdbuf.so":
@@ -239,9 +244,9 @@ set_LD_PRELOAD (void)
   /* FIXME: Do we need to support libstdbuf.dll, c:, '\' separators etc?  */
 
   if (old_libs)
-ret = asprintf (&LD_PRELOAD, "LD_PRELOAD=%s:%s", old_libs, libstdbuf);
+ret = asprintf (&LD_PRELOAD, "%s=%s:%s", preload_env, old_libs, libstdbuf);
   else
-ret = asprintf (&LD_PRELOAD, "LD_PRELOAD=%s", libstdbuf);
+ret = asprintf (&LD_PRELOAD, "%s=%s", preload_env, libstdbuf);
 
   if (ret < 0)
 xalloc_die ();
@@ -249,6 +254,10 @@ set_LD_PRELOAD (void)
   free (libstdbuf);
 
   ret = putenv (LD_PRELOAD);
+#ifdef __APPLE__
+  if (ret == 0)
+ret = putenv ("DYLD_FORCE_FLAT_NAMESPACE=y");
+#endif
 
   if (ret != 0)
 {
-- 
1.7.7.6