Re: gnu-web-doc-update: fails to add new directories

[Akim Demaille]

Hi Pádraig!

> Le 20 janv. 2016 à 15:16, Pádraig Brady  a écrit :
> 
> On 15/12/12 20:41, Jim Meyering wrote:
>> Akim Demaille wrote:
>>> Le 13 déc. 2012 à 18:06, Jim Meyering  a écrit :
>>> 
> cvsu does not report that there are directories and files inside the
> 'D's.  Is it wrong to simply cvs add everything that is there?
> Just skipping CVS, but adding all the rest using "find"?
> Say "find . -name CVS -prune -o -print".
 
 That sounds like it will work.
 Thanks again.
>>> 
>>> Here is my proposal.  I don't see anything ready to _remove_ bits
>>> that are no longer generated.  This is a problem, since for instance
>> 
>> Hi Akim!
>> Yes, it'd be good to fix that, someday.
> 
> FYI I fixed that a while ago with:
> http://git.sv.gnu.org/gitweb/?p=gnulib.git;a=commit;h=9a417cf

Thanks for letting me know!

Wow…

+  { find . \( -name CVS -o -type f -name '.*' \) -prune -o -type f -print
+(cd "$builddir"/doc/manual/ && find . -type f -print | sed p)
+  } | sort | uniq -u \
+| $XARGS --no-run-if-empty -- ${rm_stale:-$dryrun} $CVS remove -f

the ‘sed p | uniq -u’ sequence is super smart :)  I guess you had a lot of fun 
finding it :)

> 
>>> commit a1d1f4b7fc9b46fc37a8dc55b591768a80ead566
>>> Author: Akim Demaille 
>>> Date:   Fri Dec 14 12:58:35 2012 +0100
>>> 
>>>gnu-web-doc-update: add all the new files, even in new directories
>>> 
>>>See http://lists.gnu.org/archive/html/bug-gnulib/2012-12/msg00057.html
>>> 
>>>* build-aux/gnu-web-doc-update (--dry-run, $dryrun): New.
>>>Use it.
>>>(main): Don't use cvsutils to get the list of unknown files,
>>>just add all the existing files and directories.
> 
> That change is causing an issue in coreutils wrt empty dirs,
> which causes new files to be missed due to:
>  cvs [add aborted]: there is a version in ./ps already
>  ...
> 
> I've installed the attached change to add dirs separately and individually,
> as cvs seems to have different treatment for files and dirs.

Great!

I wish all this were useless, and CVS totally eradicated :)

Re: gnu-web-doc-update: fails to add new directories

[Pádraig Brady]

On 15/12/12 20:41, Jim Meyering wrote:
> Akim Demaille wrote:
>> Le 13 déc. 2012 à 18:06, Jim Meyering  a écrit :
>>
 cvsu does not report that there are directories and files inside the
 'D's.  Is it wrong to simply cvs add everything that is there?
 Just skipping CVS, but adding all the rest using "find"?
 Say "find . -name CVS -prune -o -print".
>>>
>>> That sounds like it will work.
>>> Thanks again.
>>
>> Here is my proposal.  I don't see anything ready to _remove_ bits
>> that are no longer generated.  This is a problem, since for instance
> 
> Hi Akim!
> Yes, it'd be good to fix that, someday.

FYI I fixed that a while ago with:
http://git.sv.gnu.org/gitweb/?p=gnulib.git;a=commit;h=9a417cf

>> commit a1d1f4b7fc9b46fc37a8dc55b591768a80ead566
>> Author: Akim Demaille 
>> Date:   Fri Dec 14 12:58:35 2012 +0100
>>
>> gnu-web-doc-update: add all the new files, even in new directories
>>
>> See http://lists.gnu.org/archive/html/bug-gnulib/2012-12/msg00057.html
>>
>> * build-aux/gnu-web-doc-update (--dry-run, $dryrun): New.
>> Use it.
>> (main): Don't use cvsutils to get the list of unknown files,
>> just add all the existing files and directories.

That change is causing an issue in coreutils wrt empty dirs,
which causes new files to be missed due to:
  cvs [add aborted]: there is a version in ./ps already
  ...

I've installed the attached change to add dirs separately and individually,
as cvs seems to have different treatment for files and dirs.

thanks,
Pádraig.
From 6465306025facca3881bcea411b757b0fe8e8df8 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= 
Date: Wed, 20 Jan 2016 14:10:30 +
Subject: [PATCH] gnu-web-doc-update: fix addition of new files

If there were already added (emnpty) dirs,
then cvs aborts the add with the message:
  cvs [add aborted]: there is a version in <./dirname> already

* build-aux/gnu-web-doc-update: Add directories separately
to the addition of files, to avoid the above issue
impacting the addition of files.
---
 ChangeLog| 10 ++
 build-aux/gnu-web-doc-update | 11 ++-
 2 files changed, 16 insertions(+), 5 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 9614d58..1bdf766 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,13 @@
+2016-01-20  Pádraig Brady  
+
+	gnu-web-doc-update: fix addition of new files
+	If there were already added (emnpty) dirs,
+	then cvs aborts the add with the message:
+	  cvs [add aborted]: there is a version in <./dirname> already
+	* build-aux/gnu-web-doc-update: Add directories separately
+	to the addition of files, to avoid the above issue
+	impacting the addition of files.
+
 2016-01-19  Daiki Ueno  
 
 	utimens-tests: avoid pulling gettext .m4 files
diff --git a/build-aux/gnu-web-doc-update b/build-aux/gnu-web-doc-update
index ec21168..271e693 100755
--- a/build-aux/gnu-web-doc-update
+++ b/build-aux/gnu-web-doc-update
@@ -179,11 +179,12 @@ $RSYNC -avP "$builddir"/doc/manual/ $tmp/$pkg/manual
   cd $tmp/$pkg/manual
 
   # Add all the files.  This is simpler than trying to add only the
-  # new ones because of new directories: it would require iterating on
-  # adding the outer directories, and then their contents.
-  #
-  # find guarantees that we add outer directories first.
-  find . -name CVS -prune -o -print \
+  # new ones because of new directories
+  # First add non empty dirs individually
+  find . -name CVS -prune -o -type d \! -empty -print \
+| $XARGS -n1 --no-run-if-empty -- $dryrun $CVS add -ko
+  # Now add all files
+  find . -name CVS -prune -o -type f -print \
 | $XARGS --no-run-if-empty -- $dryrun $CVS add -ko
 
   # Report/Remove stale files
-- 
2.5.0

Re: gnu-web-doc-update: fails to add new directories

[Akim Demaille]

Le 15 déc. 2012 à 23:07, Jim Meyering j...@meyering.net a écrit :

 Akim Demaille wrote:
 ...
 Did you intend to remove this comment?
 I know they're also in --help output, but it's probably
 best to leave them at the top, too.
 
 -# Run this after each non-alpha release, to update the web documentation 
 at
 -# http://www.gnu.org/software/$pkg/manual/
 
 I'm slightly confused by the singular vs. plural.  You mean
 
 As well you should be.  Initially I wrote about the two lines, then
 did s/.../comment/, but didn't remove the s in the following sentence.
 
 keeping this only comment on top as a short description of
 what this file is, right?  In which case, I concur, I was too
 eager :)
 
 Right.  Thanks.

Installed as follows.

commit a42a32e05e8406424d702f6a9e0e2c12b7e460ea
Author: Akim Demaille a...@lrde.epita.fr
Date:   Fri Dec 14 12:53:34 2012 +0100

gnu-web-doc-update: improve --help

* build-aux/gnu-web-doc-update: Move comments into --help.

diff --git a/ChangeLog b/ChangeLog
index bdd4ad5..fc36ecd 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,8 @@
+2012-12-16  Akim Demaille  a...@lrde.epita.fr
+
+   gnu-web-doc-update: improve --help
+   * build-aux/gnu-web-doc-update: Move comments into --help.
+
 2012-12-07  Eric Wong  normalper...@yhbt.net
 
mountlist: recognize more dummy file systems
diff --git a/build-aux/gnu-web-doc-update b/build-aux/gnu-web-doc-update
index 4acd69d..e09db86 100755
--- a/build-aux/gnu-web-doc-update
+++ b/build-aux/gnu-web-doc-update
@@ -1,13 +1,8 @@
 #!/bin/sh
 # Run this after each non-alpha release, to update the web documentation at
 # http://www.gnu.org/software/$pkg/manual/
-# This script must be run from the top-level directory,
-# assumes you're using git for revision control,
-# and requires a .prev-version file as well as a Makefile,
-# from which it extracts the version number and package name, respectively.
-# Also, it assumes all documentation is in the doc/ sub-directory.
 
-VERSION=2009-07-21.16; # UTC
+VERSION=2012-12-16.14; # UTC
 
 # Copyright (C) 2009-2012 Free Software Foundation, Inc.
 
@@ -37,6 +32,11 @@ Run this script from top_srcdir (no arguments) after each 
non-alpha
 release, to update the web documentation at
 http://www.gnu.org/software/\$pkg/manual/
 
+This script assumes you're using git for revision control, and
+requires a .prev-version file as well as a Makefile, from which it
+extracts the version number and package name, respectively.  Also, it
+assumes all documentation is in the doc/ sub-directory.
+
 Options:
   -C, --builddir=DIR  location of (configured) Makefile (default: .)
   --help  print this help, then exit

Re: gnu-web-doc-update: fails to add new directories

[Jim Meyering]

Akim Demaille wrote:
 Le 13 déc. 2012 à 18:06, Jim Meyering j...@meyering.net a écrit :

 cvsu does not report that there are directories and files inside the
 'D's.  Is it wrong to simply cvs add everything that is there?
 Just skipping CVS, but adding all the rest using find?
 Say find . -name CVS -prune -o -print.

 That sounds like it will work.
 Thanks again.

 Here is my proposal.  I don't see anything ready to _remove_ bits
 that are no longer generated.  This is a problem, since for instance

Hi Akim!
Yes, it'd be good to fix that, someday.

 support for PS was recently dropped in gendocs.sh.  So I can still get
 http://www.gnu.org/software/bison/manual/bison.ps from gnu.org,
 which is about an older version of Bison (2.6.4).  Besides, for some
 reason I have not tried to explore, bison.ps is actually a ps.gz
 file :(

 commit a1d1f4b7fc9b46fc37a8dc55b591768a80ead566
 Author: Akim Demaille a...@lrde.epita.fr
 Date:   Fri Dec 14 12:58:35 2012 +0100

 gnu-web-doc-update: add all the new files, even in new directories

 See http://lists.gnu.org/archive/html/bug-gnulib/2012-12/msg00057.html

 * build-aux/gnu-web-doc-update (--dry-run, $dryrun): New.
 Use it.
 (main): Don't use cvsutils to get the list of unknown files,
 just add all the existing files and directories.
...

Thanks for working on this!

 diff --git a/build-aux/gnu-web-doc-update b/build-aux/gnu-web-doc-update
 index 4acd69d..6b88bd8 100755
 --- a/build-aux/gnu-web-doc-update
 +++ b/build-aux/gnu-web-doc-update
 @@ -1,13 +1,5 @@
  #!/bin/sh
 -# Run this after each non-alpha release, to update the web documentation at
 -# http://www.gnu.org/software/$pkg/manual/
 -# This script must be run from the top-level directory,
 -# assumes you're using git for revision control,
 -# and requires a .prev-version file as well as a Makefile,
 -# from which it extracts the version number and package name, respectively.
 -# Also, it assumes all documentation is in the doc/ sub-directory.
 -
 -VERSION=2009-07-21.16; # UTC
 +VERSION=2012-12-14.11; # UTC

  # Copyright (C) 2009-2012 Free Software Foundation, Inc.

 @@ -37,6 +29,11 @@ Run this script from top_srcdir (no arguments) after each 
 non-alpha
  release, to update the web documentation at
  http://www.gnu.org/software/\$pkg/manual/

 +This script assumes you're using git for revision control, and
 +requires a .prev-version file as well as a Makefile, from which it
 +extracts the version number and package name, respectively.  Also, it
 +assumes all documentation is in the doc/ sub-directory.

Did you intend to remove this comment?
I know they're also in --help output, but it's probably
best to leave them at the top, too.

 -# Run this after each non-alpha release, to update the web documentation at
 -# http://www.gnu.org/software/$pkg/manual/

Re: gnu-web-doc-update: fails to add new directories

[Akim Demaille]

hi Jim!
Le 15 déc. 2012 à 21:41, Jim Meyering j...@meyering.net a écrit :

 Thanks for working on this!

Retour d'ascenseur :)  I'm using gnulib, so it's just fair.

 +This script assumes you're using git for revision control, and
 +requires a .prev-version file as well as a Makefile, from which it
 +extracts the version number and package name, respectively.  Also, it
 +assumes all documentation is in the doc/ sub-directory.
 
 Did you intend to remove this comment?
 I know they're also in --help output, but it's probably
 best to leave them at the top, too.
 
 -# Run this after each non-alpha release, to update the web documentation at
 -# http://www.gnu.org/software/$pkg/manual/

I'm slightly confused by the singular vs. plural.  You mean
keeping this only comment on top as a short description of
what this file is, right?  In which case, I concur, I was too
eager :)

Re: gnu-web-doc-update: fails to add new directories

[Jim Meyering]

Akim Demaille wrote:
...
 Did you intend to remove this comment?
 I know they're also in --help output, but it's probably
 best to leave them at the top, too.

 -# Run this after each non-alpha release, to update the web documentation at
 -# http://www.gnu.org/software/$pkg/manual/

 I'm slightly confused by the singular vs. plural.  You mean

As well you should be.  Initially I wrote about the two lines, then
did s/.../comment/, but didn't remove the s in the following sentence.

 keeping this only comment on top as a short description of
 what this file is, right?  In which case, I concur, I was too
 eager :)

Right.  Thanks.

Re: [off-list] CVS pains (was: Re: gnu-web-doc-update: fails to add new directories)

[Stefano Lattarini]

On 12/13/2012 11:53 PM, Karl Berry wrote:

 3) Everyone agrees it would be nice to support different VCs for web
 pages.  Nobody needs convincing.

This is good to know.

 Maybe it will happen some day.

I hope so.  It's not a big deal actually (since as Jim noted,
most of the process is hidden and/or automated away), but is
still another little grating annoyance that would be nice to
finally see disappear.

Thanks,
  Stefano

Re: gnu-web-doc-update: fails to add new directories

[Akim Demaille]

Le 13 déc. 2012 à 18:06, Jim Meyering j...@meyering.net a écrit :

 cvsu does not report that there are directories and files inside the
 'D's.  Is it wrong to simply cvs add everything that is there?
 Just skipping CVS, but adding all the rest using find?
 Say find . -name CVS -prune -o -print.
 
 That sounds like it will work.
 Thanks again.

Here is my proposal.  I don't see anything ready to _remove_ bits
that are no longer generated.  This is a problem, since for instance
support for PS was recently dropped in gendocs.sh.  So I can still get
http://www.gnu.org/software/bison/manual/bison.ps from gnu.org,
which is about an older version of Bison (2.6.4).  Besides, for some
reason I have not tried to explore, bison.ps is actually a ps.gz
file :(

commit a1d1f4b7fc9b46fc37a8dc55b591768a80ead566
Author: Akim Demaille a...@lrde.epita.fr
Date:   Fri Dec 14 12:58:35 2012 +0100

gnu-web-doc-update: add all the new files, even in new directories

See http://lists.gnu.org/archive/html/bug-gnulib/2012-12/msg00057.html

* build-aux/gnu-web-doc-update (--dry-run, $dryrun): New.
Use it.
(main): Don't use cvsutils to get the list of unknown files,
just add all the existing files and directories.

diff --git a/ChangeLog b/ChangeLog
index af716b1..a63dde1 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,14 @@
 2012-12-14  Akim Demaille  a...@lrde.epita.fr
 
+   gnu-web-doc-update: add all the new files, even in new directories
+   See http://lists.gnu.org/archive/html/bug-gnulib/2012-12/msg00057.html
+   * build-aux/gnu-web-doc-update (--dry-run, $dryrun): New.
+   Use it.
+   (main): Don't use cvsutils to get the list of unknown files,
+   just add all the existing files and directories.
+
+2012-12-14  Akim Demaille  a...@lrde.epita.fr
+
gnu-web-doc-update: improve --help
* build-aux/gnu-web-doc-update: Move comments into --help.
 
diff --git a/build-aux/gnu-web-doc-update b/build-aux/gnu-web-doc-update
index 6b88bd8..cc454e7 100755
--- a/build-aux/gnu-web-doc-update
+++ b/build-aux/gnu-web-doc-update
@@ -36,6 +36,7 @@ assumes all documentation is in the doc/ sub-directory.
 
 Options:
   -C, --builddir=DIR  location of (configured) Makefile (default: .)
+  -n, --dry-run   don't actually commit anything
   --help  print this help, then exit
   --version   print version number, then exit
 
@@ -97,12 +98,12 @@ find_tool ()
 # Requirements: everything required to bootstrap your package, plus
 # these.
 find_tool CVS cvs
-find_tool CVSU cvsu
 find_tool GIT git
 find_tool RSYNC rsync
 find_tool XARGS gxargs xargs
 
 builddir=.
+dryrun=
 while test $# != 0
 do
   # Handle --option=value by splitting apart and putting back on argv.
@@ -118,6 +119,7 @@ do
   case $1 in
 --help|--version) ${1#--};;
 -C|--builddir) shift; builddir=$1; shift ;;
+-n|--dry-run) dryrun=echo; shift;;
 --*) die unrecognized option: $1;;
 *) break;;
   esac
@@ -136,7 +138,7 @@ current_branch=$($GIT branch | sed -ne '/^\* /{s///;p;q;}')
 cleanup()
 {
   __st=$?
-  rm -rf $tmp
+  $dryrun rm -rf $tmp
   $GIT checkout $current_branch
   $GIT submodule update --recursive
   $GIT branch -d $tmp_branch
@@ -169,12 +171,15 @@ $RSYNC -avP $builddir/doc/manual/ $tmp/$pkg/manual
 (
   cd $tmp/$pkg/manual
 
-  # Add any new files:
-  $CVSU --types='?' \
-| sed s/..//\
-| $XARGS --no-run-if-empty -- $CVS add -ko
+  # Add all the files.  This is simpler than trying to add only the
+  # new ones because of new directories: it would require iterating on
+  # adding the outer directories, and then their contents.
+  #
+  # find guarantees that we add outer directories first.
+  find . -name CVS -prune -o -print \
+| $XARGS --no-run-if-empty -- $dryrun $CVS add -ko
 
-  $CVS ci -m $version
+  $dryrun $CVS ci -m $version
 )
 
 # Local variables:

commit 59e437ebc75cec5211c86db49b9ff2eb6cf5758d
Author: Akim Demaille a...@lrde.epita.fr
Date:   Fri Dec 14 12:53:34 2012 +0100

gnu-web-doc-update: improve --help

* build-aux/gnu-web-doc-update: Move comments into --help.

diff --git a/ChangeLog b/ChangeLog
index 09f7753..af716b1 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,8 @@
+2012-12-14  Akim Demaille  a...@lrde.epita.fr
+
+   gnu-web-doc-update: improve --help
+   * build-aux/gnu-web-doc-update: Move comments into --help.
+
 2012-12-11  Paul Eggert  egg...@cs.ucla.edu
 
extern-inline: avoid incompatibility with Darwin Libc
diff --git a/build-aux/gnu-web-doc-update b/build-aux/gnu-web-doc-update
index 4acd69d..6b88bd8 100755
--- a/build-aux/gnu-web-doc-update
+++ b/build-aux/gnu-web-doc-update
@@ -1,13 +1,5 @@
 #!/bin/sh
-# Run this after each non-alpha release, to update the web documentation at
-# http://www.gnu.org/software/$pkg/manual/
-# This script must be run from the top-level directory,
-# assumes you're using git for revision 

Re: gnu-web-doc-update: fails to add new directories

[Jim Meyering]

Akim Demaille wrote:

 The Bison's documentation now uses images, including in HTML.
 gendocs.sh properly sets up a directory with the images for all the
 documentation formats.  Unfortunately gnu-web-doc-update fails to add
 all the new files, because it fails to discover new directories.

 My CVS-skills are, ahem, rusty, and I have never been a cvsutils user,
 so I don't know if the bug is in cvsu or gnu-web-doc-update, but the
 problem is in these bits:

 (
   cd $tmp/$pkg/manual

   # Add any new files:
   $CVSU --types='?' \
 | sed s/..//\
 | $XARGS --no-run-if-empty -- $CVS add -ko

   $CVS ci -m $version
 )

 in Bison's case, the unknown directories are not reported as ?, but as D:
...

 So I don't know if gnu-web-doc-update should learn about --type=D,
 or if cvsu should mention unknown directories with '?'.

Hi Akim!

Thanks for the investigation and report.
If using --types='?D' makes it work, I'd go with that:
that seems simpler than changing the external cvsu program.

Re: gnu-web-doc-update: fails to add new directories

[Akim Demaille]

Le 13 déc. 2012 à 15:35, Jim Meyering j...@meyering.net a écrit :

 Hi Akim!

Hi Jim!

 Thanks for the investigation and report.
 If using --types='?D' makes it work, I'd go with that:
 that seems simpler than changing the external cvsu program.

OK.  But it will require some implementation of a fix point.

$ ls
bison
$ cd bison/
$ mkdir too
$ cvsu
D too
$ mkdir foo
$ mkdir foo/bar
$ touch foo/bar/baz
$ touch qux
$ cvsu
D foo
? qux
D too
$ cvsu --types='D?'
D foo
? qux
D too

cvsu does not report that there are directories and files inside the
'D's.  Is it wrong to simply cvs add everything that is there?
Just skipping CVS, but adding all the rest using find?
Say find . -name CVS -prune -o -print.

[off-list] CVS pains (was: Re: gnu-web-doc-update: fails to add new directories)

[Stefano Lattarini]

Ah, we all git lovers still forced to deal with that living fossil
of CVS!

Wouldn't it be great if someone could convince the Savannah admin
to switch the web pages repository to a modern, distributed VCS?

(Me ducks and run now ;-).

Regards,
  Stefano

Re: gnu-web-doc-update: fails to add new directories

[Jim Meyering]

Akim Demaille wrote:

 Le 13 déc. 2012 à 15:35, Jim Meyering j...@meyering.net a écrit :

 Hi Akim!

 Hi Jim!

 Thanks for the investigation and report.
 If using --types='?D' makes it work, I'd go with that:
 that seems simpler than changing the external cvsu program.

 OK.  But it will require some implementation of a fix point.

 $ ls
 bison
 $ cd bison/
 $ mkdir too
 $ cvsu
 D too
 $ mkdir foo
 $ mkdir foo/bar
 $ touch foo/bar/baz
 $ touch qux
 $ cvsu
 D foo
 ? qux
 D too
 $ cvsu --types='D?'
 D foo
 ? qux
 D too

 cvsu does not report that there are directories and files inside the
 'D's.  Is it wrong to simply cvs add everything that is there?
 Just skipping CVS, but adding all the rest using find?
 Say find . -name CVS -prune -o -print.

That sounds like it will work.
Thanks again.

Re: [off-list] CVS pains (was: Re: gnu-web-doc-update: fails to add new directories)

[Karl Berry]

Wouldn't it be great if someone could convince the Savannah admin

1) It's not a matter of just changing savannah.  There are numerous
(*numerous*) processes involved with web pages repositories, most of
which are written, maintained, and controlled by the FSF sysadmins, not
Savannah.  FSF sysadmin time is always at a premium.

2) Even if it was just Savannah, Savannah development is really just one
person right now - Michael Flickinger.  There are many other things he
has to do (like restoring the documentation wiki ...).

3) Everyone agrees it would be nice to support different VCs for web
pages.  Nobody needs convincing.  Maybe it will happen some day.

k