Re: man pages for Subversion

[Branko Čibej]

On 14.08.2013 23:42, Thomas Åkesson wrote:
>
> On 14 aug 2013, at 20:47, Mattias Engdegård  wrote:
>
>> 12 aug 2013 kl. 12.38 skrev Julian Foad:
>>
>>> Hi James.  I have one thing to throw into the mix, which you might be 
>>> interested in looking at.  I experimented a few months ago with generating 
>>> both the C help strings and man pages from an XML source file.  My patch to 
>>> do this is attached; it is complete and and usable, although not finished 
>>> or quite how I would want it.
>> How about including the marked-up source in the svn binary (as C constant 
>> strings) and do the rendering at run time, when the user does "svn help"? 
>> Otherwise, it would be tricky to handle translations properly - the 
>> translator must work on the marked-up text and it should be possible to try 
>> out new translations without rebuilding the binaries.
> Is that possible today?

Of course. You can, for example, take an installed Subversion and
replace one of its .mo files, and it'll pick up the new translations.

If that doesn't work, I'd like to know about it because it'd be a
serious bug in how we use gettext.

-- Brane


-- 
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. br...@wandisco.com

Re: man pages for Subversion

[Thomas Åkesson]

On 14 aug 2013, at 20:47, Mattias Engdegård  wrote:

> 12 aug 2013 kl. 12.38 skrev Julian Foad:
> 
>> Hi James.  I have one thing to throw into the mix, which you might be 
>> interested in looking at.  I experimented a few months ago with generating 
>> both the C help strings and man pages from an XML source file.  My patch to 
>> do this is attached; it is complete and and usable, although not finished or 
>> quite how I would want it.
> 
> How about including the marked-up source in the svn binary (as C constant 
> strings) and do the rendering at run time, when the user does "svn help"? 
> Otherwise, it would be tricky to handle translations properly - the 
> translator must work on the marked-up text and it should be possible to try 
> out new translations without rebuilding the binaries.

Is that possible today?

> This has the extra benefit of allowing the help text to be rendered to the 
> width of the terminal and to (optionally) use terminal effects such as bold 
> and underline.

Given the verbosity of some help texts I definitely think the source format 
should not care about terminal width. It must be a major pain to translate 
given the line breaks? Whether the line breaks are introduced during runtime or 
build matters less, but the later the better. 

> The original could either live in a separate file, as in your patch, or as 
> actual C constants. In any case, it probably makes things a lot easier for 
> the translators if the text appears as strings in the same subversion.pot as 
> everything else.
> 
> (Personally I would prefer something more light-weight than XML, but perhaps 
> it would give you an excuse to allow the --xml flag with svn help.)

Many recently developed software frameworks define their strings in XML because:
a) translation processes are well defined for XML, in a larger perspective 
since almost all documentation is done in XML (ok, not a great driver in this 
project except keeping svnbook up to date)
b) XML is so easy to transform into multiple distribution formats. 

I know there is a lot of resistance toward the heavy-weight XML, but authoring 
is actually where XML excels (as opposed to data interchange where lighter 
formats can be preferred). 

Cheers,
Thomas Å.

Re: man pages for Subversion

[Mattias Engdegård]

12 aug 2013 kl. 12.38 skrev Julian Foad:

Hi James.  I have one thing to throw into the mix, which you might  
be interested in looking at.  I experimented a few months ago with  
generating both the C help strings and man pages from an XML source  
file.  My patch to do this is attached; it is complete and and  
usable, although not finished or quite how I would want it.


How about including the marked-up source in the svn binary (as C  
constant strings) and do the rendering at run time, when the user does  
"svn help"? Otherwise, it would be tricky to handle translations  
properly - the translator must work on the marked-up text and it  
should be possible to try out new translations without rebuilding the  
binaries.


This has the extra benefit of allowing the help text to be rendered to  
the width of the terminal and to (optionally) use terminal effects  
such as bold and underline.


The original could either live in a separate file, as in your patch,  
or as actual C constants. In any case, it probably makes things a lot  
easier for the translators if the text appears as strings in the same  
subversion.pot as everything else.


(Personally I would prefer something more light-weight than XML, but  
perhaps it would give you an excuse to allow the --xml flag with svn  
help.)

Re: man pages for Subversion

[Thomas Åkesson]

Hi Julian et al,
I think your XML approach is the way to go in order to fulfill current and 
future requirements. A single source to keep updated is vital. XML is an ideal 
source format for HTML, PDF while plaintext (svn help), man etc should be no 
problems to sort out with XSLT. 

I was unable to find the XML or XSLT in the patch. Since the svnbook is Docbook 
I would go with that schema unless we find very specific requirements that we 
can't fulfill (usually doable with some creative use of attributes). The 
reasonable alternative, given that we prefer standards, would be DITA but that 
is a whole new level of complexity.

How is the stuff that you removed from the .c files translated? Given that 
different languages take very different amounts of space the line breaks must 
to be... well, problematic. 

Going with Docbook would enable the svnbook authors to quite easily generate a 
set of XML fragments from the svn XML. These XML fragments would be at minimum 
screen-elements and probably also some list-stuff. All of which can be 
XIncluded into the Svnbook instead of duplicated.

I can help out with schema discussions and XSLT programming (in September, 
preferably). I have a fair share of experience transforming Docbook-like 
schemas into XHTML, XSL-FO and other formats. 

regards,
Thomas Å.



On 12 aug 2013, at 12:38, Julian Foad  wrote:

> Hi James.  I have one thing to throw into the mix, which you might be 
> interested in looking at.  I experimented a few months ago with generating 
> both the C help strings and man pages from an XML source file.  My patch to 
> do this is attached; it is complete and and usable, although not finished or 
> quite how I would want it.
> 
> One of my requirements was to produce C strings identical to those we have 
> now, in order to prove that it can be done and to be able to check whether 
> I've missed anything.  I invented my own XML schema and hand-wrote the XSL 
> transformations, and to be honest the XSL is ugly.  Before using a method 
> like this, I would want to change it to use a somewhat standard schema, or at 
> least a standard schema as far as possible plus our own extensions for the 
> rest.
> 
> If you want to play with this patch, it applies to svn trunk as of r1456600 
> (March 2013); I haven't updated it since then.  It's integrated into the 
> build system (Unix only, so far), using xsltproc.  To create the man page, 
> use "make subversion/svn/svn-help.man", and use something like "(cd 
> subversion/svn && man -l svn.1)" to view it.  It currently creates a single 
> man page for 'svn', listing all of the subcommands within it.
> 
> 
> - Julian
> 
> 
>  
> --
> Join WANdisco's free daily demo sessions on Scaling Subversion for the 
> Enterprise
> <http://www.wandisco.com/training/webinars>
> 
> 
> 
> - Original Message -
>> From: James K. Lowden 
>> To: Daniel Shahaf 
>> Cc: dev@subversion.apache.org
>> Sent: Saturday, 10 August 2013, 4:22
>> Subject: Re: man pages for Subversion
>> 
>> On Fri, 9 Aug 2013 11:21:23 +0300
>> Daniel Shahaf  wrote:
>> 
>>>> 1.  Get the sources for 1.9 
>>>> 2.  Examine svn_cl__cmd_table
>>>> 3.  Prepare a doc build tree that creates two things:
>>>>   a.  man pages
>>>>   b.  plain text for --help, one file per subcommand
>>>> 
>>>> The plain text filenames will have the extension ".in", so a
>>>> simple Makefile rule can convert them to ".h" for inclusion 
>> into the
>>>> appropriate source module.  
>>> 
>>> You'd have to figure out something for windows as well, since we 
>> don't
>>> use make that on that platform.  (Several of our dependencies do, but
>>> we just generate Visual Studio projects.)
>> 
>> Hmm.  This is a sticky wicket.  
>> 
>> My plan is to keep mdoc sources and generate plain text for
>> inclusion in the help subsystem.  That's done easily with groff, see
>> below.  But I notice in INSTALL you support building on Windows not only
>> in a tarball but also directly in-tree.  That means the Windows
>> build machine would have to be able to convert the mdoc sources to ASCII
>> strings.  
>> 
>> I'd like to point out, btw, that in 2013 this is the tail wagging the
>> dog.  Windows command-line users are rare; most people use Tortoise.
>> Those that do use the command line don't rely on --help very much,
>> either.  Consider:
>> 
>> $ svn propset --help | wc -l
>>   95
>> 
>> and that more.exe resembles UNIX more(1) circa 1984 and tha

RE: man pages for Subversion

[Bob Archer]

> On Fri, 9 Aug 2013 11:21:23 +0300
> Daniel Shahaf  wrote:
> 
> > > 1.  Get the sources for 1.9
> > > 2.  Examine svn_cl__cmd_table
> > > 3.  Prepare a doc build tree that creates two things:
> > >  a.  man pages
> > >  b.  plain text for --help, one file per subcommand
> > >
> > > The plain text filenames will have the extension ".in", so a simple
> > > Makefile rule can convert them to ".h" for inclusion into the
> > > appropriate source module.
> >
> > You'd have to figure out something for windows as well, since we don't
> > use make that on that platform.  (Several of our dependencies do, but
> > we just generate Visual Studio projects.)
> 
> Hmm.  This is a sticky wicket.
> 
> My plan is to keep mdoc sources and generate plain text for inclusion in the
> help subsystem.  That's done easily with groff, see below.  But I notice in
> INSTALL you support building on Windows not only in a tarball but also 
> directly
> in-tree.  That means the Windows build machine would have to be able to
> convert the mdoc sources to ASCII strings.
> 
> I'd like to point out, btw, that in 2013 this is the tail wagging the dog.  
> Windows
> command-line users are rare; most people use Tortoise.

Is this an assumption, a prejudice or wishful thinking? Or, do you have facts 
to back this up? Perhaps your assumption is correct wrt to your average 
mainstream web surfer, but for devs many of them use command line quite a bit. 
While I use tortoise for more common commands, I do drop to the command line 
quite frequently to use svn.exe and to read the help text. 


> Those that do use the command line don't rely on --help very much, either.
> Consider:
> 
>   $ svn propset --help | wc -l
>   95
> 
> and that more.exe resembles UNIX more(1) circa 1984 and that most Windows
> users don't even know it exists.  "svn help" on Windows is not the model of 
> ease
> of use.
> 
> Preserving --help for anything more than a short recap of the options isn't
> worth it, especially when PDFs are available!  The time is much better spent
> improving the documentation and Subversion itself.
> 
> But let us suppose ad argumentum that we want to preserve --help, and to
> support in-tree builds on Windows.  That means something has to convert the
> mdoc source to plain text.  I don't see any better way than to require mdocml
> or groff to be installed.
> 
> > In general, it'll be best if you hook into the existing build system
> > --- configure.ac, gen-make.py, and build.conf.  The latter two are
> > used on all platforms (on unix they generate build-outputs.mk).
> 
> I don't want to get too deeply involved.  I know mdoc, and I want to see man
> pages become part of Subversion.  I'm willing to provide up-to-date mdoc
> source for them, and to generate plain text, and to write a little Perl to 
> convert
> the plain text to static constant character arrays in C. I'm prepared to do 
> the
> work to make it easy
> -- for someone who knows how -- to integrate that work into the build system. 
> I
> don't want to peek inside gen-make.py.
> 
> Below is a sample of what I'd expect from the mdoc -> ascii conversion.
> 
> --jkl
> 
> $ groff -P -bc -Tascii -mdoc man1/svn-copy.1 \
>   | sed -ne '/NAME/,/specify/p'
> NAME
>  svn copy -- Duplicate something in working copy or repository
> 
> SYNOPSIS
>  svn copy (cp) [-Fmq] [--config-dir DIR] [--config-option OPT]
>   [--editor-cmd CMD] [--encoding ENC] [--file] [--force-log]
>   [--ignore-externals] [--message] [--no-auth-cache]
>   [--non-interactive] [--parents] [--password PWD] [--quiet]
>   [{-r | --revision} REV] [--trust-server-cert] [--username USR] 
> [--with-
> revprop PROP] SRC[@REV] ... DST
> 
> DESCRIPTION
>  When copying multiple sources, they will be added as children of DST, 
> which
> must be a directory.  SRC and DST can each be either a working copy (WC) path
> or URL:
>  WC  -> WC  copy and schedule for addition (with history)
>  WC  -> URL immediately commit a copy of WC to URL
>  URL -> WC  check out URL into WC, schedule for addition
>  URL -> URL complete server-side copy;  used to branch and tag
>  All the SRCs must be of the same type.
> 
> OPTIONS
>  --editor-cmd CMD  use CMD as external editor
>  --encoding ENCtreat value as being in charset encoding ENC
>  -F --file FIL read log message from file FIL
>  --force-log   force validity of log message source
>  --ignore-externalsignore externals definitions
>  -m --message MSG  specify log message MSG
> 

RE: man pages for Subversion

[Bert Huijben]

Hi.

Can you please stop bashing Windows. We also want to support OS/2,
Netware, Be/os and any other system that the Apr library supports.

Breaking the command line client for these systems includes breaking
our entire test suite. Svn is how we test the library and we try to
avoid platform specific tests.

Please work towards a solution that helps everybody, instead of trying
to sell us an almost 45 years old solution that only handles a subset
of our requirements.

The svn help infrastructure worked great for the last decade, but can
certainly be improved. *Especially* on Windows, where 'svn help merge'
no longer fits in the default screen buffer.
(note that svn is part of the tortoisesvn setup)

We aim for feature parity between platforms. Not reducing Subversion
support to UNIX only.

Paging support doesn't require man, nroff, nor UNIX. And even updating
the scripts to use something like ezt for the help texts, while still
creating string literals would improve things a lot.

With more effort we could go a lot further then that. Personally I
don't intend to start looking at nroff for just fixing a few helptexts,
but I would assume copy&paste would work for my needs in about every
format language,

Bert From: James K. Lowden
Sent: 10/08/2013 05:23
To: Daniel Shahaf
Cc: dev@subversion.apache.org
Subject: Re: man pages for Subversion
On Fri, 9 Aug 2013 11:21:23 +0300
Daniel Shahaf  wrote:

> > 1.  Get the sources for 1.9
> > 2.  Examine svn_cl__cmd_table
> > 3.  Prepare a doc build tree that creates two things:
> >  a.  man pages
> >  b.  plain text for --help, one file per subcommand
> >
> > The plain text filenames will have the extension ".in", so a
> > simple Makefile rule can convert them to ".h" for inclusion into the
> > appropriate source module.
>
> You'd have to figure out something for windows as well, since we don't
> use make that on that platform.  (Several of our dependencies do, but
> we just generate Visual Studio projects.)

Hmm.  This is a sticky wicket.

My plan is to keep mdoc sources and generate plain text for
inclusion in the help subsystem.  That's done easily with groff, see
below.  But I notice in INSTALL you support building on Windows not only
in a tarball but also directly in-tree.  That means the Windows
build machine would have to be able to convert the mdoc sources to ASCII
strings.

I'd like to point out, btw, that in 2013 this is the tail wagging the
dog.  Windows command-line users are rare; most people use Tortoise.
Those that do use the command line don't rely on --help very much,
either.  Consider:

$ svn propset --help | wc -l
  95

and that more.exe resembles UNIX more(1) circa 1984 and that most
Windows users don't even know it exists.  "svn help" on Windows is not
the model of ease of use.

Preserving --help for anything more than a short recap of the options
isn't worth it, especially when PDFs are available!  The time is much
better spent improving the documentation and Subversion itself.

But let us suppose ad argumentum that we want to preserve --help, and
to support in-tree builds on Windows.  That means something has to
convert the mdoc source to plain text.  I don't see any better way than
to require mdocml or groff to be installed.

> In general, it'll be best if you hook into the existing build system
> --- configure.ac, gen-make.py, and build.conf.  The latter two are
> used on all platforms (on unix they generate build-outputs.mk).

I don't want to get too deeply involved.  I know mdoc, and I want to
see man pages become part of Subversion.  I'm willing to provide
up-to-date mdoc source for them, and to generate plain text, and to
write a little Perl to convert the plain text to static constant
character arrays in C. I'm prepared to do the work to make it easy
-- for someone who knows how -- to integrate that work into the build
system. I don't want to peek inside gen-make.py.

Below is a sample of what I'd expect from the mdoc -> ascii
conversion.

--jkl

$ groff -P -bc -Tascii -mdoc man1/svn-copy.1 \
| sed -ne '/NAME/,/specify/p'
NAME
 svn copy -- Duplicate something in working copy or repository

SYNOPSIS
 svn copy (cp) [-Fmq] [--config-dir DIR] [--config-option OPT]
  [--editor-cmd CMD] [--encoding ENC] [--file] [--force-log]
  [--ignore-externals] [--message] [--no-auth-cache]
  [--non-interactive] [--parents] [--password PWD] [--quiet]
  [{-r | --revision} REV] [--trust-server-cert] [--username
USR] [--with-revprop PROP] SRC[@REV] ... DST

DESCRIPTION
 When copying multiple sources, they will be added as children of
DST, which must be a directory.  SRC and DST can each be either a
working copy (WC) path or URL:
 WC  -> WC  copy and schedule

Re: man pages for Subversion

[James K. Lowden]

On Fri, 9 Aug 2013 11:21:23 +0300
Daniel Shahaf  wrote:

> > 1.  Get the sources for 1.9 
> > 2.  Examine svn_cl__cmd_table
> > 3.  Prepare a doc build tree that creates two things:
> >  a.  man pages
> >  b.  plain text for --help, one file per subcommand
> > 
> > The plain text filenames will have the extension ".in", so a
> > simple Makefile rule can convert them to ".h" for inclusion into the
> > appropriate source module.  
> 
> You'd have to figure out something for windows as well, since we don't
> use make that on that platform.  (Several of our dependencies do, but
> we just generate Visual Studio projects.)

Hmm.  This is a sticky wicket.  

My plan is to keep mdoc sources and generate plain text for
inclusion in the help subsystem.  That's done easily with groff, see
below.  But I notice in INSTALL you support building on Windows not only
in a tarball but also directly in-tree.  That means the Windows
build machine would have to be able to convert the mdoc sources to ASCII
strings.  

I'd like to point out, btw, that in 2013 this is the tail wagging the
dog.  Windows command-line users are rare; most people use Tortoise.
Those that do use the command line don't rely on --help very much,
either.  Consider:

$ svn propset --help | wc -l
  95

and that more.exe resembles UNIX more(1) circa 1984 and that most
Windows users don't even know it exists.  "svn help" on Windows is not
the model of ease of use.  

Preserving --help for anything more than a short recap of the options
isn't worth it, especially when PDFs are available!  The time is much
better spent improving the documentation and Subversion itself.  

But let us suppose ad argumentum that we want to preserve --help, and
to support in-tree builds on Windows.  That means something has to
convert the mdoc source to plain text.  I don't see any better way than
to require mdocml or groff to be installed.  

> In general, it'll be best if you hook into the existing build system
> --- configure.ac, gen-make.py, and build.conf.  The latter two are
> used on all platforms (on unix they generate build-outputs.mk).

I don't want to get too deeply involved.  I know mdoc, and I want to
see man pages become part of Subversion.  I'm willing to provide
up-to-date mdoc source for them, and to generate plain text, and to
write a little Perl to convert the plain text to static constant
character arrays in C. I'm prepared to do the work to make it easy
-- for someone who knows how -- to integrate that work into the build
system. I don't want to peek inside gen-make.py.  

Below is a sample of what I'd expect from the mdoc -> ascii
conversion.  

--jkl

$ groff -P -bc -Tascii -mdoc man1/svn-copy.1 \
| sed -ne '/NAME/,/specify/p' 
NAME
 svn copy -- Duplicate something in working copy or repository

SYNOPSIS
 svn copy (cp) [-Fmq] [--config-dir DIR] [--config-option OPT]
  [--editor-cmd CMD] [--encoding ENC] [--file] [--force-log]
  [--ignore-externals] [--message] [--no-auth-cache]
  [--non-interactive] [--parents] [--password PWD] [--quiet]
  [{-r | --revision} REV] [--trust-server-cert] [--username
USR] [--with-revprop PROP] SRC[@REV] ... DST

DESCRIPTION
 When copying multiple sources, they will be added as children of
DST, which must be a directory.  SRC and DST can each be either a
working copy (WC) path or URL:
 WC  -> WC  copy and schedule for addition (with history)
 WC  -> URL immediately commit a copy of WC to URL
 URL -> WC  check out URL into WC, schedule for addition
 URL -> URL complete server-side copy;  used to branch and tag
 All the SRCs must be of the same type.

OPTIONS
 --editor-cmd CMD  use CMD as external editor
 --encoding ENCtreat value as being in charset encoding ENC
 -F --file FIL read log message from file FIL
 --force-log   force validity of log message source
 --ignore-externalsignore externals definitions
 -m --message MSG  specify log message MSG

Re: man pages for Subversion

[Stefan Sperling]

On Thu, Aug 08, 2013 at 10:25:08PM -0400, James K. Lowden wrote:
> It sounds like we have agreement and the ball is back in my court.  If
> there are no objections, I'll work on the following as a next step:
> 
> 1.  Get the sources for 1.9 
> 2.  Examine svn_cl__cmd_table
> 3.  Prepare a doc build tree that creates two things:
>  a.  man pages
>  b.  plain text for --help, one file per subcommand
> 
> The plain text filenames will have the extension ".in", so a
> simple Makefile rule can convert them to ".h" for inclusion into the
> appropriate source module.  

Will the man pages and plain text files be derived from one another
in some way? Or will we have to maintain two versions of the docs?

Will the current 'svn help' output be changed in any way?

Re: man pages for Subversion

[Daniel Shahaf]

James K. Lowden wrote on Thu, Aug 08, 2013 at 22:25:08 -0400:
> Stefan Sperling  wrote:
> > If you manage to generate both the compiled-in strings and the man
> > pages (on UNIX only) from the same source format during the build,
> > I"m happy with that. If you manage to generate man pages from
> > 'svn help' output during the build process on UNIX, then I'm happy,
> > too.
> 
> It sounds like we have agreement and the ball is back in my court.  If
> there are no objections, I'll work on the following as a next step:
> 
> 1.  Get the sources for 1.9 
> 2.  Examine svn_cl__cmd_table
> 3.  Prepare a doc build tree that creates two things:
>  a.  man pages
>  b.  plain text for --help, one file per subcommand
> 
> The plain text filenames will have the extension ".in", so a
> simple Makefile rule can convert them to ".h" for inclusion into the
> appropriate source module.  

You'd have to figure out something for windows as well, since we don't
use make that on that platform.  (Several of our dependencies do, but we
just generate Visual Studio projects.)

In general, it'll be best if you hook into the existing build system ---
configure.ac, gen-make.py, and build.conf.  The latter two are used on
all platforms (on unix they generate build-outputs.mk).

It would probably be worthwhile to familiarize yourself with the build
system on all platforms.  Have a look at where translate_sql.py is
invoked from.  (It's invoked at autogen.sh time on unix, by the way,
meaning end-users don't need to have Python to build + install
Subversion from a tarball.)

Re: man pages for Subversion

[James K. Lowden]

On Thu, 8 Aug 2013 08:58:29 +0300
Daniel Shahaf  wrote:

> Perhaps this is a good time to point out that you were writing man
> pages for 1.7, which is in "Security fixes only" mode.  The supported
> release is 1.8 and trunk (where new development should be targeted)
> is 1.9.
> 
> > 1.  Keep the man pages in mdoc format as the "souce code".
> > 2.  Convert "svn foo --help" and "svn help foo" to invoke man(1), as
> > git does.  Or use groff to generate plain text for static inclusion
> > in the the binary.  
> 
> As Bert said: not possible.  We support Windows and that has no man(1)
> or nroff(1) binaries.

OK.  It's easier to convert mdoc to ascii than the other way around.
What you want, I think, is one long C string (with embedded newlines)
that per subcommand that you can print in response to --help.  Can do. 

> So you'd need a solution that allows compiling the man pages source
> to C strings that uses only Python (and maybe Perl).

Yeah, the poverty of Windows has been a problem since ever.  We can
leave the functionality of --help as is.  

> You realise we already have such a database?  svn_cl__cmd_table in
> subversion/svn/svn.c.

I will look into it.  I suppose it sounds odd to this group, but I'm
just a Subversion user.  I've never been tempted to hack on it, and
never looked at the code.  

> > Contrary to popular wisdom, good old troff is the best system there
> > is
> 
> Clearly you have a strong opinion on this.

Yes.  :-)   It's not as though I've been using troff for 30 years and
never learned anything else.  I came to it reluctantly, having tried
several others, and found it actually works pretty well.  As Dijkstra
said of Algol 60, "an advance over its predecessors, and over many of
its successors."  

Stefan Sperling  wrote:
> If you manage to generate both the compiled-in strings and the man
> pages (on UNIX only) from the same source format during the build,
> I"m happy with that. If you manage to generate man pages from
> 'svn help' output during the build process on UNIX, then I'm happy,
> too.

It sounds like we have agreement and the ball is back in my court.  If
there are no objections, I'll work on the following as a next step:

1.  Get the sources for 1.9 
2.  Examine svn_cl__cmd_table
3.  Prepare a doc build tree that creates two things:
 a.  man pages
 b.  plain text for --help, one file per subcommand

The plain text filenames will have the extension ".in", so a
simple Makefile rule can convert them to ".h" for inclusion into the
appropriate source module.  

I'll also subscribe to the list to make it more convenient for us to
communicate.  

Regards, 

--jkl

Re: man pages for Subversion

[Stefan Sperling]

On Thu, Aug 08, 2013 at 09:20:16AM +0200, Alan Barrett wrote:
> On Wed, 07 Aug 2013, James K. Lowden wrote:
> >If HTML is desired, there are a couple of options.  I haven't used it,
> >but troff2page sounds nice.
> >http://www.ccs.neu.edu/home/dorai/troff2page/index.html#node_toc_start
> >GNU troff also supports HTML output, but I haven't had great luck with
> >it.
> 
> For rendering man pages to plain text, HTML, or PDF, please consider
> the BSD-licenced mandoc program (which, confusingly, is part of a
> package called "mdocml").  See . The HTML and
> PDF output from mandoc is superior to the corresponding output from
> groff, but mandoc is more fussy about errors in the input.
> 
> --apb (Alan Barrett)

I concur. mandoc should be used if available, otherwise fall back
to groff.

However, as a first step, having pages generated by groff would
be just fine. Hooking in another tool is easy once that's done.

Re: man pages for Subversion

[Alan Barrett]

On Wed, 07 Aug 2013, James K. Lowden wrote:

If HTML is desired, there are a couple of options.  I haven't used it,
but troff2page sounds nice.
http://www.ccs.neu.edu/home/dorai/troff2page/index.html#node_toc_start
GNU troff also supports HTML output, but I haven't had great luck with
it.


For rendering man pages to plain text, HTML, or PDF, please 
consider the BSD-licenced mandoc program (which, confusingly, is 
part of a package called "mdocml").  See . 
The HTML and PDF output from mandoc is superior to the 
corresponding output from groff, but mandoc is more fussy about errors 
in the input.


--apb (Alan Barrett)

Re: man pages for Subversion

[Stefan Sperling]

On Thu, Aug 08, 2013 at 08:58:29AM +0300, Daniel Shahaf wrote:
> James K. Lowden wrote on Wed, Aug 07, 2013 at 20:13:16 -0400:
> > 1.  Keep the man pages in mdoc format as the "souce code".
> > 2.  Convert "svn foo --help" and "svn help foo" to invoke man(1), as
> > git does.  Or use groff to generate plain text for static inclusion in
> > the the binary.  
> 
> As Bert said: not possible.  We support Windows and that has no man(1)
> or nroff(1) binaries.

Please do not change the behaviour of 'svn help' for this.
It needs compiled-in strings so that it works on any platform.

I think man pages only matter on UNIX-like platforms. They shouldn't
even be generated during builds on other platforms. They should be an
alternative to 'svn help' so that UNIX users have a familiar interface
to this documentation.

If you manage to generate both the compiled-in strings and the man
pages (on UNIX only) from the same source format during the build,
I"m happy with that. If you manage to generate man pages from
'svn help' output during the build process on UNIX, then I'm happy, too.

AW: man pages for Subversion

[Markus Schaber]

Hi,

We could go the other way: There's an nroff writer in python for 
reStructuredText:
http://pydoc.net/Python/epubmaker/0.3.19b1/epubmaker.mydocutils.gutenberg.writers.nroff/

Best regards

Markus Schaber

CODESYS® a trademark of 3S-Smart Software Solutions GmbH

Inspiring Automation Solutions

3S-Smart Software Solutions GmbH
Dipl.-Inf. Markus Schaber | Product Development Core Technology
Memminger Str. 151 | 87439 Kempten | Germany
Tel. +49-831-54031-979 | Fax +49-831-54031-50

E-Mail: m.scha...@codesys.com | Web: http://www.codesys.com | CODESYS store: 
http://store.codesys.com
CODESYS forum: http://forum.codesys.com

Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade 
register: Kempten HRB 6186 | Tax ID No.: DE 167014915

> -Ursprüngliche Nachricht-
> Von: Daniel Shahaf [mailto:danie...@apache.org]
> Gesendet: Donnerstag, 8. August 2013 07:38
> An: Daniel Shahaf
> Cc: Bert Huijben; James K. Lowden; dev@subversion.apache.org
> Betreff: Re: man pages for Subversion
> 
> On Thu, Aug 08, 2013 at 08:35:14AM +0300, Daniel Shahaf wrote:
> > I have no idea what you're talking about.  What is the problem on
> > windows and how does a canonical format fix it?
> 
> Never mind, I see what you mean now (JAmes' suggestion to generate help
> output from nroff source)

Re: man pages for Subversion

[Daniel Shahaf]

James K. Lowden wrote on Wed, Aug 07, 2013 at 20:13:16 -0400:
> On Wed, 7 Aug 2013 12:45:36 +0300
> Daniel Shahaf  wrote:
> 
> > Then individual sections were edited by hand, sometimes to
> > > put finishing touches on the formatting, sometimes to clarify the
> > > text. 
> > 
> > How can we incorporate that into the dev cycle?
> > 
> > if manual tweaks are required, we would have to follow a scheme
> > similar to translations: the master English output
> > (subversion/svn/svn.c:svn_cl__cmd_table) is updated, and afterwards
> > someone comes along, regenerates the manpage, re-applies the manual
> > edits, and commits.  The problem is that a release may happen before
> > someone gets to update the semi-manually-generated man pages, which
> > will cause them to be out-of-date (with respect to 'svn help' output)
> > in the release.
> > 
> > So, in addition to Stefan's point about incorporating your man pages
> > into the build system, I think it'll be useful to eliminate the need
> > for manual steps.
> 
> First of all, let me say thanks for the positive response from
> everyone, and thanks for asking.  :-)  I wasn't sure how my message
> would be received, and I'm gratified I wasn't wasting my time. 
> 

Perhaps this is a good time to point out that you were writing man pages
for 1.7, which is in "Security fixes only" mode.  The supported release
is 1.8 and trunk (where new development should be targeted) is 1.9.

> 1.  Keep the man pages in mdoc format as the "souce code".
> 2.  Convert "svn foo --help" and "svn help foo" to invoke man(1), as
> git does.  Or use groff to generate plain text for static inclusion in
> the the binary.  

As Bert said: not possible.  We support Windows and that has no man(1)
or nroff(1) binaries.

The Windows build system already has Python available.  (You *might* be
able to assume Perl since it's required for OpenSSL, but that requires
discussion.)  So you'd need a solution that allows compiling the man
pages source to C strings that uses only Python (and maybe Perl).

> My manualize script is just a throwaway hack; it was easier (probably)
> to parse the help output to generate the synopsis and option lists.
> Going forward, it's probably not worthwhile to try to maintain a
> database of options keyed by subcommand or whatever.  Although the

You realise we already have such a database?  svn_cl__cmd_table in
subversion/svn/svn.c.

> Contrary to popular wisdom, good old troff is the best system there is

Clearly you have a strong opinion on this.

> So: unless you already have a database of options somewhere, I'd

See above.

> To deal with the copyright issue, I would put "Donated to the
> Subversion project by James K. Lowden" or similar in each file in the
> Authors section.  

ASF generally recommends against @author tag on source code, but having
an acknowledgement on the compiled documentation might be okay I think
--- especially given that it's pretty common for man pages to carry such
acknowledgements.

Re: man pages for Subversion

[Daniel Shahaf]

On Thu, Aug 08, 2013 at 08:35:14AM +0300, Daniel Shahaf wrote:
> I have no idea what you're talking about.  What is the problem on
> windows and how does a canonical format fix it?

Never mind, I see what you mean now (JAmes' suggestion to generate help output 
from nroff source)

Re: man pages for Subversion

[Daniel Shahaf]

Bert Huijben wrote on Wed, Aug 07, 2013 at 19:13:53 -0700:
> Subversion is used on far more platforms than that have a default 'man'
> support. Breaking support for Windows, OS/2 and Netware is not an
> option.
> 

Why are you even worried about that?  No one is going to break support
for Windows.  That's been project policy for longer than either of us
has had commit access.

> So we need a canonical format that is different than one that isn't
> platform dependent or requires more optional dependencies.

I have no idea what you're talking about.  What is the problem on
windows and how does a canonical format fix it?

RE: man pages for Subversion

[Bert Huijben]

Subversion is used on far more platforms than that have a default 'man'
support. Breaking support for Windows, OS/2 and Netware is not an
option.

So we need a canonical format that is different than one that isn't
platform dependent or requires more optional dependencies.

Bert From: James K. Lowden
Sent: 08/08/2013 02:13
To: Daniel Shahaf
Cc: dev@subversion.apache.org
Subject: Re: man pages for Subversion
On Wed, 7 Aug 2013 12:45:36 +0300
Daniel Shahaf  wrote:

> Then individual sections were edited by hand, sometimes to
> > put finishing touches on the formatting, sometimes to clarify the
> > text.
>
> How can we incorporate that into the dev cycle?
>
> if manual tweaks are required, we would have to follow a scheme
> similar to translations: the master English output
> (subversion/svn/svn.c:svn_cl__cmd_table) is updated, and afterwards
> someone comes along, regenerates the manpage, re-applies the manual
> edits, and commits.  The problem is that a release may happen before
> someone gets to update the semi-manually-generated man pages, which
> will cause them to be out-of-date (with respect to 'svn help' output)
> in the release.
>
> So, in addition to Stefan's point about incorporating your man pages
> into the build system, I think it'll be useful to eliminate the need
> for manual steps.

First of all, let me say thanks for the positive response from
everyone, and thanks for asking.  :-)  I wasn't sure how my message
would be received, and I'm gratified I wasn't wasting my time.

It seems we agree having man pages would be a good thing; the next
question is how to keep them up to date.  If we can agree on how to do
that, I'm willing to make changes that make the documentation more
amenable to the build system.

I suggest the following:

1.  Keep the man pages in mdoc format as the "souce code".
2.  Convert "svn foo --help" and "svn help foo" to invoke man(1), as
git does.  Or use groff to generate plain text for static inclusion in
the the binary.
3.  Add a cross-reference section to each page.
4.  Consider keeping the manual as PDFs on the website.  If you do, add
PDF escapes to the mdoc text to generate hyperlinks.  (That's a
feature of groff nowadays.)
5.  Consider a table of contents and an index.

If HTML is desired, there are a couple of options.  I haven't used it,
but troff2page sounds nice.
http://www.ccs.neu.edu/home/dorai/troff2page/index.html#node_toc_start
GNU troff also supports HTML output, but I haven't had great luck with
it.

My manualize script is just a throwaway hack; it was easier (probably)
to parse the help output to generate the synopsis and option lists.
Going forward, it's probably not worthwhile to try to maintain a
database of options keyed by subcommand or whatever.  Although the
synopsis and command-specific options are slightly redundant, keeping
them up to date and consistent won't be that much work.  Subversion
doesn't change all that much at that level.

Contrary to popular wisdom, good old troff is the best system there is
for maintaining documentation.  It's the least typing and the least
processing: the macros are just a few characters, and "processing"
mostly consists of copying the files to somewhere man(1) will find
them.

I say that having tried others: I used Doxygen and DocBook, and even
text2man.  Troff is ugly, sure; Brian Kernighan called the syntax
"rebarbative".  At the end of the day, though, you have to type
*something* to indicate "this is a list" and "this is an optional
function that takes an argument".  mdoc is made for that, and it's
about 10% the size of e.g. DocBook, no matter how measured.

I'm completely sympathetic to the isssue of keeping the documentation
up to date.  I've maintained the FreeTDS user guide for 10 years.
There's really nothing for it except to update as you go along, and to
review it (over and over) prior to release.

I confess I don't know how the Subversion source code is organized. IMO
the best place to keep documentation is in the source file directory.
I learned that trick from Dennis Ritchie.  ;-)

Some of the text is shared between pages.  To generate my drafts, I
just let ./manualize crank away.  If mdoc is the canonical source, then
common text goes in a shared file.  (There's really no need to assemble
the fragments; troff supports a ".so" directive that works like
#include in C.)

So: unless you already have a database of options somewhere, I'd
suggest maintaining the docs in mdoc format.  If you like that idea,
I'm willing to make it easy.  I would:

1.  regenerate them using the HEAD version (or take the output from
someone who's currently hacking on it)
2.  fix up the text some more
3.  remove the shared text to shared files
4.  

Re: man pages for Subversion

[James K. Lowden]

On Wed, 7 Aug 2013 12:45:36 +0300
Daniel Shahaf  wrote:

> Then individual sections were edited by hand, sometimes to
> > put finishing touches on the formatting, sometimes to clarify the
> > text. 
> 
> How can we incorporate that into the dev cycle?
> 
> if manual tweaks are required, we would have to follow a scheme
> similar to translations: the master English output
> (subversion/svn/svn.c:svn_cl__cmd_table) is updated, and afterwards
> someone comes along, regenerates the manpage, re-applies the manual
> edits, and commits.  The problem is that a release may happen before
> someone gets to update the semi-manually-generated man pages, which
> will cause them to be out-of-date (with respect to 'svn help' output)
> in the release.
> 
> So, in addition to Stefan's point about incorporating your man pages
> into the build system, I think it'll be useful to eliminate the need
> for manual steps.

First of all, let me say thanks for the positive response from
everyone, and thanks for asking.  :-)  I wasn't sure how my message
would be received, and I'm gratified I wasn't wasting my time. 

It seems we agree having man pages would be a good thing; the next
question is how to keep them up to date.  If we can agree on how to do
that, I'm willing to make changes that make the documentation more
amenable to the build system.  

I suggest the following:

1.  Keep the man pages in mdoc format as the "souce code".
2.  Convert "svn foo --help" and "svn help foo" to invoke man(1), as
git does.  Or use groff to generate plain text for static inclusion in
the the binary.  
3.  Add a cross-reference section to each page. 
4.  Consider keeping the manual as PDFs on the website.  If you do, add
PDF escapes to the mdoc text to generate hyperlinks.  (That's a
feature of groff nowadays.) 
5.  Consider a table of contents and an index. 

If HTML is desired, there are a couple of options.  I haven't used it,
but troff2page sounds nice.  
http://www.ccs.neu.edu/home/dorai/troff2page/index.html#node_toc_start
GNU troff also supports HTML output, but I haven't had great luck with
it.  

My manualize script is just a throwaway hack; it was easier (probably)
to parse the help output to generate the synopsis and option lists.
Going forward, it's probably not worthwhile to try to maintain a
database of options keyed by subcommand or whatever.  Although the
synopsis and command-specific options are slightly redundant, keeping
them up to date and consistent won't be that much work.  Subversion
doesn't change all that much at that level.  

Contrary to popular wisdom, good old troff is the best system there is
for maintaining documentation.  It's the least typing and the least
processing: the macros are just a few characters, and "processing"
mostly consists of copying the files to somewhere man(1) will find
them.  

I say that having tried others: I used Doxygen and DocBook, and even
text2man.  Troff is ugly, sure; Brian Kernighan called the syntax
"rebarbative".  At the end of the day, though, you have to type
*something* to indicate "this is a list" and "this is an optional
function that takes an argument".  mdoc is made for that, and it's
about 10% the size of e.g. DocBook, no matter how measured.  

I'm completely sympathetic to the isssue of keeping the documentation
up to date.  I've maintained the FreeTDS user guide for 10 years.
There's really nothing for it except to update as you go along, and to
review it (over and over) prior to release.  

I confess I don't know how the Subversion source code is organized. IMO
the best place to keep documentation is in the source file directory.
I learned that trick from Dennis Ritchie.  ;-)

Some of the text is shared between pages.  To generate my drafts, I
just let ./manualize crank away.  If mdoc is the canonical source, then
common text goes in a shared file.  (There's really no need to assemble
the fragments; troff supports a ".so" directive that works like
#include in C.)  

So: unless you already have a database of options somewhere, I'd
suggest maintaining the docs in mdoc format.  If you like that idea,
I'm willing to make it easy.  I would:

1.  regenerate them using the HEAD version (or take the output from
someone who's currently hacking on it)
2.  fix up the text some more 
3.  remove the shared text to shared files
4.  add SEE ALSO cross references
5.  deliver mdoc files and a GNU Makefile that with targets "install"
and "pdf"

More than that I won't promise, but we can discuss it.  :-)

To deal with the copyright issue, I would put "Donated to the
Subversion project by James K. Lowden" or similar in each file in the
Authors section.  

Happy to be of service.  Let me know your thoughts.  

Regards, 

--jkl

Re: man pages for Subversion

[Daniel Shahaf]

James K. Lowden wrote on Wed, Aug 07, 2013 at 00:07:08 -0400:
> I offer the above mdoc man pages to the Subversion project as a token
> of my appreciation, and in the hope they'll be included in future
> versions.  
> 

Thanks!

> The pages represent perhaps 30 hours of work.  The --help text was
> processed through the bin/manualize Perl script to create a rough
> mdoc draft.  Then individual sections were edited by hand, sometimes to
> put finishing touches on the formatting, sometimes to clarify the text. 

How can we incorporate that into the dev cycle?

We don't add new command-line options between patch releases, but we may
change the help output in other, bidirectionally-compatible ways.  If we
can just run 'make' to generate updated manpages, then the manpages will
always be in sync with the HEAD code; but if manual tweaks are required,
we would have to follow a scheme similar to translations: the master
English output (subversion/svn/svn.c:svn_cl__cmd_table) is updated, and
afterwards someone comes along, regenerates the manpage, re-applies the
manual edits, and commits.  The problem is that a release may happen
before someone gets to update the semi-manually-generated man pages,
which will cause them to be out-of-date (with respect to 'svn help'
output) in the release.

So, in addition to Stefan's point about incorporating your man pages
into the build system, I think it'll be useful to eliminate the need for
manual steps.

Re: man pages for Subversion

[Daniel Shahaf]

James K. Lowden wrote on Wed, Aug 07, 2013 at 00:07:08 -0400:
> The final result is surely a derived work, but it's not clear to me if
> the --help text is copyrighted.  I am prepare to assign it as befits
> the project, if need be.  

The ASF never requires copyright assignment.  However, license grants
are welcome, in particular for larger contributions; see
https://www.apache.org/licenses/#clas

Re: man pages for Subversion

[Daniel Shahaf]

Stefan Sperling wrote on Wed, Aug 07, 2013 at 11:27:13 +0200:
> On Wed, Aug 07, 2013 at 12:07:08AM -0400, James K. Lowden wrote:
> > The pages represent perhaps 30 hours of work.  The --help text was
> > processed through the bin/manualize Perl script to create a rough
> > mdoc draft.  Then individual sections were edited by hand, sometimes to
> > put finishing touches on the formatting, sometimes to clarify the text. 
> 
> So essentially you are running a script to produce those pages
> from 'svn help' output? Where can I find this script?
> 

bin/manualize in http://www.schemamania.org/svn/svn-manual.pax.gz

Re: man pages for Subversion

[Stefan Sperling]

On Wed, Aug 07, 2013 at 12:07:08AM -0400, James K. Lowden wrote:
> http://www.schemamania.org/svn/man1/
> e.g. http://www.schemamania.org/svn/man1/svn-propset.pdf
> 
> I offer the above mdoc man pages to the Subversion project as a token
> of my appreciation, and in the hope they'll be included in future
> versions.  
> 
> I find using any command-line tool more convenient when it's documented
> with proper man pages, so that "man foo" instantly displays the
> reference manual.  That's especially true of a complex tool like
> subversion.  The "svn help foo | less" feature is OK, but it's not as
> convenient and lacks the boldface and underscores that assist in
> scanning the text.  HTML is better-looking, but many clicks away.  
> 
> I hope you agree that the man pages are easier to read than plain
> text, and the PDFs just stunning.  

Thanks. I've been thinking about how to better produce mdoc output
for Subversion man pages for a while now.

A related discussion can be found at
http://mail-archives.apache.org/mod_mbox/subversion-dev/201209.mbox/%3C1347648306.66873.YahooMailNeo%40web87705.mail.ir2.yahoo.com%3E

Please also see the Improve 'svn help' item on this page:
http://subversion.apache.org/ideas.html#ideas

> The pages represent perhaps 30 hours of work.  The --help text was
> processed through the bin/manualize Perl script to create a rough
> mdoc draft.  Then individual sections were edited by hand, sometimes to
> put finishing touches on the formatting, sometimes to clarify the text. 

So essentially you are running a script to produce those pages
from 'svn help' output? Where can I find this script?

Can the finishing touches you mention be made on the source
material, i.e. 'svn help' as well? Ideally, we'd produce man pages
as part of our build process and ship them with Subversion releases.
Can you help us with achieving that goal?

I don't mind whether we're post-processing 'svn help' output to
produce man pages, or whether we're generating 'svn help' output
and man pages from the same source material somehow. As long as
we can get a set of quality man pages to ship with a release.

> The final result is surely a derived work, but it's not clear to me if
> the --help text is copyrighted.  I am prepare to assign it as befits
> the project, if need be.  

I would say that the ASF copyright applies, i.e. the licence is APLv2.

> You can generate your own man pages from whatever version of Subversion
> you have installed by unpacking the tarball and running "make".  

Which tarball?

man pages for Subversion

[James K. Lowden]

http://www.schemamania.org/svn/man1/
e.g. http://www.schemamania.org/svn/man1/svn-propset.pdf

I offer the above mdoc man pages to the Subversion project as a token
of my appreciation, and in the hope they'll be included in future
versions.  

I find using any command-line tool more convenient when it's documented
with proper man pages, so that "man foo" instantly displays the
reference manual.  That's especially true of a complex tool like
subversion.  The "svn help foo | less" feature is OK, but it's not as
convenient and lacks the boldface and underscores that assist in
scanning the text.  HTML is better-looking, but many clicks away.  

I hope you agree that the man pages are easier to read than plain
text, and the PDFs just stunning.  

The pages represent perhaps 30 hours of work.  The --help text was
processed through the bin/manualize Perl script to create a rough
mdoc draft.  Then individual sections were edited by hand, sometimes to
put finishing touches on the formatting, sometimes to clarify the text. 

The final result is surely a derived work, but it's not clear to me if
the --help text is copyrighted.  I am prepare to assign it as befits
the project, if need be.  

You can generate your own man pages from whatever version of Subversion
you have installed by unpacking the tarball and running "make".  

Regards, 

--jkl