automake po / pot file integration: when to merge the PO files?

[Bruno Haible]

Hi,

One issue still needs discussion within the planned po / pot file
integration [1]:
When should the PO files that are distributed be merged with the POT file?

The problem
---

PO files (translations) are produced by translators and integrated to the
project either by a maintainer (who receives them by mail from the translators
directly or through the TP robot) or by a translator herself (who commits
it into the version control repository).

When a new release is made, or shortly before a new release is made, the
maintainer circulates a tarball, and the translators are supposed to pick
the PO files from this tarball and improve them by translating new
untranslated messages.

A PO file for a translator is produced by running 'msgmerge', basically
  $ msgmerge last-translation.po new-messages-list.pot > new-translation.po

If the PO files are being put in a VCS, then each time an 'msgmerge' is done,
the PO file changes (new line numbers, new messages, dropped messages, etc.).
Maintainers don't like this because
  - If they commit the modified PO files regularly, they bloat the history
of their VCS,
  - If they don't commit them regularly, the risk of conflicts increases.
Either way, it causes regular hassles.

If the PO files are not being put in a VCS, then
  1. the VCS contents is not the complete source,
  2. the workflow where translators commit their translations directly is
 impossible.

The classical approach
--

In the approach designed in 1995, there is one PO file per language.

Logically, the POT file depends on all source files, each PO file depends on
the POT file, and each MO file depends on its corresponding PO file.

So, it would be "right" to implement Makefile dependencies in such a way that
each time a source file changes and the maintainer does a "make", the POT file
is being updated (via an 'xgettext' invocation), then the PO files are being
updated (via N 'msgmerge' invocations), then the MO files are being updated
(via N 'msgfmt' invocations). But this is too often:
  - It takes too much time to rebuild _all_ these files after every little
change.
  - The maintainer most often does not care about whether the translations are
up-to-date, because even if he runs "make install", he is not going to
start translation work.

So, the approach implemented in po/Makefile.in.in is that "make" does not
update all PO files, only "make dist" (which produces a tarball) does.
There is also a "make update-po" target which updates all PO files but does
not create a tarball. If there is a VCS, the maintainer is supposed to commit
the updated PO files when he makes and releases the tarball.

This was fine for cathedral style development, and until Automake came along.
In bazaar style development, there are more frequent releases, and committing
the updated PO files started to bloat the VCS history. Worse, Automake's
"make distcheck" becoming more popular, maintainers started to create tarballs
that were not really meant for use by translators. But the PO files were
being updated and increased the potential of VCS conflicts.

The minimalistic approach
-

It would be possible to never update the PO files, and instead produce the .mo
files by running 'msgmerge' on the fly, directly before 'msgfmt':
  $ msgmerge xx.po domain.pot | msgfmt -c - > xx.mo
So:
  - The POT file would be updated at "make dist",
  - The PO files would only be changed when the translator submits a new one,
  - The MO files would be updated at "make dist".
The VCS would only contain the PO files; and there would be no VCS conflicts.

The drawback with this approach is that translators cannot work with a PO
file that they take from a tarball; they would need to run 'msgmerge' by
themselves (if there is no TP robot that does it for them). This would be
a major hassle for the translators. Or they would need to rely on a web
service to deliver them the merged PO files - then the translators have a
methodology problem.

The inconsistent approach
-

This is a variation of the minimalistic approach: In the development tree,
never update the PO files. But implement the "make dist" target in such a
way that it puts updated PO files into the tarball.

Translators would be satisfied with this approach.

The drawback is that once a maintainer unpacks a tarball right after producing
it, its contents is different from what he has in his development tree. This
is not only surprising, it can also lead to bugs that appear only with the
release tarball and not earlier.

A radically different approach
--

It would be possible to store two PO files per language in a development tree:
  - xx.po, the last translation received from the translator,
  - xx.merged.po, the updated PO file, in sync with the latest POT file.
The VCS would only contain the xx.po files, not the xx.merged.po files. But
both sets of PO files would be present 

Re: automake po / pot file integration: when to merge the PO files?

[Roger Leigh]

On Mon, Sep 06, 2010 at 11:25:44AM +0200, Bruno Haible wrote:
> Hi,
> 
> One issue still needs discussion within the planned po / pot file
> integration [1]:
> When should the PO files that are distributed be merged with the POT file?

Just a few comments from a long-time gettext+automake user which
I hope might be useful:

The number one problem for me (as you identified) is the huge churn
in po file content as you make source changes.  I'd like to suggest
that the best way to tackle the problem is to simply stop generating
the source file/line number comments by default; I'm already doing this
in some of my projects by adding "--no-location" to XGETTEXT_OPTIONS
in po/Makevars.  It's a massive improvement.

Making this small change has a huge impact.  po file changes are now
sensible: they match source string changes only, not massive line
renumbering because I added/removed some unrelated code.  This makes
merging between branches sensible because I don't have an entire po
file full of line number conflicts I can't hope to merge manually.

A question I have is "what purpose does having the line number and
source file serve"?  Do those benefits outweigh the massive
disadvantages?  And if the original source file(s) for a string
need to be found, grep(1) is pretty fast.  At least for me, the
translators get mailed the po file, and never look at the source,
so it's not of *any practical benefit* to anyone AFAICS; I've
certainly had no complaints since I turned them off.


With this change made, I would be fully in favour of having
"update-po" run by default so that the po files are always kept
up-to-date.  In this situation, it makes sense--the po file changes
are *entirely related* to the source changes, and can be committed
together.

Updating the po files by default also makes releases easier:
if I tag a release and then "make dist" and find all the po
files were updated, modifying the repository, I need to
detag, commit the changes and retag.  Updating by default means
the repository is always in a "releasable state" whereby any
revision can be tagged without doing additional sanity checks.


Regards,
Roger 

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?   http://gutenprint.sourceforge.net/
   `-GPG Public Key: 0x25BFB848   Please GPG sign your mail.


signature.asc
Description: Digital signature

Re: [Translation-i18n] automake po / pot file integration: when to merge the PO files?

[Chusslove Illich]

> [: Bruno Haible :]
> The minimalistic approach
> -
> [...]
> The drawback with this approach is that translators cannot work with a PO
> file that they take from a tarball; [...]

I'm likely missing something, but...

Why not have a per-language PO update target, e.g.

  $ make update-po-LANG

This would require msgmerge on translator's system, but Gettext tools are
anyway the bare minimum that translators should have in a PO-based
translation workflow (for msgfmt -c if for nothing else).

This is what is conceptually done in Gnome Translation Project, only using a
specialized tool (Intltool) instead of going directly through the build
system's interface, see

http://live.gnome.org/TranslationProject/SvnHowTo#Committing_interface_translation

(Repository update-commit actions in this instruction can be substituted
with unpack tarball, ..., send PO by email. But this brings up another
advantage, and that is that translators can work both with a tarball and
with a VCS in the same way.)

> [...] Or they would need to rely on a web service to deliver them the
> merged PO files - then the translators have a methodology problem.

This has nothing to do with the issue at hand, but I'm curious, what exactly
do you mean by a "methodology problem"?

-- 
Chusslove Illich (Часлав Илић)


signature.asc
Description: This is a digitally signed message part.

Re: [Translation-i18n] automake po / pot file integration: when to merge the PO files?

[Claude Paroz]

Le lundi 06 septembre 2010 à 11:25 +0200, Bruno Haible a écrit :
> Hi,
> 
> One issue still needs discussion within the planned po / pot file
> integration [1]:
> When should the PO files that are distributed be merged with the POT file?
> 
(...)

> The minimalistic approach
> -
> 
> It would be possible to never update the PO files, and instead produce the .mo
> files by running 'msgmerge' on the fly, directly before 'msgfmt':
>   $ msgmerge xx.po domain.pot | msgfmt -c - > xx.mo
> So:
>   - The POT file would be updated at "make dist",
>   - The PO files would only be changed when the translator submits a new one,
>   - The MO files would be updated at "make dist".
> The VCS would only contain the PO files; and there would be no VCS conflicts.
> 
> The drawback with this approach is that translators cannot work with a PO
> file that they take from a tarball; they would need to run 'msgmerge' by
> themselves (if there is no TP robot that does it for them). This would be
> a major hassle for the translators. Or they would need to rely on a web
> service to deliver them the merged PO files - then the translators have a
> methodology problem.

IMHO, there is a growing use of translation platforms where the merge
step is taking care of by an ad-hoc infrastructure: Transifex, TP,
Pootle, GNOME/KDE translation projects, Launchpad, etc.

For individual projects, there is almost always some README or Wiki page
giving explanations about the translation process. The "make update-po"
target may be the standardized way to promote.

So for me, the minimalistic approach is the way to go.

Claude Paroz
-- 
www.2xlibre.net

[PATCH] nds32: add nds32 support into config_sub

[Macpaul Lin]

NDS32 is a new 32-bit RISC architecture invented by Andestech.com.

It has a 16-bit/32-bit mixed-length instruction set to achieve optimal
system performance, code density, and power efficiency.

This is the commit for supporting automake and related config.sub.

Signed-off-by: Macpaul Lin 
---
 lib/config.sub |2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/lib/config.sub b/lib/config.sub
index 204218c..6162dcd 100755
--- a/lib/config.sub
+++ b/lib/config.sub
@@ -283,6 +283,7 @@ case $basic_machine in
| moxie \
| mt \
| msp430 \
+   | nds32 | nds32le | nds32be \
| nios | nios2 \
| ns16k | ns32k \
| or32 \
@@ -378,6 +379,7 @@ case $basic_machine in
| mmix-* \
| mt-* \
| msp430-* \
+   | nds32-* | nds32le-* | nds32be-* \
| nios-* | nios2-* \
| none-* | np1-* | ns16k-* | ns32k-* \
| orion-* \
-- 
1.7.2.2

Re: Nesting m4 macros

[Stefano Lattarini]

[adding autoc...@gnu.org and autom...@gnu.org]

On Monday 06 September 2010, Ralf Wildenhues wrote:
> > > > Have you read any tutorial about the autotools?  If not, I
> > > > suggest this: ,
> > > > which I find it very clear, and which helped me a lot in the
> > > > past.
> > > 
> > > This is a very good suggestion indeed.
> > 
> > Maybe we should add a link to it in the autotools manual, to
> > avoid the poor newbies the pain of trying to learn autoconf and
> > automake from scratch from their reference manual.
> 
> We do so already, in the "Autotools Introduction" node.
Yes, in the automake manual, but not in the autoconf manual, nor in 
the autoconf/automake home pages.  Adding the link in those places
too might make a big difference IMHO, especially for newcomers.

Regards,
   Stefano

[RFC] Make build and install dirs used by distcheck configurable. (was: Re: absolute build directory with spaces)

[Stefano Lattarini]

On Sunday 05 September 2010, Ralf Wildenhues wrote:
> Hello,
> 
> * Jim Meyering wrote on Sat, Sep 04, 2010 at 07:28:58PM CEST:
> > Stefano Lattarini wrote:
> > > What about instead making the names of the temporaries
> > > source/build/install directories used by "make distcheck"
> > > configurable?
It turns out that making the srcdir configurable is not very easy or
natural; also, I don't see any real use case for that.  So I didn't
add this feature in the patch series.  If anyone would find this
feature useful nonetheless, please speak up.
> > > This will offer more flexibility, and won't
> > > introduce still another automake option which would make
> > > backward-compatibility more problematic.
> > > 
> > > I was thinking of something on these lines:
> > >   $ cat Makefile.am
> > >   ...
> > >   AM_DISTCHECK_BUILDDIR_NAME = _ b u i l d ## will be relative to 
> > > $(distdir)
> > >   AM_DISTCHECK_SRCDIR_NAME = . ## likewise
> > >   AM_DISTCHECK_INSTALLDIR_NAME = i...@l1   ## likewise
In the end, I went for the slighty shorter names `AM_DISTCHECK_BUILDDIR'
and `AM_DISTCHECK_INSTALLDIR'.
> > > 
> > > If you like the proposal, I might try to implement this (but not
> > > right away).
> > 
> > I do like it.  Thanks!
> 
> Me too, but the onus for quoting should probably be with the person
> defining the variables, because there is no well-defined way to do
> otherwise.  (And of course you cannot have comments in variables.)
In my patch, I let Automake take the onus for quoting, the only additional
limit being that AM_DISTCHECK_BUILDDIR and AM_DISTCHECK_INSTALLDIR cannot
contain the single-quote character.  We might still change this of course,
as the attached patch is mostly tentative.
> I like it also because some packages will never want to care about
> being buildable under more stressful circumstances.

IMPORTANT NOTE: I still haven't run the whole testsuite against this
patch.  I will if they are no serious objection or proposals of
changes/extensions.

Regards,
  Stefano

-*-*-*-

Make build and install dirs used by distcheck configurable.

* automake.in (generate_makefile): Define makefile variables
AM_DISTCHECK_BUILDDIR and AM_DISTCHECK_INSTALLDIR when needed;
they default respectively to `_build' and `_inst'.
* lib/am/distdir.am (distcheck): Sanitize and honour variables
$(AM_DISTCHECK_BUILDDIR) and $(AM_DISTCHECK_INSTALLDIR).
* tests/distcheck0.test: New test script.
* tests/distcheck1a.test: Likewise.
* tests/distcheck1b.test: Likewise.
* tests/distcheck2a.test: Likewise.
* tests/distcheck2b.test: Likewise.
* tests/distcheck3.test: Likewise.
* tests/distcheck4.test: Likewise.
* tests/distcheck5.test: Likewise.
* tests/distcheck6.test: Likewise.
* tests/distcheck7.test: Likewise.
* tests/distcheck8.test: Likewise.
* tests/distcheck9.test: Likewise.
* tests/Makefile.am (TESTS): Updated.
---
 ChangeLog  |   22 +++
 Makefile.in|   42 +++--
 automake.in|9 +++
 lib/am/distdir.am  |   50 +---
 tests/Makefile.am  |   12 
 tests/Makefile.in  |   12 
 tests/distcheck0.test  |   45 ++
 tests/distcheck1a.test |   56 +
 tests/distcheck1b.test |   61 +++
 tests/distcheck2a.test |   69 +
 tests/distcheck2b.test |   74 +++
 tests/distcheck3.test  |   79 
 tests/distcheck4.test  |   71 ++
 tests/distcheck5.test  |   77 
 tests/distcheck6.test  |   88 +++
 tests/distcheck7.test  |   86 ++
 tests/distcheck8.test  |  156 
 tests/distcheck9.test  |   50 +++
 18 files changed, 1044 insertions(+), 15 deletions(-)
 create mode 100755 tests/distcheck0.test
 create mode 100755 tests/distcheck1a.test
 create mode 100755 tests/distcheck1b.test
 create mode 100755 tests/distcheck2a.test
 create mode 100755 tests/distcheck2b.test
 create mode 100755 tests/distcheck3.test
 create mode 100755 tests/distcheck4.test
 create mode 100755 tests/distcheck5.test
 create mode 100755 tests/distcheck6.test
 create mode 100755 tests/distcheck7.test
 create mode 100755 tests/distcheck8.test
 create mode 100755 tests/distcheck9.test
From bd9c7804d24297ef9e3c9e7492a1247a71f9279a Mon Sep 17 00:00:00 2001
From: Stefano Lattarini 
Date: Sun, 5 Sep 2010 12:00:05 +0200
Subject: [PATCH] Make build and install dirs used by distcheck configurable.

* automake.in (generate_makefile): Define makefile variables
AM_DISTCHECK_BUILDDIR and AM_DISTCHECK_INSTALLDIR when needed;
they default respectively to `_build' and `_inst'.
* lib/am/distdir.am (distcheck): Sanitize and honour variables
$(AM_DISTCHECK_BUILDDIR) and $(AM_DISTCHECK_INSTALLDIR).
* tests/distcheck0.test: New test script.
* tests/distcheck1a.test: Likewise.
* tests/distcheck1b.test: Likewise.
* tests/distcheck2a.test: Likewise.
* tests/distch

Re: [PATCH] nds32: add nds32 support into config_sub

[Eric Blake]

On 09/06/2010 06:03 AM, Macpaul Lin wrote:

NDS32 is a new 32-bit RISC architecture invented by Andestech.com.

This is the commit for supporting automake and related config.sub.


Thanks for the patch.  However, as documented in config.sub itself, you 
need to send this upstream to config-patc...@gnu.org; automake will then 
automatically pick this up once it is accepted upstream.  And make sure 
you also check whether config.guess needs a matching patch.


--
Eric Blake   ebl...@redhat.com+1-801-349-2682
Libvirt virtualization library http://libvirt.org