Re: [Libreoffice] [PATCH] speed up localized builds by introducing po2lo
Hi Miklos, 2011/9/10 Miklos Vajna vmik...@frugalware.org: I'm attaching newer versions of both patches: - the translations one is fixed wrt python (see 82f6c0502e51afbc25e5bf0fcee7914a1a5b3f28, the patch had the same problem) - the core one should be fixed wrt the above two issues Now it works perfectly. Your script does its job well. Unfortunately I still have not found the bug in fast_merge.pl which cause misterious omission even together with your script. I would like to push everything when the whole toolchain works reliably. Cheers, Andras ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] [PATCH] speed up localized builds by introducing po2lo
On Wed, Sep 14, 2011 at 08:15:16PM +0200, Andras Timar tima...@gmail.com wrote: Hi Miklos, 2011/9/10 Miklos Vajna vmik...@frugalware.org: I'm attaching newer versions of both patches: - the translations one is fixed wrt python (see 82f6c0502e51afbc25e5bf0fcee7914a1a5b3f28, the patch had the same problem) - the core one should be fixed wrt the above two issues Now it works perfectly. Your script does its job well. Unfortunately I still have not found the bug in fast_merge.pl which cause misterious omission even together with your script. I would like to push everything when the whole toolchain works reliably. Actually that's not an omission: The accessibility strings end up at translations/unxlngx6.pro/misc/sdf/avmedia/source/framework/localize.sdf here. pgpDmrzuD8mQl.pgp Description: PGP signature ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] [PATCH] speed up localized builds by introducing po2lo
On Thu, Sep 15, 2011 at 12:44:01AM +0200, Miklos Vajna vmik...@frugalware.org wrote: Actually that's not an omission: The accessibility strings end up at translations/unxlngx6.pro/misc/sdf/avmedia/source/framework/localize.sdf here. The attached patch fixes the issue here. From 3381db95cc026f404094daabbed4a8d84685a142 Mon Sep 17 00:00:00 2001 From: Miklos Vajna vmik...@frugalware.org Date: Thu, 15 Sep 2011 01:13:05 +0200 Subject: [PATCH] fast_merge: fix mis-merge of first module's strings The problem was that in write_lines() when $first_run was true, the add_to_buffer() call already set the current module's name to the second module. A fix for this is to use let make_paths() take a parameter, and in case of the first run, pass the really first module name, not the current one. --- l10ntools/scripts/fast_merge.pl | 10 ++ 1 files changed, 6 insertions(+), 4 deletions(-) diff --git a/l10ntools/scripts/fast_merge.pl b/l10ntools/scripts/fast_merge.pl index 5dc63cf..73b824e 100644 --- a/l10ntools/scripts/fast_merge.pl +++ b/l10ntools/scripts/fast_merge.pl @@ -91,7 +91,7 @@ while( hasLines() ) } if( $#current+1 ne 0 ) { -( $path , $localize_file ) = make_paths(); +( $path , $localize_file ) = make_paths($current[ 0 ]-module); add_to_buffer(); write_buffer( $path , $localize_file ); } @@ -240,7 +240,8 @@ sub hasLines sub make_paths { -my $localizeFile = $merge_dir.\\.$current[ 0 ]-module.\\.$current[ 0 ]-file; +my $module = shift ; +my $localizeFile = $merge_dir.\\.$module.\\.$current[ 0 ]-file; my $path = getDir( $localizeFile ); $path =~ s/\\/\//g; @@ -251,8 +252,9 @@ sub make_paths sub write_lines { if( $first_run ){ +my $module = $current[ 0 ]-module; add_to_buffer(); -my( $path , $localize_file ) = make_paths(); +my( $path , $localize_file ) = make_paths($module); $last_path = $path; $last_localize_file = $localize_file; mkpath $path; @@ -262,7 +264,7 @@ sub write_lines else { return , if ( $#current+1 eq 0 ); -my( $path , $localize_file ) = make_paths(); +my( $path , $localize_file ) = make_paths($current[ 0 ]-module); if( $path eq $last_path ) { add_to_buffer(); -- 1.7.6 pgpUyiOQ7sznm.pgp Description: PGP signature ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] [PATCH] speed up localized builds by introducing po2lo
Hi Miklos, 2011/9/8 Miklos Vajna vmik...@frugalware.org: I'm attaching an updated patch that fixes po2lo on Windows, where it did not translate anything due to path translating (/ vs \) issues. First of all, many thanks for your work. Full build time in translations module with all languages took 82 minutes, with your patch it took only 19 minutes on a Windows box. There are 2 minor issues. 1. When the English string contains \n and translation does not contain \n, the script outputs the English string instead of the translation. See for example this line: basctl source\basicide\basidesh.src0 string RID_STR_SOURCETOBIG 15504 sd The source text is too large and can be neither compiled nor saved.\nDelete some of the comments or transfer some methods into another module. 2. Your script produces sdf lines even when the corresponding po file does not exist at all. For example many languages do not have help translations at all, but your script copies English heIp lines to localized sdf files in this case. Best regards, Andras ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] [PATCH] speed up localized builds by introducing po2lo
On Fri, Sep 09, 2011 at 10:40:54AM +0200, Andras Timar tima...@gmail.com wrote: There are 2 minor issues. 1. When the English string contains \n and translation does not contain \n, the script outputs the English string instead of the translation. See for example this line: basctlsource\basicide\basidesh.src0 string RID_STR_SOURCETOBIG 15504 sd The source text is too large and can be neither compiled nor saved.\nDelete some of the comments or transfer some methods into another module. 2. Your script produces sdf lines even when the corresponding po file does not exist at all. For example many languages do not have help translations at all, but your script copies English heIp lines to localized sdf files in this case. I'm attaching newer versions of both patches: - the translations one is fixed wrt python (see 82f6c0502e51afbc25e5bf0fcee7914a1a5b3f28, the patch had the same problem) - the core one should be fixed wrt the above two issues [ The newline issue was a bug in the po parser, handling multiline msgid/msgstr entires, the second problem is simply fixed by checking if the relevant po file exists or not. ] This time I tested it with not only hu, but af and sd locales as well. Miklos From 60173c8ed5180067773b7df503300f16752235c2 Mon Sep 17 00:00:00 2001 From: Miklos Vajna vmik...@frugalware.org Date: Wed, 7 Sep 2011 23:39:15 +0200 Subject: [PATCH] Add po2lo tool --- solenv/bin/po2lo | 205 ++ 1 files changed, 205 insertions(+), 0 deletions(-) create mode 100755 solenv/bin/po2lo diff --git a/solenv/bin/po2lo b/solenv/bin/po2lo new file mode 100755 index 000..0f81ebc --- /dev/null +++ b/solenv/bin/po2lo @@ -0,0 +1,205 @@ +#!/usr/bin/env python +# Version: MPL 1.1 / GPLv3+ / LGPLv3+ +# +# The contents of this file are subject to the Mozilla Public License Version +# 1.1 (the License); you may not use this file except in compliance with +# the License or as specified alternatively below. You may obtain a copy of +# the License at http://www.mozilla.org/MPL/ +# +# Software distributed under the License is distributed on an AS IS basis, +# WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License +# for the specific language governing rights and limitations under the +# License. +# +# The Initial Developer of the Original Code is +# Miklos Vajna vmik...@frugalware.org +# Portions created by the Initial Developer are Copyright (C) 2011 the +# Initial Developer. All Rights Reserved. +# +# Major Contributor(s): +# +# For minor contributions see the git repository. +# +# Alternatively, the contents of this file may be used under the terms of +# either the GNU General Public License Version 3 or later (the GPLv3+), or +# the GNU Lesser General Public License Version 3 or later (the LGPLv3+), +# in which case the provisions of the GPLv3+ or the LGPLv3+ are applicable +# instead of those above. + +import getopt, sys, os, re + +class Options: +Options of this script. + +def __init__(self): +self.input = None +self.output = None +self.language = None +self.template = None + +class Entry: +Represents a single line in an SDF file. + +def __init__(self, items): +self.items = items # list of 15 fields +path = self.items[1].split('\\') +self.po = %s/%s/%s.po % (options.input.replace('\\', '/'), self.items[0], /.join(path[:-1])) +prefix = +if len(self.items[5]): +prefix += %s. % self.items[5] +if len(self.items[3]): +prefix += %s. % self.items[3] +self.keys = [] +# 10..13 are translation types +for idx in range(10, 14): +if len(self.items[idx]): +t = {10:'text', 12:'quickhelptext', 13:'title'}[idx] +self.keys.append((idx, self.sdf2po(%s#%s.%s%s % (path[-1], self.items[4], prefix, t + +def translate(self, translations): +Translates text in the entry based on translations. + +self.items[9] = options.language +for idx, key in self.keys: +try: +self.items[idx] = translations.data[(self.po, key)] + +self.items[14] = 2002-02-02 02:02:02 +except KeyError: +pass +self.items[14] = self.items[14].strip() + +def sdf2po(self, s): +Escapes special chars in po key names. + +return s.translate(normalizetable) + +class Template: +Represents a reference template in SDF format. + +def __init__(self, path): +sock = open(path) +self.lines = [] +for line in sock: +entry = Entry(line.split('\t')) +if os.path.exists(entry.po): +self.lines.append(entry) + +def translate(self, translations): +Translates entires in the template based on translations. + +sock =
Re: [Libreoffice] [PATCH] speed up localized builds by introducing po2lo
I'm attaching an updated patch that fixes po2lo on Windows, where it did not translate anything due to path translating (/ vs \) issues. From a1e8bad75279a05bf6a061c53351898eea72433d Mon Sep 17 00:00:00 2001 From: Miklos Vajna vmik...@frugalware.org Date: Wed, 7 Sep 2011 23:39:15 +0200 Subject: [PATCH] Add po2lo tool --- solenv/bin/po2lo | 202 ++ 1 files changed, 202 insertions(+), 0 deletions(-) create mode 100755 solenv/bin/po2lo diff --git a/solenv/bin/po2lo b/solenv/bin/po2lo new file mode 100755 index 000..08b5519 --- /dev/null +++ b/solenv/bin/po2lo @@ -0,0 +1,202 @@ +#!/usr/bin/env python +# Version: MPL 1.1 / GPLv3+ / LGPLv3+ +# +# The contents of this file are subject to the Mozilla Public License Version +# 1.1 (the License); you may not use this file except in compliance with +# the License or as specified alternatively below. You may obtain a copy of +# the License at http://www.mozilla.org/MPL/ +# +# Software distributed under the License is distributed on an AS IS basis, +# WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License +# for the specific language governing rights and limitations under the +# License. +# +# The Initial Developer of the Original Code is +# Miklos Vajna vmik...@frugalware.org +# Portions created by the Initial Developer are Copyright (C) 2011 the +# Initial Developer. All Rights Reserved. +# +# Major Contributor(s): +# +# For minor contributions see the git repository. +# +# Alternatively, the contents of this file may be used under the terms of +# either the GNU General Public License Version 3 or later (the GPLv3+), or +# the GNU Lesser General Public License Version 3 or later (the LGPLv3+), +# in which case the provisions of the GPLv3+ or the LGPLv3+ are applicable +# instead of those above. + +import getopt, sys, os, re + +class Options: +Options of this script. + +def __init__(self): +self.input = None +self.output = None +self.language = None +self.template = None + +class Entry: +Represents a single line in an SDF file. + +def __init__(self, items): +self.items = items # list of 15 fields +path = self.items[1].split('\\') +self.po = %s/%s/%s.po % (options.input.replace('\\', '/'), self.items[0], /.join(path[:-1])) +prefix = +if len(self.items[5]): +prefix += %s. % self.items[5] +if len(self.items[3]): +prefix += %s. % self.items[3] +self.keys = [] +# 10..13 are translation types +for idx in range(10, 14): +if len(self.items[idx]): +t = {10:'text', 12:'quickhelptext', 13:'title'}[idx] +self.keys.append((idx, self.sdf2po(%s#%s.%s%s % (path[-1], self.items[4], prefix, t + +def translate(self, translations): +Translates text in the entry based on translations. + +self.items[9] = options.language +for idx, key in self.keys: +try: +self.items[idx] = translations.data[(self.po, key)] + +self.items[14] = 2002-02-02 02:02:02 +except KeyError: +pass +self.items[14] = self.items[14].strip() + +def sdf2po(self, s): +Escapes special chars in po key names. + +return s.translate(normalizetable) + +class Template: +Represents a reference template in SDF format. + +def __init__(self, path): +sock = open(path) +self.lines = [] +for line in sock: +self.lines.append(Entry(line.split('\t'))) + +def translate(self, translations): +Translates entires in the template based on translations. + +sock = open(options.output, w) +for line in self.lines: +line.translate(translations) +sock.write(\t.join(line.items)+\r\n) +sock.close() + +class Translations: +Represents a set of .po files, containing translations. + +def __init__(self): +key = None +self.data = {} +for root, dirs, files in os.walk(options.input): +for file in files: +path = %s/%s % (root, file) +sock = open(path) +buf = [] +multiline = False +fuzzy = False +for line in sock: +if line.startswith(#: ): +key = line.strip()[3:] +elif line.startswith(#, fuzzy): +fuzzy = True +elif line.startswith(msgstr ): +trans = line.strip()[8:-1] +if len(trans): +if fuzzy: +fuzzy = False +else: +self.setdata(path, key, trans) +else: +buf = [] +
[Libreoffice] [PATCH] speed up localized builds by introducing po2lo
Hi, We discussed with Andras that one of the bottleneck of a build with many languages is the slowness of the po2oo script from translations-toolkit. I gave it a try to rewrite it as a standalone script, and it seems to be quite fast here: $ time po2oo --skipsource -i hu/ -o hu.sdf -l hu -t en-US.sdf hu.sdf real0m32.842s user0m29.205s sys 0m1.439s $ time ./po2lo --skipsource -i hu/ -o hu.sdf -l hu -t en-US.sdf hu.sdf real0m3.756s user0m3.344s sys 0m0.143s The localized sdf file differs, but once you sort its contents, it's the same for me. Note that this does not remove the dependency on translations-toolkit: the oo2po script is still used (but that's not invoked during a normal build). I'm attaching two patches: - one for core.git, which adds the new po2lo script (localize was already in solenv/bin, so I pot this one there as well). - one for translations.git to actually use the new script: here I took a look at how python is invoked in filters and did the same. Andras, could you please give it some testing? I didn't want to push it without your review, just in case I missed something. :) Thanks, Miklos From 93d0db294db97a5815b4d2f1267826a59b756985 Mon Sep 17 00:00:00 2001 From: Miklos Vajna vmik...@frugalware.org Date: Wed, 7 Sep 2011 23:39:15 +0200 Subject: [PATCH] Add po2lo tool --- solenv/bin/po2lo | 202 ++ 1 files changed, 202 insertions(+), 0 deletions(-) create mode 100755 solenv/bin/po2lo diff --git a/solenv/bin/po2lo b/solenv/bin/po2lo new file mode 100755 index 000..272d6f9 --- /dev/null +++ b/solenv/bin/po2lo @@ -0,0 +1,202 @@ +#!/usr/bin/env python +# Version: MPL 1.1 / GPLv3+ / LGPLv3+ +# +# The contents of this file are subject to the Mozilla Public License Version +# 1.1 (the License); you may not use this file except in compliance with +# the License or as specified alternatively below. You may obtain a copy of +# the License at http://www.mozilla.org/MPL/ +# +# Software distributed under the License is distributed on an AS IS basis, +# WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License +# for the specific language governing rights and limitations under the +# License. +# +# The Initial Developer of the Original Code is +# Miklos Vajna vmik...@frugalware.org +# Portions created by the Initial Developer are Copyright (C) 2011 the +# Initial Developer. All Rights Reserved. +# +# Major Contributor(s): +# +# For minor contributions see the git repository. +# +# Alternatively, the contents of this file may be used under the terms of +# either the GNU General Public License Version 3 or later (the GPLv3+), or +# the GNU Lesser General Public License Version 3 or later (the LGPLv3+), +# in which case the provisions of the GPLv3+ or the LGPLv3+ are applicable +# instead of those above. + +import getopt, sys, os, re + +class Options: +Options of this script. + +def __init__(self): +self.input = None +self.output = None +self.language = None +self.template = None + +class Entry: +Represents a single line in an SDF file. + +def __init__(self, items): +self.items = items # list of 15 fields +path = self.items[1].split('\\') +self.po = %s/%s/%s.po % (options.input, self.items[0], /.join(path[:-1])) +prefix = +if len(self.items[5]): +prefix += %s. % self.items[5] +if len(self.items[3]): +prefix += %s. % self.items[3] +self.keys = [] +# 10..13 are translation types +for idx in range(10, 14): +if len(self.items[idx]): +t = {10:'text', 12:'quickhelptext', 13:'title'}[idx] +self.keys.append((idx, self.sdf2po(%s#%s.%s%s % (path[-1], self.items[4], prefix, t + +def translate(self, translations): +Translates text in the entry based on translations. + +self.items[9] = options.language +for idx, key in self.keys: +try: +self.items[idx] = translations.data[(self.po, key)] + +self.items[14] = 2002-02-02 02:02:02 +except KeyError: +pass +self.items[14] = self.items[14].strip() + +def sdf2po(self, s): +Escapes special chars in po key names. + +return s.translate(normalizetable) + +class Template: +Represents a reference template in SDF format. + +def __init__(self, path): +sock = open(path) +self.lines = [] +for line in sock: +self.lines.append(Entry(line.split('\t'))) + +def translate(self, translations): +Translates entires in the template based on translations. + +sock = open(options.output, w) +for line in self.lines: +line.translate(translations) +sock.write(\t.join(line.items)+\r\n) +sock.close() + +class Translations: +Represents a set of .po files,