[Moses-support] Truecaser question
Hi guys, I would like to ask you how can I improve Truecaser models. For example this is source sentence: "Delineating Supply Chain and Logistics as a set of standard occupational codes would serve to more appropriately spotlight the workforce needed in this industry." And there is translation: "Wyznaczająca Supply Chain and Logistics jako zestaw standardowych kodów zawodowym służyły bardziej odpowiednio reflektorów pracowników potrzebnych w tej branży." As you see "Supply Chain and Logistics" phrase is untranslated. I've checked that the same lowercased phrase is translated without problem. So my question is: what can I do to fix this issue (or at least reduce such results)? Regards, Thomas ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Crushed tuning neural model
Hi, I have strange problem while tuning neural model. Tuning crashes with following error: LOG: EXECUTE STEPS number of steps doable or running: 1 at pią, 5 sie 2016, 09:50:58 CEST doable: TUNING:tune executing /home/moses/working/experiments/ENPL33M6NRN/steps/7/TUNING_tune.7 via sh (1 active) step TUNING:tune crashed TUNING_tune.7.STDERR: Not a label, not a score 'nan'. Failed to parse the scores string: 'NPLMNEURAL= nan LexicalReordering0= 0 0 0 0 0 0 Distortion0= -8 LM0= -67.8925 WordPenalty0= -5 PhrasePenalty0= 5 TranslationModel0= 0 0 0 0' of nbestlist 'run1.best100.out' at /home/moses/src/mosesdecoder/scripts/training/mert-moses.pl line 1503. Run.best100.out: moses@smtserver:~/working/experiments/ENPL33M6NRN/tuning/tmp.7$ head run1.best100.out 0 ||| let . something s try ||| NPLMNEURAL= nan LexicalReordering0= 0 0 0 0 0 0 Distortion0= -8 LM0= -67.8925 WordPenalty0= -5 PhrasePenalty0= 5 TranslationModel0= 0 0 0 0 ||| nan 0 ||| try let . s something ||| NPLMNEURAL= nan LexicalReordering0= 0 0 0 0 0 0 Distortion0= -13 LM0= -68.7339 WordPenalty0= -5 PhrasePenalty0= 5 TranslationModel0= 0 0 0 0 ||| nan 0 ||| something s . try let ||| NPLMNEURAL= nan LexicalReordering0= 0 0 0 0 0 0 Distortion0= -14 LM0= -70.253 WordPenalty0= -5 PhrasePenalty0= 5 TranslationModel0= 0 0 0 0 ||| nan What is the reason of appearing these "nan" values? I've successfully trained smaller (1000 sentences) corpus using the same config file. Please help! :) Regards, Thomas ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] error when building moses with nplm
Hi, I'm trying to compile mosesdecoder with NPLM with INTEL MKL but without success. I did some searches in Google and found similar request but unanswered: http://comments.gmane.org/gmane.comp.nlp.moses.user/11985 I compiled and installed NPLM and MLK without problems. Here is my error: gcc.compile.c++ lm/bin/gcc-4.8/release/link-static/threading-multi/fragment_main.o gcc.link lm/bin/gcc-4.8/release/link-static/threading-multi/fragment /home/moses/src/nplm/src/libnplm.a(model.o): In function `nplm::model::premultiply()': model.cpp:(.text+0x6864): undefined reference to `sgemm_' /home/moses/src/nplm/src/libnplm.a(util.o): In function `nplm::setup_threads(int)': util.cpp:(.text+0x57e): undefined reference to `MKL_Set_Threading_Layer' util.cpp:(.text+0x585): undefined reference to `MKL_Set_Num_Threads' collect2: error: ld returned 1 exit status It looks like problem with MKL but I wouldn't like to resign of MKL.. Do you have any ideas how to fix this problem? Best regards, Thomas ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] EMS: pruning question
Hi! I have one question about pruning translation table during EMS training. What method is better SALM or based on low scores (described here: http://www.statmt.org/moses/?n=Advanced.RuleTables#ntoc5) ? SALM filtering takes relatively more time than pruning while LM creation. But I'm not sure what to do: 1. Keep both, 2. Choose one of them (which one and why? :) ) 3. Don't prune (why?) Thank you in advance for any suggestions :) Regards, Thomas ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Tuning crashed
Hi, I have a problem with tuning crashed. It seems that moses.ini is missing in temporary folder but I have no idea why. I attached link to my config file. Please help. Regards Thomas moses@smtserver:~/working/experiments/NGRAM5/steps/2$ more TUNING_tune.2.STDERR Using SCRIPTS_ROOTDIR: /home/moses/src/mosesdecoder/scripts Asking moses for feature names and values from /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 Executing: /home/moses/src/mosesdecoder/bin/moses -threads 16 -v 0 -config /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 -show-weights exec: /home/moses/src/mosesdecoder/bin/moses -threads 16 -v 0 -config /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 -show-weights Executing: /home/moses/src/mosesdecoder/bin/moses -threads 16 -v 0 -config /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 -show-weights > ./features.list 2> /dev/null Exit code: 1 ERROR: Failed to run '/home/moses/src/mosesdecoder/bin/moses -threads 16 -v 0 -config /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 -show-weights'. at /home/moses/src/mosesdecoder/scripts/training/mert-moses.pl line 1748. cp: cannot stat '/home/moses/working/experiments/NGRAM5/tuning/tmp.2/moses.ini': No such file or directory https://docs.google.com/document/d/1gI7YVUx8VoktIfIQvvU54jKSm5Ta6UZjYcszBFtP -V8/edit?usp=sharing ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] DEBUG_LEVEL:1 Error: lower order count-of-counts cannot be estimated properly
Hi, I'm trying to build 10 ngram's model but my training pipeline ends with error: "Error: lower order count-of-counts cannot be estimated properly". Corpus has 33 mln sentences. I successfully trained much smaller corpus (around 5 mln sentences) using the same config file. Would you suggest me something how to fix this problem? Regards, Thomas -- # more steps/2/LM_ACROSS-BIGMAMA-OPENSUB2016_train.2.STDERR Generating successor statistics level 2 level 3 level 4 level 5 level 6 level 7 level 8 level 9 level 10 level 1 computing statistics n1: 1 n2: 0 n3: 0 n4: 0 unover3: 0 DEBUG_LEVEL:1 Error: lower order count-of-counts cannot be estimated properly Hint: use another smoothing method with this corpus. EXECUTING rm -rf /home/moses/working/experiments/NGRAM10-A/tmp/irstlm-build-tmp.6920 FINISH. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Moses-support Digest, Vol 115, Issue 8
Hi, Please forget my post. I found where was the problem. I tried to use advanced version of xmrpc-c (should use stable release). Regards, Tomek Date: Fri, 6 May 2016 15:42:08 +0200 From: "Tomasz Gawryl" <tomasz.gaw...@skrivanek.pl> Subject: [Moses-support] Failed mosesdecoder compilation with xmlrpc To: <moses-support@mit.edu> Message-ID: <009701d1a79d$158273f0$40875bd0$@skrivanek.pl> Content-Type: text/plain; charset="us-ascii" Hi, I'm trying to compile mosesdecoder with xml-rpc support for mosesserver but without success. I downloaded and installed this library in custom directory and use this command to run compilation process: ./bjam --prefix=/home/moses/src/mosesdecoder -q --with-irstlm=/home/moses/src/irstlm-5.80.08/trunk --with-cmph=/home/moses/src/cmph-2.0 --with-xmlrpc-c=/home/moses/src/mosesdecoder/xmlrpc-c --with=mm --with-probing-pt=/home/moses/src/Pr obingPT --max-kenlm-order=10 -j$(getconf _NPROCESSORS_ONLN) $@ But it produces error and fails compilation (log below). Compilation without xmlrpc support ends with success. Any ideas? Best Regards, Tomek /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `channelRead': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:320: undefined reference to `SSL_read' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:325: undefined reference to `SSL_get_error' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `channelWrite': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:285: undefined reference to `SSL_write' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:290: undefined reference to `SSL_get_error' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `channelDestroy': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:258: undefined reference to `SSL_shutdown' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslCreate': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:121: undefined reference to `SSL_new' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslSetFd': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:142: undefined reference to `SSL_set_fd' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslAccept': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:191: undefined reference to `SSL_accept' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:196: undefined reference to `SSL_get_error' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslErrorMsg': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:71: undefined reference to `ERR_get_error_line' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `getPeerAddrFromSsl': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:415: undefined reference to `SSL_get_fd' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `createChannelFromAcceptedConn': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:671: undefined reference to `SSL_free' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslErrorMsg': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:71: undefined reference to `ERR_get_error_line' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `createSslFromAcceptedConn': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:623: undefined reference to `SSL_free' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslErrorMsg': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:78: undefined reference to `ERR_error_string' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:71: undefined reference to `ERR_get_error_line' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:78: undefined reference to `ERR_error_string' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:78: undefined reference to `ERR_error_string' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `traceCipherList': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:170: undefined reference to `SSL_get_cipher_list' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:170: undefined reference to `SSL_get_cipher_list' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `SocketOpenSslInit': /home/moses/s
[Moses-support] Failed mosesdecoder compilation with xmlrpc
Hi, I'm trying to compile mosesdecoder with xml-rpc support for mosesserver but without success. I downloaded and installed this library in custom directory and use this command to run compilation process: ./bjam --prefix=/home/moses/src/mosesdecoder -q --with-irstlm=/home/moses/src/irstlm-5.80.08/trunk --with-cmph=/home/moses/src/cmph-2.0 --with-xmlrpc-c=/home/moses/src/mosesdecoder/xmlrpc-c --with=mm --with-probing-pt=/home/moses/src/Pr obingPT --max-kenlm-order=10 -j$(getconf _NPROCESSORS_ONLN) $@ But it produces error and fails compilation (log below). Compilation without xmlrpc support ends with success. Any ideas? Best Regards, Tomek /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `channelRead': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:320: undefined reference to `SSL_read' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:325: undefined reference to `SSL_get_error' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `channelWrite': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:285: undefined reference to `SSL_write' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:290: undefined reference to `SSL_get_error' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `channelDestroy': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:258: undefined reference to `SSL_shutdown' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslCreate': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:121: undefined reference to `SSL_new' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslSetFd': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:142: undefined reference to `SSL_set_fd' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslAccept': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:191: undefined reference to `SSL_accept' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:196: undefined reference to `SSL_get_error' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslErrorMsg': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:71: undefined reference to `ERR_get_error_line' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `getPeerAddrFromSsl': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:415: undefined reference to `SSL_get_fd' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `createChannelFromAcceptedConn': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:671: undefined reference to `SSL_free' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslErrorMsg': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:71: undefined reference to `ERR_get_error_line' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `createSslFromAcceptedConn': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:623: undefined reference to `SSL_free' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslErrorMsg': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:78: undefined reference to `ERR_error_string' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:71: undefined reference to `ERR_get_error_line' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:78: undefined reference to `ERR_error_string' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:78: undefined reference to `ERR_error_string' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `traceCipherList': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:170: undefined reference to `SSL_get_cipher_list' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:170: undefined reference to `SSL_get_cipher_list' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `SocketOpenSslInit': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:228: undefined reference to `SSL_load_error_strings' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:230: undefined reference to `SSL_library_init' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `getPeerAddrFromSsl': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:415: undefined reference to `SSL_get_fd'
[Moses-support] Moses, python_server, matecat_utils, MateCAT - Moses-support Digest, Vol 109, Issue 19
Hi Philipp,
Thank you for this solution. I tested python_server and this wrapper works
very good and does exactly what I want. Unfortunately it won't help in case
of integration with Across Language Server because Across requires direct
RPC calls to Moses Server.
So, I'm looking for another CAT tool which supports python_server wrapper.
It's a fork of matecat_utils package so I suppose deliver tools for Matecat
project. If any of you use MateCAT software? I need to confirm that Matecat
supports python_server (I cannot check it myself because of Matecat's login
requirement by Google authorization - which isn't possible on my virtual
machine at this moment).
Sorry for my "english" ;)
Regards,
Tomek
--
Message: 1
Date: Thu, 12 Nov 2015 12:36:20 -0500
From: Philipp Koehn <p...@jhu.edu>
Subject: Re: [Moses-support] Moses-support Digest, Vol 109, Issue 19
To: Panos Kanavos <panoskana...@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAAFADDDkgWN-0iqK_J_9baRCOMNpj+RSFz6r6=3=c1x=udh...@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi,
there are a lot of different pre and post processing steps that you may want
to apply for any given language pair, so it makes sense to keep them out of
the decoder.
If you are interested in a server implementation that integrates
tokenization, truecasing, etc., check out Christian Buck's implementation of
the Google Translate API for Moses:
https://github.com/casmacat/moses-mt-server/tree/master/python_server
-phi
On Thu, Nov 12, 2015 at 9:52 AM, Panos Kanavos <panoskana...@gmail.com>
wrote:
> Thanks for the info Hieu, didn't know that:) I'll try it sometime.
>
> Best,
>
> Panos
>
>
> On 12/11/2015 4:41 ??, Hieu Hoang wrote:
>
> there has been thoughts. There is a c++ tokenizer in
>contrib/c++tokenizer
> it compiles into a library file, ready for integration.
>
> The last time i checked, it gave a slightly worse BLEU. Not much, but
> consistent.
>
> If anyone wants to carry on with it, they're welcome to
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 12 November 2015 at 14:37, Panos Kanavos <panoskana...@gmail.com>
> wrote:
>
>> Hi Barry,
>>
>> Have there ever been any thoughts about implementing
>> tokenization/detokenization directly in Moses? I suppose this is some
>> work as Moses should become language-aware, but I can only see
>> advantanges from this. Besides, Moses is a language tool so these
>> concepts shouldn't be so disconnected from its core logic.
>>
>> Best,
>>
>> Panos
>>
>>
>> On 12/11/2015 4:02 ??, Barry Haddow wrote:
>> > Hi Tomasz
>> >
>> > The moseserver is just the decoder, so it doesn't do any of the
>> > pre- and post-processing steps that you also need. In particular it
>> > does not do tokenisation. You need to send it tokenised text, and
>> > then de-tokenise the output,
>> >
>> > cheers - Barry
>> >
>> > On 12/11/15 13:40, Tomasz Gawryl wrote:
>> >> Hi Ulrich,
>> >>
>> >> I have a question about Moses server too. I'm testing it as a
>> >> wrapper
>> for
>> >> Across server to check pre-translation possibilities. It generally
>> works but
>> >> there is one problem. Input segments are translated without
>> tokenization, so
>> >> every word close to special character (for example `this is small
>> house.`)
>> >> remains untranslated ('to jest ma?y house.'). I was searching list
>> archive
>> >> and I found similar question here:
>> >> http://comments.gmane.org/gmane.comp.nlp.moses.user/14020 but for
>> >> me
>> it's
>> >> not yet answered. I would appreciate any information on this subject.
>> >>
>> >> Best regards,
>> >> Tomek
>> >>
>> >> -Original Message-
>> >> From: moses-support-boun...@mit.edu [mailto:
>> moses-support-boun...@mit.edu]
>> >> On Behalf Of moses-support-requ...@mit.edu
>> >> Sent: Wednesday, November 11, 2015 10:31 AM
>> >> To: moses-support@mit.edu
>> >> Subject: Moses-support Digest, Vol 109, Issue 19
>> >>
>> >> Send Moses-support mailing list submissions to
>> >> moses-support@mit.edu
>> >>
>> >> To subscribe or unsubscribe via the World Wide Web, visit
>> >> http://mailman.mit.edu/mailman/listinfo/moses-support
>> >> or, via email, send a message with subject or body 'help' to
>> &g
Re: [Moses-support] Moses-support Digest, Vol 109, Issue 19
Hi Ulrich,
I have a question about Moses server too. I'm testing it as a wrapper for
Across server to check pre-translation possibilities. It generally works but
there is one problem. Input segments are translated without tokenization, so
every word close to special character (for example `this is small house.`)
remains untranslated ('to jest mały house.'). I was searching list archive
and I found similar question here:
http://comments.gmane.org/gmane.comp.nlp.moses.user/14020 but for me it's
not yet answered. I would appreciate any information on this subject.
Best regards,
Tomek
-Original Message-
From: moses-support-boun...@mit.edu [mailto:moses-support-boun...@mit.edu]
On Behalf Of moses-support-requ...@mit.edu
Sent: Wednesday, November 11, 2015 10:31 AM
To: moses-support@mit.edu
Subject: Moses-support Digest, Vol 109, Issue 19
Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-requ...@mit.edu
You can reach the person managing the list at
moses-support-ow...@mit.edu
When replying, please edit your Subject line so it is more specific than
"Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: use placeholder with mosesserver (Ulrich Germann)
2. Re: use placeholder with mosesserver (Evgeny Matusov)
--
Message: 1
Date: Wed, 11 Nov 2015 01:58:40 +
From: Ulrich Germann
Subject: Re: [Moses-support] use placeholder with mosesserver
To: Evgeny Matusov
Cc: "moses-support@mit.edu"
Message-ID:
Content-Type: text/plain; charset="utf-8"
Hi all,
I've just pushed what I believe might address a few of the issues in this
thread:
- the more fine-grained configuration options for request handling and
queuing, server timeouts etc. (added in August due to threading issue) have
been transferred to the main moses executable.
- the server now pays attention to the xml-input option specified via json;
the range of accepted values is the same as when specified on the command
line. I have not written the xml-input handling and do not actively use it,
so it may or may not work. I don't think there are any regression tests that
test this right now. Reports from the trenches are welcome.
- mosesserver.cpp is deprecated. It is now merely a shell around the regular
moses call with --server. I did not remove it from the code base entirely,
as I assume that there's a plethora of setups out there that rely on the
existence of mosesserver. What the wrapper does is add --server to the
options and then call run regular moses.
- anyone adding stuff to mosesserver.cpp from now on owes me a lifetime
supply of the finest Laphroaig. Just send me a quarter cask every year for
Burns Nicht for the rest of my life if you do. If I haven't pushed anything
for two years, you may assume I'm dead.
- Uli
On Tue, Nov 10, 2015 at 2:58 PM, Ulrich Germann
wrote:
> Hi all,
>
> mosesserver is deprecated and should not be used any more. I'll
> transfer the threading-related changes to the server implementation in
> the regular moses executable and let you know once I'm done so that
> other things (like
> passthrough) can be added. By the looks of it, the changes are fairly
> straightforward, so it shouldn't take long. However, I can't guarantee
> that the new server will do everything the old server did, (or do it
> the same way).
>
> It would be fantastic if a few people could design and contribute test
> cases so that we can do some regression testing for the server.
> Ideally a test case should provide:
>
> - tiny models to work with (or we may be able to recycle some that
> already
> exist)
> - sample input (json)
> - expected output (json)
>
> Cheers - Uli
>
> On Tue, Nov 10, 2015 at 11:37 AM, Evgeny Matusov
> wrote:
>
>>
>> Hi,
>>
>> can any of the more active recent developers advise what is the
>> latest stable mosesserver implementation?
>>
>> It seems to be the one in moses/server, but the one in in
>> contrib/server/mosesserver.cpp has been updated in August of this
>> year with an important fix related to multiple threads:
>>
>>
>> https://github.com/moses-smt/mosesdecoder/commit/3c682fa8b05af6bff1a0
>> 9f420141795875cf9685
>> https://www.mail-archive.com/moses-support%40mit.edu/msg12875.html
>>
>> As Gregor mentioned, we would like to share our fix so that
>> Mosesserver correctly supports placeholders. I want to make sure that
>> this is a fix for something that many people use without problems.
>>
>> Thanks,
>> Evgeny.
>>
>>
>>
>> From:
[Moses-support] factored tuning time
Hi, I've one question to you about time of factored tuning. How many times longer it takes compared to phrase based tuning? I'm asking because it's 7'th day and it's still tuning (3,3 mln corpus sentences). Phrase based tuning took around 3h for the same corpus. Top shows me that moses uses near 100% CPU. So the speed is the same. Regards, Tomek Gawryl ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] error during tuning step (factored training)
Hi Hieu,
Command below (I hope this is it ;).
Regards,
Tomek
moses@SKR-moses:~/working/experiments/FACTORED/steps/1$ more
TRAINING_build-ttable.1
#!/bin/bash
[...]
/home/moses/src/mosesdecoder/scripts/training/train-model.perl -mgiza
-mgiza-cpus 8 -cores 8 -parallel -sort-buffer-size 10G -sort-batch-size 253
-sort-compre
ss gzip -sort-parallel 10 -dont-zip -first-step 6 -last-step 6
-external-bin-dir /home/moses/src/mosesdecoder/tools -f en -e pl -alignment
grow-diag-final-and
-max-phrase-length 5 -parts 5 -reordering msd-bidirectional-fe
-score-options '--GoodTuring --MinScore 2:0.0001' -parallel
-input-factor-max 0 -alignment-fac
tors 0-0 -translation-factors 0-0,1 -reordering-factors 0-0 -decoding-steps
t0 -extract-file /home/moses/working/experiments/FACTORED/model/extract.1
-lexical
-file /home/moses/working/experiments/FACTORED/model/lex.1
-phrase-translation-table
/home/moses/working/experiments/FACTORED/model/phrase-table.1.0-0,1
moses@SKR-moses:~/working/experiments/FACTORED/steps/1$ more
TRAINING_binarize-config.1
#!/bin/bash
[...]
mkdir -p /home/moses/working/experiments/FACTORED/model
/home/moses/src/mosesdecoder/scripts/training/binarize-model.perl
/home/moses/working/experiments/FACTORED/model/moses.ini.1
/home/moses/working/experiments/FACTORED/model/moses.bin.ini.1 -B
inarizer "/home/moses/src/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2"
--
whats the exact command you used to binarize the phrase-table?
On 11/10/2015 17:00, Tomasz Gawryl wrote:
>
> Hi,
>
> My factored training ends core dumped during tuning step.:
>
> /home/moses/working/experiments/FACTORED/lm/ACROSS=pos.lm.1: line 11:
> warning: n
>
> on-zero probability for in closed-vocabulary LM
>
> terminate called after throwing an instance of 'util::Exception'
>
> what():
> moses/TranslationModel/RuleTable/PhraseDictionaryOnDisk.cpp:100 in vi
>
> rtual void Moses::PhraseDictionaryOnDisk::InitializeForInput(const
> ttasksptr&) t
>
> hrew util::Exception because `obj->GetMisc("NumTargetFactors") !=
> m_output.size(
>
> )'.
>
> On-disk phrase table has 1 target factors.. The ini file specified 2
> target factors
>
> Aborted (core dumped)
>
> Exit code: 134
>
> My config file is similar to example config file for factored trainings.
>
> What should I do to fix it?
>
> Regards
>
> **
>
> *Tomek *
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] error during tuning step (factored training)
Hi,
My factored training ends core dumped during tuning step.:
/home/moses/working/experiments/FACTORED/lm/ACROSS=pos.lm.1: line 11:
warning: n
on-zero probability for in closed-vocabulary LM
terminate called after throwing an instance of 'util::Exception'
what(): moses/TranslationModel/RuleTable/PhraseDictionaryOnDisk.cpp:100
in vi
rtual void Moses::PhraseDictionaryOnDisk::InitializeForInput(const
ttasksptr&) t
hrew util::Exception because `obj->GetMisc("NumTargetFactors") !=
m_output.size(
)'.
On-disk phrase table has 1 target factors.. The ini file specified 2 target
factors
Aborted (core dumped)
Exit code: 134
My config file is similar to example config file for factored trainings.
What should I do to fix it?
Regards
Tomek
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] experiment.perl, processPhraseTableMin and threads option
Hi Philipp, Shame on me, you are right. Thanx for help! Regards, TG From: phko...@gmail.com [mailto:phko...@gmail.com] On Behalf Of Philipp Koehn Sent: Wednesday, September 23, 2015 3:30 PM To: Tomasz Gawryl Cc: moses-support@mit.edu Subject: Re: [Moses-support] experiment.perl, processPhraseTableMin and threads option Hi, I ran the command as you provided it, and it worked. The "-threads all" functionality was added in August - did you compile the latest version of the code? -phi On Tue, Sep 22, 2015 at 12:16 PM, Tomasz Gawryl <tomasz.gaw...@skrivanek.pl> wrote: Hi, I'm trying to add this line to config: ttable-binarizer = "$moses-bin-dir/processPhraseTableMin -threads all" This option is supported by processPhraseTableMin command(https://github.com/moses-smt/mosesdecoder/blob/master/misc/processPh raseTableMin.cpp line 24). /home/moses/src/mosesdecoder/scripts/training/binarize-model.perl /home/moses/working/experiments/model/moses.ini.1 /home/mose s/working/experiments/model/moses.bin.ini.3 -Binarizer /home/moses/src/mosesdecoder/bin/processPhraseTableMin -threads all But it produces error "Unknown option: threads" in file TRAINING_binarize-config.3.STDERR (and stops training). I removed this option but it seems that such case it uses only one thread: moses 1470 113 10.8 4255200 3448392 pts/13 Sl 11:05 66:03 /home/moses/src/mosesdecoder/bin/processPhraseTableMin -in /home/moses/working/experiments/model/moses.bin.ini.5.tables/phrase-table.0- 0.1.1.gz.sorted -out /home/moses/working/experiments/model/moses.bin.ini.5.tables/phrase-table.0- 0.1.1 -nscores 4 -threads 1 I know that my server is able to run around 16 threads (and indeed does during former steps). What can I do to improve this step to use more threads? Regards, TG ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] experiment.perl, processPhraseTableMin and threads option
Hi, I'm trying to add this line to config: ttable-binarizer = "$moses-bin-dir/processPhraseTableMin -threads all" This option is supported by processPhraseTableMin command(https://github.com/moses-smt/mosesdecoder/blob/master/misc/processPh raseTableMin.cpp line 24). /home/moses/src/mosesdecoder/scripts/training/binarize-model.perl /home/moses/working/experiments/model/moses.ini.1 /home/mose s/working/experiments/model/moses.bin.ini.3 -Binarizer /home/moses/src/mosesdecoder/bin/processPhraseTableMin -threads all But it produces error "Unknown option: threads" in file TRAINING_binarize-config.3.STDERR (and stops training). I removed this option but it seems that such case it uses only one thread: moses 1470 113 10.8 4255200 3448392 pts/13 Sl 11:05 66:03 /home/moses/src/mosesdecoder/bin/processPhraseTableMin -in /home/moses/working/experiments/model/moses.bin.ini.5.tables/phrase-table.0- 0.1.1.gz.sorted -out /home/moses/working/experiments/model/moses.bin.ini.5.tables/phrase-table.0- 0.1.1 -nscores 4 -threads 1 I know that my server is able to run around 16 threads (and indeed does during former steps). What can I do to improve this step to use more threads? Regards, TG ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] BLEU score
Hi All! This is my first post here and AT first I want to apologize for my English but I would like to ask you some questions. I finished a full phrase based Moses training of EN-PL (English - Polish) corpus (few million sentences from free sources + half million sentences from commercial tmx). Training pipeline always ends with test translation and BLEU score. I didn't expect the first score around 30% but my result 4.5% surprised me. Why my result is so bad? Is it a consequence of chosen language pair? Polish language is very flexible - we can interchange words in a sentence without losing sense. What should I do to improve this result? Or maybe that's all I can get ;). Regards, Tomek ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
