Re: [Moses-support] Moses-support Digest, Vol 122, Issue 29
Hi, If you want to speed up decoding time maybe you should consider changing searching algorithm. I'm also using compact phrase tables and after some test I realised that cube pruning gives almost exactly the same quality but is much faster. For example you can add something like this to your config file: # Cube Pruning [search-algorithm] 1 [cube-pruning-pop-limit] 1000 [stack] 50 If your model allows you may also try moses2 binary which is faster than original. Regards, Thomas -- Message: 1 Date: Thu, 15 Dec 2016 19:12:01 +0530 From: Shubham Khandelwal Subject: Re: [Moses-support] Regarding Decoding Time To: Hieu Hoang Cc: moses-support Message-ID: Content-Type: text/plain; charset="utf-8" Hello, Currently, I am using phrase-table.minphr , reordering-table.minlexr and language model (total size of these 3 are 6 GB). Now, I tried to decode on two different machines (8 core-16GB RAM *&* 4 core-40GB RAM) using them. So, During decoding of around 500 words, it took 90 seconds and 100 seconds respectively on those machines. I am already using compact phrase and reordering table representations for faster decoding. Is there any other way to reduce this decoding time. Also, In Moses, Do we have distributed way of decoding on multiple machines ? Looking forward for your response. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] EMS: filter-model-given-input.pl and threads
Hi All, I'm trying to create compact phrase table during BilingualLM training. It works fine but there is one moment when process slows down. I set up 18 threads for TRAINING:binarize-config step : ttable-binarizer = "$moses-bin-dir/processPhraseTableMin -threads 18" But one of scripts (filter-model-given-input.pl) adds own default setting "threads 1" what overrides 18 threads: moses 8081 0.0 0.0 24840 6976 pts/5SN 11:13 0:00 perl /home/moses/src/mosesdecoder/scripts/training/binarize-model.perl /home/moses/working/experiments/EN-PL/BilingualLM/model/moses.ini.1 /home/moses/working/experiments/EN-PL/BilingualLM/model/moses.bin.ini.2 -Binarizer /home/moses/src/mosesdecoder/bin/processPhraseTableMin -threads 18 moses 8083 0.0 0.0 20984 7120 pts/5SN 11:13 0:00 perl /home/moses/src/mosesdecoder/scripts/training/filter-model-given-input.pl /home/moses/working/experiments/EN-PL/BilingualLM/model/moses.bin.ini.2.tabl es /home/moses/working/experiments/EN-PL/BilingualLM/model/moses.ini.1 /dev/null -nofilter -Binarizer /home/moses/src/mosesdecoder/bin/processPhraseTableMin -threads 18 moses31792 126 2.6 2357912 1996868 pts/5 SNl 13:48 46:44 /home/moses/src/mosesdecoder/bin/processPhraseTableMin -threads 18 -in /home/moses/working/experiments/EN-PL/BilingualLM/model/moses.bin.ini.2.tabl es/phrase-table.0-0.1.1.gz.sorted.gz -out /home/moses/working/experiments/EN-PL/BilingualLM/model/moses.bin.ini.2.tabl es/phrase-table.0-0.1.1 -nscores 4 -threads 1 My question is : should it be as it is or I made some error in configuration? Regards, Thomas ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Truecaser question
Hi guys, I would like to ask you how can I improve Truecaser models. For example this is source sentence: "Delineating Supply Chain and Logistics as a set of standard occupational codes would serve to more appropriately spotlight the workforce needed in this industry." And there is translation: "Wyznaczająca Supply Chain and Logistics jako zestaw standardowych kodów zawodowym służyły bardziej odpowiednio reflektorów pracowników potrzebnych w tej branży." As you see "Supply Chain and Logistics" phrase is untranslated. I've checked that the same lowercased phrase is translated without problem. So my question is: what can I do to fix this issue (or at least reduce such results)? Regards, Thomas ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Crushed tuning neural model
Hi, I have strange problem while tuning neural model. Tuning crashes with following error: LOG: EXECUTE STEPS number of steps doable or running: 1 at pią, 5 sie 2016, 09:50:58 CEST doable: TUNING:tune executing /home/moses/working/experiments/ENPL33M6NRN/steps/7/TUNING_tune.7 via sh (1 active) step TUNING:tune crashed TUNING_tune.7.STDERR: Not a label, not a score 'nan'. Failed to parse the scores string: 'NPLMNEURAL= nan LexicalReordering0= 0 0 0 0 0 0 Distortion0= -8 LM0= -67.8925 WordPenalty0= -5 PhrasePenalty0= 5 TranslationModel0= 0 0 0 0' of nbestlist 'run1.best100.out' at /home/moses/src/mosesdecoder/scripts/training/mert-moses.pl line 1503. Run.best100.out: moses@smtserver:~/working/experiments/ENPL33M6NRN/tuning/tmp.7$ head run1.best100.out 0 ||| let . something 's try ||| NPLMNEURAL= nan LexicalReordering0= 0 0 0 0 0 0 Distortion0= -8 LM0= -67.8925 WordPenalty0= -5 PhrasePenalty0= 5 TranslationModel0= 0 0 0 0 ||| nan 0 ||| try let . 's something ||| NPLMNEURAL= nan LexicalReordering0= 0 0 0 0 0 0 Distortion0= -13 LM0= -68.7339 WordPenalty0= -5 PhrasePenalty0= 5 TranslationModel0= 0 0 0 0 ||| nan 0 ||| something 's . try let ||| NPLMNEURAL= nan LexicalReordering0= 0 0 0 0 0 0 Distortion0= -14 LM0= -70.253 WordPenalty0= -5 PhrasePenalty0= 5 TranslationModel0= 0 0 0 0 ||| nan What is the reason of appearing these "nan" values? I've successfully trained smaller (1000 sentences) corpus using the same config file. Please help! :) Regards, Thomas ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] error when building moses with nplm
Hi, I'm trying to compile mosesdecoder with NPLM with INTEL MKL but without success. I did some searches in Google and found similar request but unanswered: http://comments.gmane.org/gmane.comp.nlp.moses.user/11985 I compiled and installed NPLM and MLK without problems. Here is my error: gcc.compile.c++ lm/bin/gcc-4.8/release/link-static/threading-multi/fragment_main.o gcc.link lm/bin/gcc-4.8/release/link-static/threading-multi/fragment /home/moses/src/nplm/src/libnplm.a(model.o): In function `nplm::model::premultiply()': model.cpp:(.text+0x6864): undefined reference to `sgemm_' /home/moses/src/nplm/src/libnplm.a(util.o): In function `nplm::setup_threads(int)': util.cpp:(.text+0x57e): undefined reference to `MKL_Set_Threading_Layer' util.cpp:(.text+0x585): undefined reference to `MKL_Set_Num_Threads' collect2: error: ld returned 1 exit status It looks like problem with MKL but I wouldn't like to resign of MKL.. Do you have any ideas how to fix this problem? Best regards, Thomas ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] EMS: pruning question
Hi! I have one question about pruning translation table during EMS training. What method is better SALM or based on low scores (described here: http://www.statmt.org/moses/?n=Advanced.RuleTables#ntoc5) ? SALM filtering takes relatively more time than pruning while LM creation. But I'm not sure what to do: 1. Keep both, 2. Choose one of them (which one and why? :) ) 3. Don't prune (why?) Thank you in advance for any suggestions :) Regards, Thomas ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] EMS as translator
Hi, I'm wondering is it possible to create EMS config not for training translation model but only for translation related steps (tokenization, truecasing, translation, detokenization, recasing etc)? If yes maybe someone Has such config to share or instruction how to do it? I was googling for it but without success. If not maybe you know similar solution? Regards Thomas ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Tuning crashed
Hi, I have a problem with tuning crashed. It seems that moses.ini is missing in temporary folder but I have no idea why. I attached link to my config file. Please help. Regards Thomas moses@smtserver:~/working/experiments/NGRAM5/steps/2$ more TUNING_tune.2.STDERR Using SCRIPTS_ROOTDIR: /home/moses/src/mosesdecoder/scripts Asking moses for feature names and values from /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 Executing: /home/moses/src/mosesdecoder/bin/moses -threads 16 -v 0 -config /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 -show-weights exec: /home/moses/src/mosesdecoder/bin/moses -threads 16 -v 0 -config /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 -show-weights Executing: /home/moses/src/mosesdecoder/bin/moses -threads 16 -v 0 -config /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 -show-weights > ./features.list 2> /dev/null Exit code: 1 ERROR: Failed to run '/home/moses/src/mosesdecoder/bin/moses -threads 16 -v 0 -config /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 -show-weights'. at /home/moses/src/mosesdecoder/scripts/training/mert-moses.pl line 1748. cp: cannot stat '/home/moses/working/experiments/NGRAM5/tuning/tmp.2/moses.ini': No such file or directory https://docs.google.com/document/d/1gI7YVUx8VoktIfIQvvU54jKSm5Ta6UZjYcszBFtP -V8/edit?usp=sharing ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] DEBUG_LEVEL:1 Error: lower order count-of-counts cannot be estimated properly
Hi, I'm trying to build 10 ngram's model but my training pipeline ends with error: "Error: lower order count-of-counts cannot be estimated properly". Corpus has 33 mln sentences. I successfully trained much smaller corpus (around 5 mln sentences) using the same config file. Would you suggest me something how to fix this problem? Regards, Thomas -- # more steps/2/LM_ACROSS-BIGMAMA-OPENSUB2016_train.2.STDERR Generating successor statistics level 2 level 3 level 4 level 5 level 6 level 7 level 8 level 9 level 10 level 1 computing statistics n1: 1 n2: 0 n3: 0 n4: 0 unover3: 0 DEBUG_LEVEL:1 Error: lower order count-of-counts cannot be estimated properly Hint: use another smoothing method with this corpus. EXECUTING rm -rf /home/moses/working/experiments/NGRAM10-A/tmp/irstlm-build-tmp.6920 FINISH. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Moses-support Digest, Vol 115, Issue 8
Hi, Please forget my post. I found where was the problem. I tried to use advanced version of xmrpc-c (should use stable release). Regards, Tomek Date: Fri, 6 May 2016 15:42:08 +0200 From: "Tomasz Gawryl" Subject: [Moses-support] Failed mosesdecoder compilation with xmlrpc To: Message-ID: <009701d1a79d$158273f0$40875bd0$@skrivanek.pl> Content-Type: text/plain; charset="us-ascii" Hi, I'm trying to compile mosesdecoder with xml-rpc support for mosesserver but without success. I downloaded and installed this library in custom directory and use this command to run compilation process: ./bjam --prefix=/home/moses/src/mosesdecoder -q --with-irstlm=/home/moses/src/irstlm-5.80.08/trunk --with-cmph=/home/moses/src/cmph-2.0 --with-xmlrpc-c=/home/moses/src/mosesdecoder/xmlrpc-c --with=mm --with-probing-pt=/home/moses/src/Pr obingPT --max-kenlm-order=10 -j$(getconf _NPROCESSORS_ONLN) $@ But it produces error and fails compilation (log below). Compilation without xmlrpc support ends with success. Any ideas? Best Regards, Tomek /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `channelRead': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:320: undefined reference to `SSL_read' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:325: undefined reference to `SSL_get_error' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `channelWrite': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:285: undefined reference to `SSL_write' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:290: undefined reference to `SSL_get_error' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `channelDestroy': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:258: undefined reference to `SSL_shutdown' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslCreate': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:121: undefined reference to `SSL_new' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslSetFd': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:142: undefined reference to `SSL_set_fd' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslAccept': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:191: undefined reference to `SSL_accept' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:196: undefined reference to `SSL_get_error' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslErrorMsg': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:71: undefined reference to `ERR_get_error_line' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `getPeerAddrFromSsl': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:415: undefined reference to `SSL_get_fd' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `createChannelFromAcceptedConn': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:671: undefined reference to `SSL_free' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslErrorMsg': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:71: undefined reference to `ERR_get_error_line' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `createSslFromAcceptedConn': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:623: undefined reference to `SSL_free' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslErrorMsg': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:78: undefined reference to `ERR_error_string' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:71: undefined reference to `ERR_get_error_line' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:78: undefined reference to `ERR_error_string' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:78: undefined reference to `ERR_error_string' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `traceCipherList': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:170: undefined reference to `SSL_get_cipher_list' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:170: undefined reference to `SSL_get_cipher_list' /home/moses/sr
[Moses-support] Failed mosesdecoder compilation with xmlrpc
Hi, I'm trying to compile mosesdecoder with xml-rpc support for mosesserver but without success. I downloaded and installed this library in custom directory and use this command to run compilation process: ./bjam --prefix=/home/moses/src/mosesdecoder -q --with-irstlm=/home/moses/src/irstlm-5.80.08/trunk --with-cmph=/home/moses/src/cmph-2.0 --with-xmlrpc-c=/home/moses/src/mosesdecoder/xmlrpc-c --with=mm --with-probing-pt=/home/moses/src/Pr obingPT --max-kenlm-order=10 -j$(getconf _NPROCESSORS_ONLN) $@ But it produces error and fails compilation (log below). Compilation without xmlrpc support ends with success. Any ideas? Best Regards, Tomek /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `channelRead': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:320: undefined reference to `SSL_read' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:325: undefined reference to `SSL_get_error' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `channelWrite': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:285: undefined reference to `SSL_write' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:290: undefined reference to `SSL_get_error' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `channelDestroy': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:258: undefined reference to `SSL_shutdown' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslCreate': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:121: undefined reference to `SSL_new' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslSetFd': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:142: undefined reference to `SSL_set_fd' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslAccept': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:191: undefined reference to `SSL_accept' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:196: undefined reference to `SSL_get_error' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslErrorMsg': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:71: undefined reference to `ERR_get_error_line' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `getPeerAddrFromSsl': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:415: undefined reference to `SSL_get_fd' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `createChannelFromAcceptedConn': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:671: undefined reference to `SSL_free' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslErrorMsg': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:71: undefined reference to `ERR_get_error_line' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `createSslFromAcceptedConn': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:623: undefined reference to `SSL_free' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `sslErrorMsg': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:78: undefined reference to `ERR_error_string' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:71: undefined reference to `ERR_get_error_line' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:78: undefined reference to `ERR_error_string' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:78: undefined reference to `ERR_error_string' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `traceCipherList': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:170: undefined reference to `SSL_get_cipher_list' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:170: undefined reference to `SSL_get_cipher_list' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `SocketOpenSslInit': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:228: undefined reference to `SSL_load_error_strings' /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:230: undefined reference to `SSL_library_init' /home/moses/src/mosesdecoder/xmlrpc-c/lib/libxmlrpc_abyss.a(socket_openssl.o ): In function `getPeerAddrFromSsl': /home/moses/src/mosesdecoder/xmlrpc-c/lib/abyss/src/socket_openssl.c:415: undefined reference to `SSL_get_fd' /home/moses/src/mosesdeco
[Moses-support] Moses, python_server, matecat_utils, MateCAT - Moses-support Digest, Vol 109, Issue 19
Hi Philipp, Thank you for this solution. I tested python_server and this wrapper works very good and does exactly what I want. Unfortunately it won't help in case of integration with Across Language Server because Across requires direct RPC calls to Moses Server. So, I'm looking for another CAT tool which supports python_server wrapper. It's a fork of matecat_utils package so I suppose deliver tools for Matecat project. If any of you use MateCAT software? I need to confirm that Matecat supports python_server (I cannot check it myself because of Matecat's login requirement by Google authorization - which isn't possible on my virtual machine at this moment). Sorry for my "english" ;) Regards, Tomek -- Message: 1 Date: Thu, 12 Nov 2015 12:36:20 -0500 From: Philipp Koehn Subject: Re: [Moses-support] Moses-support Digest, Vol 109, Issue 19 To: Panos Kanavos Cc: moses-support Message-ID: Content-Type: text/plain; charset="utf-8" Hi, there are a lot of different pre and post processing steps that you may want to apply for any given language pair, so it makes sense to keep them out of the decoder. If you are interested in a server implementation that integrates tokenization, truecasing, etc., check out Christian Buck's implementation of the Google Translate API for Moses: https://github.com/casmacat/moses-mt-server/tree/master/python_server -phi On Thu, Nov 12, 2015 at 9:52 AM, Panos Kanavos wrote: > Thanks for the info Hieu, didn't know that:) I'll try it sometime. > > Best, > > Panos > > > On 12/11/2015 4:41 ??, Hieu Hoang wrote: > > there has been thoughts. There is a c++ tokenizer in >contrib/c++tokenizer > it compiles into a library file, ready for integration. > > The last time i checked, it gave a slightly worse BLEU. Not much, but > consistent. > > If anyone wants to carry on with it, they're welcome to > > Hieu Hoang > http://www.hoang.co.uk/hieu > > On 12 November 2015 at 14:37, Panos Kanavos > wrote: > >> Hi Barry, >> >> Have there ever been any thoughts about implementing >> tokenization/detokenization directly in Moses? I suppose this is some >> work as Moses should become language-aware, but I can only see >> advantanges from this. Besides, Moses is a language tool so these >> concepts shouldn't be so disconnected from its core logic. >> >> Best, >> >> Panos >> >> >> On 12/11/2015 4:02 ??, Barry Haddow wrote: >> > Hi Tomasz >> > >> > The moseserver is just the decoder, so it doesn't do any of the >> > pre- and post-processing steps that you also need. In particular it >> > does not do tokenisation. You need to send it tokenised text, and >> > then de-tokenise the output, >> > >> > cheers - Barry >> > >> > On 12/11/15 13:40, Tomasz Gawryl wrote: >> >> Hi Ulrich, >> >> >> >> I have a question about Moses server too. I'm testing it as a >> >> wrapper >> for >> >> Across server to check pre-translation possibilities. It generally >> works but >> >> there is one problem. Input segments are translated without >> tokenization, so >> >> every word close to special character (for example `this is small >> house.`) >> >> remains untranslated ('to jest ma?y house.'). I was searching list >> archive >> >> and I found similar question here: >> >> http://comments.gmane.org/gmane.comp.nlp.moses.user/14020 but for >> >> me >> it's >> >> not yet answered. I would appreciate any information on this subject. >> >> >> >> Best regards, >> >> Tomek >> >> >> >> -Original Message- >> >> From: moses-support-boun...@mit.edu [mailto: >> moses-support-boun...@mit.edu] >> >> On Behalf Of moses-support-requ...@mit.edu >> >> Sent: Wednesday, November 11, 2015 10:31 AM >> >> To: moses-support@mit.edu >> >> Subject: Moses-support Digest, Vol 109, Issue 19 >> >> >> >> Send Moses-support mailing list submissions to >> >> moses-support@mit.edu >> >> >> >> To subscribe or unsubscribe via the World Wide Web, visit >> >> http://mailman.mit.edu/mailman/listinfo/moses-support >> >> or, via email, send a message with subject or body 'help' to >> >> moses-support-requ...@mit.edu >> >> >> >> You can reach the person managing the list at >&g
Re: [Moses-support] Moses-support Digest, Vol 109, Issue 19
Hi Ulrich, I have a question about Moses server too. I'm testing it as a wrapper for Across server to check pre-translation possibilities. It generally works but there is one problem. Input segments are translated without tokenization, so every word close to special character (for example `this is small house.`) remains untranslated ('to jest mały house.'). I was searching list archive and I found similar question here: http://comments.gmane.org/gmane.comp.nlp.moses.user/14020 but for me it's not yet answered. I would appreciate any information on this subject. Best regards, Tomek -Original Message- From: moses-support-boun...@mit.edu [mailto:moses-support-boun...@mit.edu] On Behalf Of moses-support-requ...@mit.edu Sent: Wednesday, November 11, 2015 10:31 AM To: moses-support@mit.edu Subject: Moses-support Digest, Vol 109, Issue 19 Send Moses-support mailing list submissions to moses-support@mit.edu To subscribe or unsubscribe via the World Wide Web, visit http://mailman.mit.edu/mailman/listinfo/moses-support or, via email, send a message with subject or body 'help' to moses-support-requ...@mit.edu You can reach the person managing the list at moses-support-ow...@mit.edu When replying, please edit your Subject line so it is more specific than "Re: Contents of Moses-support digest..." Today's Topics: 1. Re: use placeholder with mosesserver (Ulrich Germann) 2. Re: use placeholder with mosesserver (Evgeny Matusov) -- Message: 1 Date: Wed, 11 Nov 2015 01:58:40 + From: Ulrich Germann Subject: Re: [Moses-support] use placeholder with mosesserver To: Evgeny Matusov Cc: "moses-support@mit.edu" Message-ID: Content-Type: text/plain; charset="utf-8" Hi all, I've just pushed what I believe might address a few of the issues in this thread: - the more fine-grained configuration options for request handling and queuing, server timeouts etc. (added in August due to threading issue) have been transferred to the main moses executable. - the server now pays attention to the xml-input option specified via json; the range of accepted values is the same as when specified on the command line. I have not written the xml-input handling and do not actively use it, so it may or may not work. I don't think there are any regression tests that test this right now. Reports from the trenches are welcome. - mosesserver.cpp is deprecated. It is now merely a shell around the regular moses call with --server. I did not remove it from the code base entirely, as I assume that there's a plethora of setups out there that rely on the existence of mosesserver. What the wrapper does is add --server to the options and then call run regular moses. - anyone adding stuff to mosesserver.cpp from now on owes me a lifetime supply of the finest Laphroaig. Just send me a quarter cask every year for Burns Nicht for the rest of my life if you do. If I haven't pushed anything for two years, you may assume I'm dead. - Uli On Tue, Nov 10, 2015 at 2:58 PM, Ulrich Germann wrote: > Hi all, > > mosesserver is deprecated and should not be used any more. I'll > transfer the threading-related changes to the server implementation in > the regular moses executable and let you know once I'm done so that > other things (like > passthrough) can be added. By the looks of it, the changes are fairly > straightforward, so it shouldn't take long. However, I can't guarantee > that the new server will do everything the old server did, (or do it > the same way). > > It would be fantastic if a few people could design and contribute test > cases so that we can do some regression testing for the server. > Ideally a test case should provide: > > - tiny models to work with (or we may be able to recycle some that > already > exist) > - sample input (json) > - expected output (json) > > Cheers - Uli > > On Tue, Nov 10, 2015 at 11:37 AM, Evgeny Matusov > wrote: > >> >> Hi, >> >> can any of the more active recent developers advise what is the >> latest stable mosesserver implementation? >> >> It seems to be the one in moses/server, but the one in in >> contrib/server/mosesserver.cpp has been updated in August of this >> year with an important fix related to multiple threads: >> >> >> https://github.com/moses-smt/mosesdecoder/commit/3c682fa8b05af6bff1a0 >> 9f420141795875cf9685 >> https://www.mail-archive.com/moses-support%40mit.edu/msg12875.html >> >> As Gregor mentioned, we would like to share our fix so that >> Mosesserver correctly supports placeholders. I want to make sure that >> this is a fix for something that many people use without problems. >> >> Thanks, >> Evgeny. >> >> >> >> From: moses-support-boun...@mit.edu >> on behalf of moses-support-requ...@mit.edu >> >> Sent: Monday, November 9, 2015 6:02 PM >> To: moses-support@mit.edu >> Subject: Moses-support Digest, Vol 1
[Moses-support] factored tuning time
Hi, I've one question to you about time of factored tuning. How many times longer it takes compared to phrase based tuning? I'm asking because it's 7'th day and it's still tuning (3,3 mln corpus sentences). Phrase based tuning took around 3h for the same corpus. Top shows me that moses uses near 100% CPU. So the speed is the same. Regards, Tomek Gawryl ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] error during tuning step (factored training)
Hi Hieu, Command below (I hope this is it ;). Regards, Tomek moses@SKR-moses:~/working/experiments/FACTORED/steps/1$ more TRAINING_build-ttable.1 #!/bin/bash [...] /home/moses/src/mosesdecoder/scripts/training/train-model.perl -mgiza -mgiza-cpus 8 -cores 8 -parallel -sort-buffer-size 10G -sort-batch-size 253 -sort-compre ss gzip -sort-parallel 10 -dont-zip -first-step 6 -last-step 6 -external-bin-dir /home/moses/src/mosesdecoder/tools -f en -e pl -alignment grow-diag-final-and -max-phrase-length 5 -parts 5 -reordering msd-bidirectional-fe -score-options '--GoodTuring --MinScore 2:0.0001' -parallel -input-factor-max 0 -alignment-fac tors 0-0 -translation-factors 0-0,1 -reordering-factors 0-0 -decoding-steps t0 -extract-file /home/moses/working/experiments/FACTORED/model/extract.1 -lexical -file /home/moses/working/experiments/FACTORED/model/lex.1 -phrase-translation-table /home/moses/working/experiments/FACTORED/model/phrase-table.1.0-0,1 moses@SKR-moses:~/working/experiments/FACTORED/steps/1$ more TRAINING_binarize-config.1 #!/bin/bash [...] mkdir -p /home/moses/working/experiments/FACTORED/model /home/moses/src/mosesdecoder/scripts/training/binarize-model.perl /home/moses/working/experiments/FACTORED/model/moses.ini.1 /home/moses/working/experiments/FACTORED/model/moses.bin.ini.1 -B inarizer "/home/moses/src/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2" -- whats the exact command you used to binarize the phrase-table? On 11/10/2015 17:00, Tomasz Gawryl wrote: > > Hi, > > My factored training ends core dumped during tuning step.: > > /home/moses/working/experiments/FACTORED/lm/ACROSS=pos.lm.1: line 11: > warning: n > > on-zero probability for in closed-vocabulary LM > > terminate called after throwing an instance of 'util::Exception' > > what(): > moses/TranslationModel/RuleTable/PhraseDictionaryOnDisk.cpp:100 in vi > > rtual void Moses::PhraseDictionaryOnDisk::InitializeForInput(const > ttasksptr&) t > > hrew util::Exception because `obj->GetMisc("NumTargetFactors") != > m_output.size( > > )'. > > On-disk phrase table has 1 target factors.. The ini file specified 2 > target factors > > Aborted (core dumped) > > Exit code: 134 > > My config file is similar to example config file for factored trainings. > > What should I do to fix it? > > Regards > > ** > > *Tomek * > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] error during tuning step (factored training)
Hi, My factored training ends core dumped during tuning step.: /home/moses/working/experiments/FACTORED/lm/ACROSS=pos.lm.1: line 11: warning: n on-zero probability for in closed-vocabulary LM terminate called after throwing an instance of 'util::Exception' what(): moses/TranslationModel/RuleTable/PhraseDictionaryOnDisk.cpp:100 in vi rtual void Moses::PhraseDictionaryOnDisk::InitializeForInput(const ttasksptr&) t hrew util::Exception because `obj->GetMisc("NumTargetFactors") != m_output.size( )'. On-disk phrase table has 1 target factors.. The ini file specified 2 target factors Aborted (core dumped) Exit code: 134 My config file is similar to example config file for factored trainings. What should I do to fix it? Regards Tomek ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] experiment.perl, processPhraseTableMin and threads option
Hi Philipp, Shame on me, you are right. Thanx for help! Regards, TG From: phko...@gmail.com [mailto:phko...@gmail.com] On Behalf Of Philipp Koehn Sent: Wednesday, September 23, 2015 3:30 PM To: Tomasz Gawryl Cc: moses-support@mit.edu Subject: Re: [Moses-support] experiment.perl, processPhraseTableMin and threads option Hi, I ran the command as you provided it, and it worked. The "-threads all" functionality was added in August - did you compile the latest version of the code? -phi On Tue, Sep 22, 2015 at 12:16 PM, Tomasz Gawryl wrote: Hi, I'm trying to add this line to config: ttable-binarizer = "$moses-bin-dir/processPhraseTableMin -threads all" This option is supported by processPhraseTableMin command(https://github.com/moses-smt/mosesdecoder/blob/master/misc/processPh raseTableMin.cpp line 24). /home/moses/src/mosesdecoder/scripts/training/binarize-model.perl /home/moses/working/experiments/model/moses.ini.1 /home/mose s/working/experiments/model/moses.bin.ini.3 -Binarizer /home/moses/src/mosesdecoder/bin/processPhraseTableMin -threads all But it produces error "Unknown option: threads" in file TRAINING_binarize-config.3.STDERR (and stops training). I removed this option but it seems that such case it uses only one thread: moses 1470 113 10.8 4255200 3448392 pts/13 Sl 11:05 66:03 /home/moses/src/mosesdecoder/bin/processPhraseTableMin -in /home/moses/working/experiments/model/moses.bin.ini.5.tables/phrase-table.0- 0.1.1.gz.sorted -out /home/moses/working/experiments/model/moses.bin.ini.5.tables/phrase-table.0- 0.1.1 -nscores 4 -threads 1 I know that my server is able to run around 16 threads (and indeed does during former steps). What can I do to improve this step to use more threads? Regards, TG ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] experiment.perl, processPhraseTableMin and threads option
Hi, I'm trying to add this line to config: ttable-binarizer = "$moses-bin-dir/processPhraseTableMin -threads all" This option is supported by processPhraseTableMin command(https://github.com/moses-smt/mosesdecoder/blob/master/misc/processPh raseTableMin.cpp line 24). /home/moses/src/mosesdecoder/scripts/training/binarize-model.perl /home/moses/working/experiments/model/moses.ini.1 /home/mose s/working/experiments/model/moses.bin.ini.3 -Binarizer /home/moses/src/mosesdecoder/bin/processPhraseTableMin -threads all But it produces error "Unknown option: threads" in file TRAINING_binarize-config.3.STDERR (and stops training). I removed this option but it seems that such case it uses only one thread: moses 1470 113 10.8 4255200 3448392 pts/13 Sl 11:05 66:03 /home/moses/src/mosesdecoder/bin/processPhraseTableMin -in /home/moses/working/experiments/model/moses.bin.ini.5.tables/phrase-table.0- 0.1.1.gz.sorted -out /home/moses/working/experiments/model/moses.bin.ini.5.tables/phrase-table.0- 0.1.1 -nscores 4 -threads 1 I know that my server is able to run around 16 threads (and indeed does during former steps). What can I do to improve this step to use more threads? Regards, TG ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] BLEU score
Hi All! This is my first post here and AT first I want to apologize for my English but I would like to ask you some questions. I finished a full phrase based Moses training of EN-PL (English - Polish) corpus (few million sentences from free sources + half million sentences from commercial tmx). Training pipeline always ends with test translation and BLEU score. I didn't expect the first score around 30% but my result 4.5% surprised me. Why my result is so bad? Is it a consequence of chosen language pair? Polish language is very flexible - we can interchange words in a sentence without losing sense. What should I do to improve this result? Or maybe that's all I can get ;). Regards, Tomek ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support