[Moses-support] Problems with moses compilation
Dear Moses supports, I am running into an error when compiling Moses ( https://github.com/moses-smt/mosesdecoder/commit/23556697c4e6831bfd0574351717f5a7b79dff77) with the options threading=single and --static. (x86_64-redhat-linux) The command I used is: ./bjam threading=single --static --prefix=$MOSES_INSTALLDIR --install-scripts=$MOSES_SCRIPTS --with-srilm=$SRILM --with-irstlm=$IRSTLMDIR --with-boost=$BOOSTDIR --with-xmlrpc-c=$XMLRPCDIR --with-cmph=$CMPHDIR --with-giza=$MOSES_INSTALLDIR/bin --max-kenlm-order=5 --with-tcmalloc --without-libsegfault -j8 -a -q It was a linking error with pcqueue_test. The partial error message is attached. If I remove threading=single, the compilation finishes without problems. It seems -lpthread could help with this step, but I don't know how to add such an option to the bjam command. (Adding linkflags=-lpthread alone seems to cause the compilation to freeze.) It would be great if someone could tell me how to deal with it. Thank you very much for the help in advance! Best regards, Yun gcc.link util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/pcqueue_test g++ -Llib -Llib64 -Wl,-R -Wl,lib -Wl,-R -Wl,lib64 -Wl,-rpath-link -Wl,lib -Wl,-rpath-link -Wl,lib64 -o util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/pcqueue_test -Wl,--start-group util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/pcqueue_test.o util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/read_compressed.o util/double-conversion/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/bignum-dtoa.o util/double-conversion/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/bignum.o util/double-conversion/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/cached-powers.o util/double-conversion/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/diy-fp.o util/double-conversion/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/double-conversion.o util/double-conversion/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/fast-dtoa.o util/double-conversion/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/fixed-dtoa.o util/double-conversion/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/strtod.o util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/bit_packing.o util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/ersatz_progress.o util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/exception.o util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/file.o util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/file_piece.o util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/mmap.o util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/murmur_hash.o util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/pool.o util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/scoped.o util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/string_piece.o util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/usage.o -lboost_system -lboost_unit_test_framework -lbz2 -lz -lrt -ldl -lboost_system -Wl,--end-group -g -static util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/pcqueue_test.o: In function `semaphore_init': include/boost/interprocess/sync/posix/semaphore_wrapper.hpp:115: undefined reference to `sem_init' include/boost/interprocess/sync/posix/semaphore_wrapper.hpp:115: undefined reference to `sem_init' util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/pcqueue_test.o: In function `mutex': include/boost/thread/pthread/mutex.hpp:39: undefined reference to `pthread_mutex_init' include/boost/thread/pthread/mutex.hpp:39: undefined reference to `pthread_mutex_init' util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/pcqueue_test.o: In function `semaphore_wait': include/boost/interprocess/sync/posix/semaphore_wrapper.hpp:142: undefined reference to `sem_wait' util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/pcqueue_test.o: In function `~mutex': include/boost/thread/pthread/mutex.hpp:47: undefined reference to `pthread_mutex_destroy' util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/pcqueue_test.o: In function `semaphore_destroy': include/boost/interprocess/sync/posix/semaphore_wrapper.hpp:126: undefined reference to `sem_destroy' util/bin/gcc-4.1.2/release/debug-symbols-on/link-static/runtime-link-static/pcqueue_test.o: In function `semaphore_destroy': util/pcqueue_test.cc:16: undefined reference to `sem_destroy'
Re: [Moses-support] Help. First request to MosesServer very slow
HI Marcos For command-line Moses, it has to output the sentences in the same order as they are input. So if, say, you have 4 threads and provide Moses with a large file of input sentences, it will give 1 sentence to each of the 4 threads. But no output will be produced until sentence 1 is translated, so if all sentences are about the same length then there will be a delay for the first one, then all others will then be output very quickly. The MTMonkey paper would have used an older version of Moses, which had a single global translation options cache, and this could explain the difference between your observed behaviour and their reported performance. The change in the cache happened some time in 2013, and unfortunately seems to have had a negative impact on Moses server. As Kenneth has pointed out though, it is something that needs to be fixed in Moses server. cheers - Barry On 10/03/14 09:22, Marcos Fernandez wrote: Hi Marcos I think the problem is that the rules (or phrase pairs) are now cached on a per thread basis. This is good for command-line Moses as it uses a pool of threads, and having per-thread caches means that there is no locking on the caches, as there used to be. Barry, I am not sure that that is the cause, because in that case my 20 short sentences file would get translated much faster with command-line Moses, however it takes 4.5 seconds to translate it. This is, 2 seconds (measured) for loading tables in memory, and the same 2.5 seconds than with MosesServer to translate the file. And the behaviour is also the same as before, the first sentence in the file takes much longer than the rest. However, what you say perhaps could be the cause of the difference in time between using a different xmlrpc ServerProxy object for each request (probably, in this case xmlrpc executes each request in a different thread), or reusing one only ServerProxy for all the requests (where there would be only one thread, so it could take advantage of the cache). What I understand then, is that the cache stores information of the previously translated sentences to accelerate the translation of the next ones. But that does not eliminate the problem of the first slower request. As you can see, I am finding that issue even in command-line Moses (not with the first request, but with the first sentence in a file). I am thinking that perhaps I have no problem, and this is just the usual way in which Moses works. Just to make sure: 1. would you say that a time of 2-3 seconds for translating (spa-eng) a single sentence (~15-20 words) could be a normal response time (discounting the time of loading tables)? (Intel Xeon with 32GB RAM) 2. if now you write 3 similar sentences in a file, and execute Moses (command-line, serial) over this file, would you expect it to take a much shorter time (perhaps the half) than the sum of the times for the 3 single sentences? If the answer is yes to both (specially the second), then I am probably worrying in vain. My worries started when I read the MTMonkey paper: http://ufal.mff.cuni.cz/~pecina/files/pbml-2013.pdf Here the authors use the approach of creating a new ServerProxy instance each time a request is sent to MosesServer (the worst case scenario for me), and they get great results, so I thought they were not experiencing that overhead for every request. But perhaps they just used sentences that get translated very fast even with that overhead. Well, in the case that this is the usual way of working for Moses, if what Kenneth suggests is possible, that would eliminate the overhead almost completely for MosesServer. I mean, there would still be an overhead for the first request that each thread serves after the server is just launched, but never more, because the caches would be filled with useful information from that point on. I think that this would be extremely interesting for web translating services, or to do web-page translations on the fly. However, I don't see a way to avoid that overhead in command-line Moses, as it dies after each execution. Marcos. mosesserver, afaik, creates a new thread for each connection, so it can't take advantage of the cache. This is done in the xmlrpc-c library so we don't have much control over it. If you dig around in the xmlrpc-c documentation (or code!) you might find a way to control the threading policy. I just spoke to Marcin about the problem, and we're not sure if loading the compact phrase table into memory would help, as you still would need the higher level cache (in PhraseDictionary). But you could try this anyway. cheers - Barry On 06/03/14 17:20, Marcos Fernandez wrote: Hi, I am having an issue with MosesServer. I am using compact phrase and reordering table, and KENLM. The problem is this (I'll explain with an example): - I have one file with 20 very short sentences. I split and tokenize them and send one XMLPRC request per sentence to
[Moses-support] manual: 3.2.1 Train an unfactored model
Hi, From the manual: http://www.statmt.org/moses/manual/manual.pdf I have run the example from part 3.2.1, page 64, to create unfactored/model/moses.ini At the bottom of page 67 is this example: moses -f unfactored/model/moses.ini in -d 0.2 The file 'in' contains: menschen beschreibt putin . (The -d 0.2 causes an error in all examples, so I leave that out) I get an error: Exception: moses/Word.cpp:112 in void Moses::Word::CreateFromString(Moses::FactorDirection, const std::vectorlong unsigned int, const StringPiece, bool) threw StrayFactorException because `fit'. You have configured 1 factors but the word !|!|. contains factor delimiter | too many times. What is wrong, and how do I fix this? All other examples in section 3.2 work OK. I have moses built with both IRSTLM and SRILM. -- Peter Kleiweg http://pkleiweg.home.xs4all.nl/ ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Problems with moses compilation
Hi, Please pull and try again. I've set pcqueue_test so it won't be compiled as part of the single-threaded build. Also fixed rt linkage. Kenneth On 03/11/14 01:25, Yun Chung wrote: Dear Moses supports, I am running into an error when compiling Moses (https://github.com/moses-smt/mosesdecoder/commit/23556697c4e6831bfd0574351717f5a7b79dff77) with the options threading=single and --static. (x86_64-redhat-linux) The command I used is: ./bjam threading=single --static --prefix=$MOSES_INSTALLDIR --install-scripts=$MOSES_SCRIPTS --with-srilm=$SRILM --with-irstlm=$IRSTLMDIR --with-boost=$BOOSTDIR --with-xmlrpc-c=$XMLRPCDIR --with-cmph=$CMPHDIR --with-giza=$MOSES_INSTALLDIR/bin --max-kenlm-order=5 --with-tcmalloc --without-libsegfault -j8 -a -q It was a linking error with pcqueue_test. The partial error message is attached. If I remove threading=single, the compilation finishes without problems. It seems -lpthread could help with this step, but I don't know how to add such an option to the bjam command. (Adding linkflags=-lpthread alone seems to cause the compilation to freeze.) It would be great if someone could tell me how to deal with it. Thank you very much for the help in advance! Best regards, Yun ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] manual: 3.2.1 Train an unfactored model
Hi Peter, There is obviously an issue with your training data, have you used the proj-syndicate corpora? if so please paste here your training command I'd be interested to see that. Regards M 2014-03-11 19:25 GMT+01:00 Peter Kleiweg p.c.j.klei...@rug.nl: Hi, From the manual: http://www.statmt.org/moses/manual/manual.pdf I have run the example from part 3.2.1, page 64, to create unfactored/model/moses.ini At the bottom of page 67 is this example: moses -f unfactored/model/moses.ini in -d 0.2 The file 'in' contains: menschen beschreibt putin . (The -d 0.2 causes an error in all examples, so I leave that out) I get an error: Exception: moses/Word.cpp:112 in void Moses::Word::CreateFromString(Moses::FactorDirection, const std::vectorlong unsigned int, const StringPiece, bool) threw StrayFactorException because `fit'. You have configured 1 factors but the word !|!|. contains factor delimiter | too many times. What is wrong, and how do I fix this? All other examples in section 3.2 work OK. I have moses built with both IRSTLM and SRILM. -- Peter Kleiweg http://pkleiweg.home.xs4all.nl/ ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* ü Pensez à l'environnement, n'imprimez ce courriel que si nécessaire. Please do not print this email unless it is absolutely necessary. Spread environmental awareness. inline: image001.jpg___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] train-model log file?
Hi, does the train-model.perl script (or more specifically, mgiza) create a log file? thanks, ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] manual: 3.2.1 Train an unfactored model
Massinissa Ahmim schreef op de 11e dag van de lentemaand van het jaar 2014: Hi Peter, There is obviously an issue with your training data, have you used the proj-syndicate corpora? if so please paste here your training command I'd be interested to see that. I have data downloaded from here: http://www.statmt.org/moses/download/factored-corpus.tgz This is the training command: train-model.perl \ --root-dir unfactored \ --corpus factored-corpus/proj-syndicate \ --f de --e en \ --lm 0:3:/net/aistaff/kleiweg/moses/test/factored-corpus/surface.lm:0 \ --external-bin-dir /net/aps/64/opt/moses/giza-pp/bin \ training.out \ 2 training.err Regards M 2014-03-11 19:25 GMT+01:00 Peter Kleiweg p.c.j.klei...@rug.nl: Hi, From the manual: http://www.statmt.org/moses/manual/manual.pdf I have run the example from part 3.2.1, page 64, to create unfactored/model/moses.ini At the bottom of page 67 is this example: moses -f unfactored/model/moses.ini in -d 0.2 The file 'in' contains: menschen beschreibt putin . (The -d 0.2 causes an error in all examples, so I leave that out) I get an error: Exception: moses/Word.cpp:112 in void Moses::Word::CreateFromString(Moses::FactorDirection, const std::vectorlong unsigned int, const StringPiece, bool) threw StrayFactorException because `fit'. You have configured 1 factors but the word !|!|. contains factor delimiter | too many times. What is wrong, and how do I fix this? All other examples in section 3.2 work OK. I have moses built with both IRSTLM and SRILM. -- Peter Kleiweg http://pkleiweg.home.xs4all.nl/ ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] train-model log file?
Hi Victor Yes, all you have to do is to add training.out after to your training command Regards Massinissa 2014-03-11 19:39 GMT+01:00 Viktor Pless viktor.pl...@gmail.com: Hi, does the train-model.perl script (or more specifically, mgiza) create a log file? thanks, ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* ü Pensez à l'environnement, n'imprimez ce courriel que si nécessaire. Please do not print this email unless it is absolutely necessary. Spread environmental awareness. inline: image001.jpg___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] manual: 3.2.1 Train an unfactored model
Shouldn't the moses option come before the redirection like this? moses -d 0.2 -f unfactored/model/moses.ini in On 03/12/2014 01:46 AM, Peter Kleiweg wrote: moses -f unfactored/model/moses.ini in -d 0.2 ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Problem while running mert-moses.pl
Dear all, I try to run tuning using this command. nohup nice /home/wei_lu/tools/mosesdecoder/scripts/training/mert-moses.pl \ /home/wei_lu/experiment/data/dev/dev2010.lowercased.indexed.zh /home/wei_lu/experiment/data/dev/dev2010.lowercased.en \ /home/wei_lu/tools/mosesdecoder/bin/moses /home/wei_lu/pku_seg/model/moses.ini --working-dir /home/wei_lu/pku_seg/mert-work-index --mertdir /home/wei_lu/tools/mosesdecoder/bin/ -decoder-flags=-threads 4 But the result I got fail, here is the log. *Using SCRIPTS_ROOTDIR: /home/wei_lu/tools/mosesdecoder/scripts* *filtering the phrase tables... Wed Mar 12 12:18:43 SGT 2014* *exec: /home/wei_lu/tools/mosesdecoder/scripts/training/filter-model-given-input.pl http://filter-model-given-input.pl ./filtered /home/wei_lu/pku_seg/model/moses.ini /home/wei_lu/experiment/data/dev/dev2010.lowercased.indexed.zh* *Executing: /home/wei_lu/tools/mosesdecoder/scripts/training/filter-model-given-input.pl http://filter-model-given-input.pl ./filtered /home/wei_lu/pku_seg/model/moses.ini /home/wei_lu/experiment/data/dev/dev2010.lowercased.indexed.zh filterphrases.out 2 filterphrases.err* *Asking moses for feature names and values from filtered/moses.ini* *Executing: /home/wei_lu/tools/mosesdecoder/bin/moses -threads 4 -config filtered/moses.ini -inputtype 0 -show-weights ./features.list* *Defined parameters (per moses.ini or switch):* * config: filtered/moses.ini * * distortion-limit: 6 * * feature: UnknownWordPenalty WordPenalty PhrasePenalty PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=6 path=/home/wei_lu/pku_seg/mert-work-index/filtered/phrase-table.0-0.1.1.gz input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/wei_lu/pku_seg/mert-work-index/filtered/reordering-table.wbe-msd-bidirectional-fe Distortion KENLM lazyken=0 name=LM0 factor=0 path=/home/wei_lu/experiment/lm/allan_lm.lm order=3 * * input-factors: 0 * * inputtype: 0 * * mapping: 0 T 0 * * show-weights: * * threads: 4 * * weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 TranslationModel0= 0.2 0.2 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5 * */home/wei_lu/tools/mosesdecoder/bin* *line=UnknownWordPenalty* *FeatureFunction: UnknownWordPenalty0 start: 0 end: 0* *line=WordPenalty* *FeatureFunction: WordPenalty0 start: 1 end: 1* *line=PhrasePenalty* *FeatureFunction: PhrasePenalty0 start: 2 end: 2* *line=PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=6 path=/home/wei_lu/pku_seg/mert-work-index/filtered/phrase-table.0-0.1.1.gz input-factor=0 output-factor=0* *FeatureFunction: TranslationModel0 start: 3 end: 8* *line=LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/wei_lu/pku_seg/mert-work-index/filtered/reordering-table.wbe-msd-bidirectional-fe* *FeatureFunction: LexicalReordering0 start: 9 end: 14* *Initializing LexicalReordering..* *line=Distortion* *FeatureFunction: Distortion0 start: 15 end: 15* *line=KENLM lazyken=0 name=LM0 factor=0 path=/home/wei_lu/experiment/lm/allan_lm.lm order=3* *FeatureFunction: LM0 start: 16 end: 16* *Loading the LM will be faster if you build a binary file.* *Reading /home/wei_lu/experiment/lm/allan_lm.lm* *5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100* ** *Loading table into memory...done.* *Start loading text SCFG phrase table. Moses format : [1507.000] seconds* *Reading /home/wei_lu/pku_seg/mert-work-index/filtered/phrase-table.0-0.1.1.gz* *5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100* *sh: line 1: 49997 Killed /home/wei_lu/tools/mosesdecoder/bin/moses -threads 4 -config filtered/moses.ini -inputtype 0 -show-weights ./features.list* *Exit code: 137* *Failed to run moses with the config filtered/moses.ini at /home/wei_lu/tools/mosesdecoder/scripts/training/mert-moses.pl http://mert-moses.pl line 1271*. I have no idea why this happened. And then I try to run a small dataset(only one sentence in the tuning instance and relevant phrase-table), that works fine. I don't know what's this problem. Best Regards, Allan -- Research student, final-year undergraduate Extreme Scale Network Computing and service laboratory( http://www.cloud-uestc.cn/cloud-uestc-EN/index.html), School of Computer Science Engineering, University of Electronic Science and Technology of China( http://www.oice.uestc.edu.cn/en/). Website: http://www.allanjie.ml/ ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
