Re: [Moses-support] moses threads compilation problem (with RandLM)
Making RandLM thread-safe is something I've been thinking about. There are a number of bug fixes which need dealing with too, so perhaps at some point I'll push out a new release. Miles 2009/12/17 Alexander Fraser : > Hi Barry and Philipp, > >> Philipp is correct, multi-threaded moses is unlikely to work with randlm >> since >> the latter uses a (presumably) non-thread-safe cache. > > Darn, RandLM would be really useful together with threading because > our cluster has low memory machines with 8 processors. David or Miles, > any chance I could convince you to fix this soon? > >> As regards the compile error, this is due to a recent change in the mbr code, >> and the fact that we don't have a regression test to pick it up. I should be >> able to fix the compile error fairly quickly, but at the moment I'm not sure >> what to do about the regression test. Ideally I'd like to get rid of the >> separate main for mosesmt, although we'd still have to have compiler switches >> which would leave us open to these issues. >> >> If you roll back to 2636 then mosesmt should compile fine. > > It compiles for me without RandLM fine. BTW, it would be cool to stick > something about -threads in the moses usage when compiled with > threads. > >> You mentioned some "weird use of -DWITH_THREADS" - do you mean the failure of >> (eg) the test of Ngram.h ? I think this is due to a different problem with >> configure.ac, which would explain why I keep seeing errors like "WARNING: >> Ngram.h: accepted by the compiler, rejected by the preprocessor!" ! > > > Ngram.h always fails for me (regardless of whether using threads or > not), there is something that causes it to try to invoke a null > command: > > configure:5997: checking Ngram.h presence > configure:6012: -I/home/users6/fraser/statmt/srilm-1.5.7/include > conftest.cpp > ./configure: line 6014: > -I/home/users6/fraser/statmt/srilm-1.5.7/include: No such file or > directory > > > The -DWITH_THREADS thing causes other things to fail (only when > building with threads), such as getopt.h. > > > These failures make no difference, since all of the things that fail > get something like: > > configure:6537: WARNING: getopt.h: accepted by the compiler, rejected > by the preprocessor! > configure:6539: WARNING: getopt.h: proceeding with the compiler's result > > > See the config.log I posted on my previous message (or let me know if > I should send you a copy directly) for more examples. > > Cheers, Alex > > >> >> cheers >> Barry >> >> On Wednesday 16 December 2009 16:28, Alexander Fraser wrote: >>> Hi Barry and other folks, >>> >>> I'm also having trouble compiling Moses with threads and RandLM, there >>> seems to be a bug in MainMT.cpp ? >>> >>> Here is what I am doing: >>> >>> Get fresh copy of Moses (I did this on Monday night). >>> >>> ./regenerate-makefiles.sh >>> ./configure --enable-threads >>> --with-srilm=/home/users6/fraser/statmt/srilm-1.5.7 >>> --with-randlm=/home/users6/fraser/statmt/randlm-v0.11 >>> --with-boost=/home/users6/fraser --with-boost-thread=boost_thread >>> make >>> >>> (The last argument --with-boost-thread is necessary to stop it from >>> picking up the globally installed boost thread library). >>> >>> I attach config.log, which makes it through fine (though I think there >>> is some weird use of -DWITH_THREADS in there which might be >>> interesting). >>> >>> I also attach make.log (which only contains the compilation error, I >>> typed make twice). >>> >>> Let me know if I can provide any more info. >>> >>> Thanks a lot for your help! >>> >>> Cheers, Alex >> >> -- >> The University of Edinburgh is a charitable body, registered in >> Scotland, with registration number SC005336. >> >> > > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] -lmodel-dub parameter
Thank a lot Marcello! --Q On Thu, Dec 17, 2009 at 4:47 PM, Marcello Federico wrote: > Hi, it is just related to the LM penalty assigned to OOV words: > the default value is set to 10^7. So you should not need to > change it. DUB stands for dictionary upper bound. > > marcello > > > > > From: moses-support-boun...@mit.edu [moses-support-boun...@mit.edu] On > Behalf Of Qin Gao [q...@cs.cmu.edu] > Sent: Wednesday, December 09, 2009 10:21 PM > To: moses-support@mit.edu > Subject: [Moses-support] -lmodel-dub parameter > > Hi List, > > Could anyone please explain what is the usage of -lmodel-dub parameter? > Does it related to language model filtering? In the help message it says > "-lmodel-dub parameter" but no furthur information is given e.g. format of > dictionary, how the boundary works etc. > > Thanks a lot! > --Q > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses threads compilation problem (with RandLM)
Hi Alex I've checked in a fix for the compile problem, so it should be ok to use the latest svn version, regards Barry On Thursday 17 Dec 2009 22:17:16 Alexander Fraser wrote: > > It compiles for me without RandLM fine. BTW, it would be cool to stick > > something about -threads in the moses usage when compiled with threads. > > No, I was wrong about that. I ran the wrong binary. It didn't compile, > I'll roll back. Thanks. > > Cheers, Alex > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses threads compilation problem (with RandLM)
> It compiles for me without RandLM fine. BTW, it would be cool to stick > something about -threads in the moses usage when compiled with threads. No, I was wrong about that. I ran the wrong binary. It didn't compile, I'll roll back. Thanks. Cheers, Alex ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses threads compilation problem (with RandLM)
Hi Barry and Philipp, > Philipp is correct, multi-threaded moses is unlikely to work with randlm since > the latter uses a (presumably) non-thread-safe cache. Darn, RandLM would be really useful together with threading because our cluster has low memory machines with 8 processors. David or Miles, any chance I could convince you to fix this soon? > As regards the compile error, this is due to a recent change in the mbr code, > and the fact that we don't have a regression test to pick it up. I should be > able to fix the compile error fairly quickly, but at the moment I'm not sure > what to do about the regression test. Ideally I'd like to get rid of the > separate main for mosesmt, although we'd still have to have compiler switches > which would leave us open to these issues. > > If you roll back to 2636 then mosesmt should compile fine. It compiles for me without RandLM fine. BTW, it would be cool to stick something about -threads in the moses usage when compiled with threads. > You mentioned some "weird use of -DWITH_THREADS" - do you mean the failure of > (eg) the test of Ngram.h ? I think this is due to a different problem with > configure.ac, which would explain why I keep seeing errors like "WARNING: > Ngram.h: accepted by the compiler, rejected by the preprocessor!" ! Ngram.h always fails for me (regardless of whether using threads or not), there is something that causes it to try to invoke a null command: configure:5997: checking Ngram.h presence configure:6012: -I/home/users6/fraser/statmt/srilm-1.5.7/include conftest.cpp ./configure: line 6014: -I/home/users6/fraser/statmt/srilm-1.5.7/include: No such file or directory The -DWITH_THREADS thing causes other things to fail (only when building with threads), such as getopt.h. These failures make no difference, since all of the things that fail get something like: configure:6537: WARNING: getopt.h: accepted by the compiler, rejected by the preprocessor! configure:6539: WARNING: getopt.h: proceeding with the compiler's result See the config.log I posted on my previous message (or let me know if I should send you a copy directly) for more examples. Cheers, Alex > > cheers > Barry > > On Wednesday 16 December 2009 16:28, Alexander Fraser wrote: >> Hi Barry and other folks, >> >> I'm also having trouble compiling Moses with threads and RandLM, there >> seems to be a bug in MainMT.cpp ? >> >> Here is what I am doing: >> >> Get fresh copy of Moses (I did this on Monday night). >> >> ./regenerate-makefiles.sh >> ./configure --enable-threads >> --with-srilm=/home/users6/fraser/statmt/srilm-1.5.7 >> --with-randlm=/home/users6/fraser/statmt/randlm-v0.11 >> --with-boost=/home/users6/fraser --with-boost-thread=boost_thread >> make >> >> (The last argument --with-boost-thread is necessary to stop it from >> picking up the globally installed boost thread library). >> >> I attach config.log, which makes it through fine (though I think there >> is some weird use of -DWITH_THREADS in there which might be >> interesting). >> >> I also attach make.log (which only contains the compilation error, I >> typed make twice). >> >> Let me know if I can provide any more info. >> >> Thanks a lot for your help! >> >> Cheers, Alex > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] -lmodel-dub parameter
Hi, it is just related to the LM penalty assigned to OOV words: the default value is set to 10^7. So you should not need to change it. DUB stands for dictionary upper bound. marcello From: moses-support-boun...@mit.edu [moses-support-boun...@mit.edu] On Behalf Of Qin Gao [q...@cs.cmu.edu] Sent: Wednesday, December 09, 2009 10:21 PM To: moses-support@mit.edu Subject: [Moses-support] -lmodel-dub parameter Hi List, Could anyone please explain what is the usage of -lmodel-dub parameter? Does it related to language model filtering? In the help message it says "-lmodel-dub parameter" but no furthur information is given e.g. format of dictionary, how the boundary works etc. Thanks a lot! --Q ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] moses threads compilation problem (with RandLM)
Hi Alex Philipp is correct, multi-threaded moses is unlikely to work with randlm since the latter uses a (presumably) non-thread-safe cache. As regards the compile error, this is due to a recent change in the mbr code, and the fact that we don't have a regression test to pick it up. I should be able to fix the compile error fairly quickly, but at the moment I'm not sure what to do about the regression test. Ideally I'd like to get rid of the separate main for mosesmt, although we'd still have to have compiler switches which would leave us open to these issues. If you roll back to 2636 then mosesmt should compile fine. You mentioned some "weird use of -DWITH_THREADS" - do you mean the failure of (eg) the test of Ngram.h ? I think this is due to a different problem with configure.ac, which would explain why I keep seeing errors like "WARNING: Ngram.h: accepted by the compiler, rejected by the preprocessor!" ! cheers Barry On Wednesday 16 December 2009 16:28, Alexander Fraser wrote: > Hi Barry and other folks, > > I'm also having trouble compiling Moses with threads and RandLM, there > seems to be a bug in MainMT.cpp ? > > Here is what I am doing: > > Get fresh copy of Moses (I did this on Monday night). > > ./regenerate-makefiles.sh > ./configure --enable-threads > --with-srilm=/home/users6/fraser/statmt/srilm-1.5.7 > --with-randlm=/home/users6/fraser/statmt/randlm-v0.11 > --with-boost=/home/users6/fraser --with-boost-thread=boost_thread > make > > (The last argument --with-boost-thread is necessary to stop it from > picking up the globally installed boost thread library). > > I attach config.log, which makes it through fine (though I think there > is some weird use of -DWITH_THREADS in there which might be > interesting). > > I also attach make.log (which only contains the compilation error, I > typed make twice). > > Let me know if I can provide any more info. > > Thanks a lot for your help! > > Cheers, Alex -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] How do you solve this moses problem? Thanks
Thank you all very much. Just now, I managed this problem. My original running environment is only 4G ram. When I changed to another environment with 16G ram, the running is right. Best wishes; Jun Lang 2009/12/16 Danish Contractor > Also clean the giza2bal.pl files > You can run dos2unix command to get rid of the ^M characters. > > > On Wed, Dec 16, 2009 at 7:46 PM, wrote: > >> Hi, >> >> I met the same problem recently and found the problem caused by perl >> interpreter in the ~~/training/symal/giza2bal.pl which is >> "/usr/bin/perl^M", you can use "s/\s+$//" to clean the ^M in the first line >> and maybe this could help you. >> >> Best wishes >> Gary >> >> >> > Message: 2 >> > Date: Tue, 15 Dec 2009 23:27:16 +0800 >> > From: "Bill_Lang(Gmail)" >> > Subject: [Moses-support] How do you solve this moses problem? Thanks >> > To: moses-support@mit.edu >> > Message-ID: >> > <7bbace5f0912150727j4c8b6702kdd70df01bb8e1...@mail.gmail.com> >> > Content-Type: text/plain; charset="iso-8859-1" >> > >> > Hi friend, >> > I have encountered the same problem as somebody, like in this >> link: >> > http://www.mail-archive.com/moses-support@mit.edu/msg01646.html >> > >> > Before three months, I did the step-by-step guide for moses in >> > http://www.statmt.org/moses_steps.html >> > Then, everything is right. >> > >> > I do not know how to solve this problem. I want to know whether you have >> > solved it. If so, could you please tell me how to manage it? >> > >> > Best wishes; >> > Jun Lang >> > -- next part -- >> > An HTML attachment was scrubbed... >> > URL: >> http://mailman.mit.edu/mailman/private/moses-support/attachments/20091215/8ab261d7/attachment-0001.htm >> > >> > -- >> > >> > Message: 3 >> > Date: Tue, 15 Dec 2009 22:00:47 +0530 >> > From: Danish Contractor >> > Subject: Re: [Moses-support] How do you solve this moses problem? >> > Thanks >> > To: "Bill_Lang(Gmail)" >> > Cc: moses-support@mit.edu >> > Message-ID: >> > <42ba53300912150830t5228ed0ap57e89f18ac060...@mail.gmail.com> >> > Content-Type: text/plain; charset="iso-8859-1" >> > >> > Hi! >> > >> > Yes, I managed to solve it. >> > >> > During one my earlier attempts Giza++ had generated the vocabulary >> > incorrectly for some reason. >> > When the training perl script ran again it did not create the giza++ >> output >> > files again as they existed earlier. >> > This was the problem in my case - the faulty files that were being used. >> > >> > I realised that you should always clean up your work directory if you >> face >> > an error as it can lead to some inconsistencies. >> > Delete all files generated by Giza++/Moses and try again. >> > >> > I also removed some trailing non-significant terms from the vocab file >> > (terms with very low frequency such as numeric values, symbols, >> incorrect >> > spellings etc that creep in as part of the vocabulary). >> > Dont think this would cause the problem as it would affect translations >> more >> > than anything else but I think its a good idea to do so nevertheless. >> > >> > Ensure your training data is clean, sentences are aligned and lower >> cased(if >> > applicable). >> > Hope this helps. >> > >> > Regards, >> > Danish >> > >> > On Tue, Dec 15, 2009 at 8:15 PM, Bill_Lang(Gmail) < >> billlang...@gmail.com>wrote: >> > >> > > Hi friend, >> > > I have encountered the same problem as yours, like in this link: >> > > http://www.mail-archive.com/moses-support@mit.edu/msg01646.html >> > > >> > > Before three months, I did the step-by-step guide for moses in >> > > http://www.statmt.org/moses_steps.html >> > > Then, everything is right. >> > > >> > > I do not know how to solve this problem. I want to know whether you >> have >> > > solved it. If so, could you please tell me how to manage it? >> > > >> > > Best wishes; >> > > Jun Lang >> > > >> > > >> > > >> > -- next part -- >> > An HTML attachment was scrubbed... >> > URL: >> http://mailman.mit.edu/mailman/private/moses-support/attachments/20091215/5a805908/attachment-0001.htm >> > >> > -- >> > >> > ___ >> > Moses-support mailing list >> > Moses-support@mit.edu >> > http://mailman.mit.edu/mailman/listinfo/moses-support >> > >> > >> > End of Moses-support Digest, Vol 38, Issue 17 >> > * >> >> ___ >> Moses-support mailing list >> Moses-support@mit.edu >> http://mailman.mit.edu/mailman/listinfo/moses-support >> > > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
