Re: [Moses-support] TOKENIZER.PERL

2015-02-16 Thread Tom Hoar
I think you'll find what you're looking for in the manual.pdf. Search for "nonbreaking_prefixes" files. You'll also find more information in the existing scripts/share/nonbreaking_prefixes language files. For example, the files' header says: #Anything in this file, followed by a period (and an

[Moses-support] TOKENIZER.PERL

2015-02-16 Thread doc
Hello, I am using the tokenizer.perl script which I found on https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perlI have tried to make it work for Indic languages which use the same punctuation markers with the exception of the full-stop which is a । U+0964 DEVANAGA

Re: [Moses-support] Error in Train Phrase Model

2015-02-16 Thread Barry Haddow
Hi Mohammed I don't think you should reinstall your OS, but I do strongly advise not to run Moses training as root. Not only is it a security risk, but it is quite possible that some of the Moses training components are not working properly as they are not intended to be run as root. Create

Re: [Moses-support] Untuneable feature score components?

2015-02-16 Thread Matthias Huck
Hi, I've added a feature function configuration parameter "tuneable-components", as discussed in previous mails. https://github.com/moses-smt/mosesdecoder/commit/6028c7cf9c256e5df80f71b52d912c47dab31abd Let me know in case you notice any issues related to this. Cheers, Matthias On Fri, 2015-0

Re: [Moses-support] compilation problems

2015-02-16 Thread Matt Post
I did threading=single because multi used to demand the "-mt" variant of the boost libraries, which I've never quite understood. But I just tried again with "threading=multi" and everything works, with that commit below. Thanks again. matt > On Feb 16, 2015, at 1:56 PM, Kenneth Heafield wrote

[Moses-support] Querying IRSTLM

2015-02-16 Thread Abdelfetah Boumerdas
Hello I'm new in the domain and i'm trying to understand how language modeling tools work. So i used IRSTLM, which i trained on the europarl monolingual corpus, and now i'm trying to query it using moses with the command provided in The Moses manual and here i've a couple of questions: 1. is th

Re: [Moses-support] compilation problems

2015-02-16 Thread Kenneth Heafield
Hi, Adding Fabienne because this was the same problem. I've pushed commit 93ab057eda69a7915efbc9fa92d4ce6341e6ca02 which will hopefully handle BOOST_TEST_DYN_LINK correctly. Still unclear what the behavior should be for threading=single. Compile it anyway, forcing two com

Re: [Moses-support] compilation problems

2015-02-16 Thread Kenneth Heafield
Actually, that's by design. Your command line has threading=single and lmplz doesn't have a single-threaded option. Kenneth On 02/16/2015 01:29 PM, Matt Post wrote: > Okay, that worked. The whole project now builds, EXCEPT there is no > bin/lmplz. fragment, build_binary, and query all exist, bu

Re: [Moses-support] compilation problems

2015-02-16 Thread Matt Post
Okay, that worked. The whole project now builds, EXCEPT there is no bin/lmplz. fragment, build_binary, and query all exist, but not lmplz. It is not mentioned in the logs... But that's okay, because I'll just copy it from KenLM directly. Thanks for your help. matt > On Feb 16, 2015, at 1:01

Re: [Moses-support] compilation problems

2015-02-16 Thread Kenneth Heafield
Tests to be dynamically linked must be _compiled_ using -DBOOST_TEST_DYN_LINK . The build system did this prior to Hieu's change. After reverting Hieu's change, force it to recompile the object file: rm util/bin/gcc-4.9.2/release/file_piece_test.o (or just run with -a and rebuild everything).

Re: [Moses-support] compilation problems

2015-02-16 Thread Matt Post
Hmm; I got a bit further, but still have linking errors for the tests. build_binary built, but still no lmplz, and it's not mentioned at all in the log output. I was able to get lmplz to build by modifying Jamroot to build only the "lm" project, but still get all these linking errors when I try

Re: [Moses-support] compilation problems

2015-02-16 Thread Hieu Hoang
can't the unit testing only be run when the regression tests are run? Users don't need to know about the restults of the unit tests, and that's the thing thats causing problems Hieu Hoang Research Associate (until March 2015) ** searching for interesting commercial MT position ** University of Edi

Re: [Moses-support] compilation problems

2015-02-16 Thread Kenneth Heafield
Hi Matt, lmplz should be compiling anyway, despite the tests failing. Try reverting this commit, which broke shared compilation for tests: commit d7f5bb41faaac5ca93b9cbb723ad558b2c67d3c2 Author: Hieu Hoang Date: Tue Jan 27 16:22:15 2015 + Regarding boost_filesystem we'll probably

[Moses-support] compilation problems

2015-02-16 Thread Matt Post
Hi,I am running into a number of problems compiling Moses 3.0. I am using GCC 4.9.2 and a custom (correct) install of Boost 1.57.0.1. First, I had to add this: boost_filesystemto line 174 of Jamroot (per this discussion: https://github.com/moses-smt/mosesdecoder/issues/89 )2. Things like lmplz do n

Re: [Moses-support] Error in Train Phrase Model

2015-02-16 Thread Tom Hoar
As Barry noted, your logs show the command prompt path is "/root" which means you're probably running as root. Several years ago, we found are several ill-behaved utilities (in Moses and IRSTLM) that relied on access privileges to prevent deleting system files. Running as root bypassed this saf

[Moses-support] Error while building Moses

2015-02-16 Thread Saumitra Yadav
Sir, i was compiling moses and ran following command *sudo ./bjam* can you please advise how to go about this confirm ea22b4a02880e9a79148193ad77d58b9edf0a8fd Regards, Saumitra Yadav M.Tech. Department Of Computer Science And Technology Goa University build.log.gz Description: GNU Zip compre

Re: [Moses-support] Error in Train Phrase Model

2015-02-16 Thread Hieu Hoang
if you don't find /dev/stdin, then something is wrong with your OS. You must re-install your operating system. you may have deleted /dev/stdin by accident, or a script has deleted it. It is highly recommended not to use the root user, running as an ordinary user will prevent this. On 16/02/1

Re: [Moses-support] Error in Train Phrase Model

2015-02-16 Thread mohamed hasanien
also when i dir dev i didn't find  stdin mohammed hassanien Mohammed Egyption Programmers Vice-captain 01000121556 Egyption Programmers Syndicate On Monday, February 16, 2015 12:39 PM, mohamed hasanien wrote: hi barry,  the out put of the cmd is ls: cannot access /dev/stdin: No su

Re: [Moses-support] Error in Train Phrase Model

2015-02-16 Thread mohamed hasanien
hi barry,  the out put of the cmd is ls: cannot access /dev/stdin: No such file or directory  i have only this user in the system mohammed hassanien Mohammed Egyption Programmers Vice-captain 01000121556 Egyption Programmers Syndicate On Monday, February 16, 2015 11:06 AM, Barry Haddow

Re: [Moses-support] Error in Train Phrase Model

2015-02-16 Thread Barry Haddow
Hi Mohammed Sorry, I meant "ls -l /dev/stdin" I would suggest you log in and run train-model.perl as a user other than root. cheers - Barry On 15/02/15 20:17, mohamed hasanien wrote: > Hi Barr, > this is the out put > from the command you told me and also from nohup.out > ls -l /dev/stdout >