subject:"\[Moses\-support\] help"

Re: [Moses-support] help

2020-04-16 Thread Hieu Hoang

my advise though - use Linux instead of MacOSX.

The scripts are written for Linux and rarely tested on Mac, you're on you
own if there's a problem

Hieu Hoang
http://statmt.org/hieu


On Thu, 16 Apr 2020 at 09:55, Hieu Hoang  wrote:

> inmert/Jamfile, delete all the unit tests. if necessary, delete all
> unit tests in all Jamfiles you find
>
> Hieu Hoang
> http://statmt.org/hieu
>
>
> On Thu, 16 Apr 2020 at 09:51, Bounyalit Taymany 
> wrote:
>
>> Excuse me when I use the command " find . -name "*Test.cpp" | xargs rm "
>> and I try running bjam again. But It is like this
>>
>> $ ./bjam --with-boost=~/demo/boost_1_72_0 -j4
>>
>> XMLRPC-C: USING VERSION 1.39.12 FROM /usr/local/Cellar/xmlrpc-c/1.39.12
>>
>> fatal: No names found, cannot describe anything.
>>
>> Tip: install tcmalloc for faster threading.  See BUILD-INSTRUCTIONS.txt
>> for more information.
>>
>> BUILDING MOSES SERVER!
>>
>> Performing configuration checks
>>
>>
>> - Shared Boost : no
>>
>> - Static Boost : no
>>
>> error: Unable to find file or target named
>>
>> error: 'ScoreFeatureTest.cpp'
>>
>> error: referred from project at
>>
>> error: 'phrase-extract'
>>
>>
>> On Thu, 16 Apr 2020 at 12:25, Hieu Hoang  wrote:
>>
>>> Try deleting all the unit test file before running the bjam command
>>> again. You can delete the unit test files with the following command
>>>find . -name "*Test.cpp" | xargs rm
>>>
>>> Moses used to compile fine with macOS but macOS's compiler and gcc are
>>> slowly diverging.
>>>
>>> Hieu Hoang
>>> http://statmt.org/hieu
>>>
>>>
>>> On Wed, 15 Apr 2020 at 21:21, Bounyalit Taymany 
>>> wrote:
>>>
 I used command for install Moses "./bjam
 --with-boost=~/workspace/temp/boost_1_64_0 -j4"
 on MacOS Catalina 10.15
 but it failed

 ...failed darwin.compile.c++
 mert/bin/darwin-11.0.3/release/link-static/threading-multi/UtilTest.o...

 ...skipped
 util_test for
 lack of
 UtilTest.o...

 ...skipped
 util_test.passed
 for lack of
 util_test...

 darwin.compile.c++
 mert/bin/darwin-11.0.3/release/link-static/threading-multi/VocabularyTest.o

 clang: error: unknown argument: '-fcoalesce-templates'


 "g++"  -ftemplate-depth-128 -fcoalesce-templates -O3
 -finline-functions -Wno-inline -Wall -gdwarf-2 -fexceptions
 -Wno-long-double -I/usr/local/Cellar/xmlrpc-c/1.39.12/include  -std=c++0x
 -DBOOST_FILE_SYSTEM_DYN_LINK -DBOOST_PROGRAM_OPTIONS_DYN_LINK
 -DBOOST_SYSTEM_DYN_LINK -DBOOST_TEST_DYN_LINK -DBOOST_THREAD_DYN_DLL
 -DHAVE_XMLRPC_C -DKENLM_MAX_ORDER=6 -DMOSES_VERSION_ID=\"\" -DNDEBUG
 -DTRACE_ENABLE=1 -DWITH_THREADS -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES  
 -I"."
 -I"util/double-conversion" -I"~/demo/boost_1_72_0/include" -c -o
 "mert/bin/darwin-11.0.3/release/link-static/threading-multi/VocabularyTest.o"
 "mert/VocabularyTest.cpp"


 ...failed darwin.compile.c++
 mert/bin/darwin-11.0.3/release/link-static/threading-multi/VocabularyTest.o...

 ...skipped
 vocabulary_test
 for lack of
 VocabularyTest.o...

 ...skipped
 vocabulary_test.passed
 for lack of
 vocabulary_test...

 ...failed updating 762 targets...

 ...skipped 219 targets...
 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support

>>>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help

2020-04-16 Thread Hieu Hoang

inmert/Jamfile, delete all the unit tests. if necessary, delete all
unit tests in all Jamfiles you find

Hieu Hoang
http://statmt.org/hieu


On Thu, 16 Apr 2020 at 09:51, Bounyalit Taymany  wrote:

> Excuse me when I use the command " find . -name "*Test.cpp" | xargs rm "
> and I try running bjam again. But It is like this
>
> $ ./bjam --with-boost=~/demo/boost_1_72_0 -j4
>
> XMLRPC-C: USING VERSION 1.39.12 FROM /usr/local/Cellar/xmlrpc-c/1.39.12
>
> fatal: No names found, cannot describe anything.
>
> Tip: install tcmalloc for faster threading.  See BUILD-INSTRUCTIONS.txt
> for more information.
>
> BUILDING MOSES SERVER!
>
> Performing configuration checks
>
>
> - Shared Boost : no
>
> - Static Boost : no
>
> error: Unable to find file or target named
>
> error: 'ScoreFeatureTest.cpp'
>
> error: referred from project at
>
> error: 'phrase-extract'
>
>
> On Thu, 16 Apr 2020 at 12:25, Hieu Hoang  wrote:
>
>> Try deleting all the unit test file before running the bjam command
>> again. You can delete the unit test files with the following command
>>find . -name "*Test.cpp" | xargs rm
>>
>> Moses used to compile fine with macOS but macOS's compiler and gcc are
>> slowly diverging.
>>
>> Hieu Hoang
>> http://statmt.org/hieu
>>
>>
>> On Wed, 15 Apr 2020 at 21:21, Bounyalit Taymany 
>> wrote:
>>
>>> I used command for install Moses "./bjam
>>> --with-boost=~/workspace/temp/boost_1_64_0 -j4"
>>> on MacOS Catalina 10.15
>>> but it failed
>>>
>>> ...failed darwin.compile.c++
>>> mert/bin/darwin-11.0.3/release/link-static/threading-multi/UtilTest.o...
>>>
>>> ...skipped
>>> util_test for
>>> lack of
>>> UtilTest.o...
>>>
>>> ...skipped
>>> util_test.passed
>>> for lack of
>>> util_test...
>>>
>>> darwin.compile.c++
>>> mert/bin/darwin-11.0.3/release/link-static/threading-multi/VocabularyTest.o
>>>
>>> clang: error: unknown argument: '-fcoalesce-templates'
>>>
>>>
>>> "g++"  -ftemplate-depth-128 -fcoalesce-templates -O3
>>> -finline-functions -Wno-inline -Wall -gdwarf-2 -fexceptions
>>> -Wno-long-double -I/usr/local/Cellar/xmlrpc-c/1.39.12/include  -std=c++0x
>>> -DBOOST_FILE_SYSTEM_DYN_LINK -DBOOST_PROGRAM_OPTIONS_DYN_LINK
>>> -DBOOST_SYSTEM_DYN_LINK -DBOOST_TEST_DYN_LINK -DBOOST_THREAD_DYN_DLL
>>> -DHAVE_XMLRPC_C -DKENLM_MAX_ORDER=6 -DMOSES_VERSION_ID=\"\" -DNDEBUG
>>> -DTRACE_ENABLE=1 -DWITH_THREADS -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES  -I"."
>>> -I"util/double-conversion" -I"~/demo/boost_1_72_0/include" -c -o
>>> "mert/bin/darwin-11.0.3/release/link-static/threading-multi/VocabularyTest.o"
>>> "mert/VocabularyTest.cpp"
>>>
>>>
>>> ...failed darwin.compile.c++
>>> mert/bin/darwin-11.0.3/release/link-static/threading-multi/VocabularyTest.o...
>>>
>>> ...skipped
>>> vocabulary_test
>>> for lack of
>>> VocabularyTest.o...
>>>
>>> ...skipped
>>> vocabulary_test.passed
>>> for lack of
>>> vocabulary_test...
>>>
>>> ...failed updating 762 targets...
>>>
>>> ...skipped 219 targets...
>>> ___
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help

2020-04-15 Thread Hieu Hoang

Try deleting all the unit test file before running the bjam command again.
You can delete the unit test files with the following command
   find . -name "*Test.cpp" | xargs rm

Moses used to compile fine with macOS but macOS's compiler and gcc are
slowly diverging.

Hieu Hoang
http://statmt.org/hieu


On Wed, 15 Apr 2020 at 21:21, Bounyalit Taymany  wrote:

> I used command for install Moses "./bjam
> --with-boost=~/workspace/temp/boost_1_64_0 -j4"
> on MacOS Catalina 10.15
> but it failed
>
> ...failed darwin.compile.c++
> mert/bin/darwin-11.0.3/release/link-static/threading-multi/UtilTest.o...
>
> ...skipped
> util_test for
> lack of
> UtilTest.o...
>
> ...skipped
> util_test.passed
> for lack of
> util_test...
>
> darwin.compile.c++
> mert/bin/darwin-11.0.3/release/link-static/threading-multi/VocabularyTest.o
>
> clang: error: unknown argument: '-fcoalesce-templates'
>
>
> "g++"  -ftemplate-depth-128 -fcoalesce-templates -O3
> -finline-functions -Wno-inline -Wall -gdwarf-2 -fexceptions
> -Wno-long-double -I/usr/local/Cellar/xmlrpc-c/1.39.12/include  -std=c++0x
> -DBOOST_FILE_SYSTEM_DYN_LINK -DBOOST_PROGRAM_OPTIONS_DYN_LINK
> -DBOOST_SYSTEM_DYN_LINK -DBOOST_TEST_DYN_LINK -DBOOST_THREAD_DYN_DLL
> -DHAVE_XMLRPC_C -DKENLM_MAX_ORDER=6 -DMOSES_VERSION_ID=\"\" -DNDEBUG
> -DTRACE_ENABLE=1 -DWITH_THREADS -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES  -I"."
> -I"util/double-conversion" -I"~/demo/boost_1_72_0/include" -c -o
> "mert/bin/darwin-11.0.3/release/link-static/threading-multi/VocabularyTest.o"
> "mert/VocabularyTest.cpp"
>
>
> ...failed darwin.compile.c++
> mert/bin/darwin-11.0.3/release/link-static/threading-multi/VocabularyTest.o...
>
> ...skipped
> vocabulary_test
> for lack of
> VocabularyTest.o...
>
> ...skipped
> vocabulary_test.passed
> for lack of
> vocabulary_test...
>
> ...failed updating 762 targets...
>
> ...skipped 219 targets...
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] help

2020-04-15 Thread Bounyalit Taymany

I used command for install Moses "./bjam
--with-boost=~/workspace/temp/boost_1_64_0 -j4"
on MacOS Catalina 10.15
but it failed

...failed darwin.compile.c++
mert/bin/darwin-11.0.3/release/link-static/threading-multi/UtilTest.o...

...skipped
util_test for
lack of
UtilTest.o...

...skipped
util_test.passed
for lack of
util_test...

darwin.compile.c++
mert/bin/darwin-11.0.3/release/link-static/threading-multi/VocabularyTest.o

clang: error: unknown argument: '-fcoalesce-templates'


"g++"  -ftemplate-depth-128 -fcoalesce-templates -O3 -finline-functions
-Wno-inline -Wall -gdwarf-2 -fexceptions -Wno-long-double
-I/usr/local/Cellar/xmlrpc-c/1.39.12/include  -std=c++0x
-DBOOST_FILE_SYSTEM_DYN_LINK -DBOOST_PROGRAM_OPTIONS_DYN_LINK
-DBOOST_SYSTEM_DYN_LINK -DBOOST_TEST_DYN_LINK -DBOOST_THREAD_DYN_DLL
-DHAVE_XMLRPC_C -DKENLM_MAX_ORDER=6 -DMOSES_VERSION_ID=\"\" -DNDEBUG
-DTRACE_ENABLE=1 -DWITH_THREADS -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES  -I"."
-I"util/double-conversion" -I"~/demo/boost_1_72_0/include" -c -o
"mert/bin/darwin-11.0.3/release/link-static/threading-multi/VocabularyTest.o"
"mert/VocabularyTest.cpp"


...failed darwin.compile.c++
mert/bin/darwin-11.0.3/release/link-static/threading-multi/VocabularyTest.o...

...skipped
vocabulary_test
for lack of
VocabularyTest.o...

...skipped
vocabulary_test.passed
for lack of
vocabulary_test...

...failed updating 762 targets...

...skipped 219 targets...
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Help on transliteration

2020-04-04 Thread Nameet Mankar

Hello everyone I want to ask help regarding transliteration. How can I do
that in mosses using indic nlp or other if anyone has done this then I
kindly ask for their guidance.
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help me plz

2019-03-02 Thread Asteway

Hi Zemichael,

Can you provide the build command that caused this error?
If you are using any arguments with the build command use full paths not
relative paths.

On Sat, Mar 2, 2019, 10:34 AM Zemichael Berihu  wrote:

>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help with some moses features

2017-03-21 Thread Hieu Hoang

Hi Maria

* Looking for MT/NLP opportunities *
Hieu Hoang
http://moses-smt.org/


On 21 March 2017 at 14:23, Maria Braga  wrote:

> Hi Hieu,
>
> I am Maria and work at Unbabel. We've met in Prague at the MT Marathon
> last year.
>
> I am currently working with the feature weight-overwrite to switch between
> domain-adapt models. Attached the moses.ini I am using.
>
> The weights of the models are zero in moses.ini and the real weights
> (obtained during tuning) are sent along with the request to the
> mosesserver. E.g.:
>
> params = {"text": 'Hello world'}
>
> When I increase the number of models (in moses.ini) loaded, I observe the
> decoding time increasing. Example:
>
>- With a general model and two domain adapt model loaded, the decoding
>time was 1.95s
>- When added a third domain adapt model, decoding time went up to 2.80s
>
> Some notes: This values are the average decoding time for the same test
> corpus with ~650 sentences. The second bullet was a test performed with the
> three domain adapt models being the same (loaded three times the same
> model) and being different (two were the same model and a different model).
> Also the machine I am running this experiments have enough memory (the
> loaded models use a bit more than half of the available memory and there is
> enough CPUs).
>
> The questions are:
>
>
>1. Do you know this "weight-overwrite" feature?
>
> yes, this over-rides the weights in the ini file. I don't know much about
the moses server though.

>
>1. Why does by increasing the number of models from two to three
>(initialised at zeros) the decoding time increases?
>
> Even though the weights are zero, the models are still used. Phrase-pairs
need to be lookup up, new translations are created from the phrase-pairs
etc.

>
>1. If you don't know, can you point where should I look to find the
>answers to these questions?
>
> Best if you email the moses mailing list (cc'ed) so other people can chip
in too

>
> Cheers,
>
> Maria Braga
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] help

2017-01-29 Thread SõómÃh .

Hi ,

i am new to use moses so can anyone help me to install it in windows and how 
can i build translation model


Regads,

Ebtesam


Sent from Outlook
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] help in incremental training

2016-12-16 Thread Adel Khalifa

Dear All,


I just need a clear steps on how to do incremental training in moses, as
the illustration in the manual is not cleared enough

Thanks
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help me

2016-09-15 Thread Kenneth Heafield

https://github.com/moses-smt/mosesdecoder/archive/master.zip

On 09/15/16 11:20, Selva Nalladurai wrote:
> Hello guys,
>
>  Please provide me with the link, where i can download the
> complete moses toolkit 
>
>
>
>Regards,
>Selva Nalladurai
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] help me

2016-09-15 Thread Selva Nalladurai

Hello guys,

 Please provide me with the link, where i can download the
complete moses toolkit



   Regards,
   Selva Nalladurai
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] help regarding evaluation metrices

2016-07-19 Thread Selva Nalladurai

Hello guys,

 Can someone provide me with some evaluation metrics for
pairwise machine functions in statistical machine translation, other than

1)bleu
2)WER
3)TER
4)METEOR
5)Cosine similarity
6)LEPOR
7)NIST
7)ROUGE

-- 


   Regards,
   Selva Nalladurai
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help me guys

2016-07-18 Thread Hieu Hoang


add this to your moses.ini file

   [n-best-list]

   nbest.txt

   4

   distinct

or add these arguments when you run moses:

   -n-best-list nbest.txt 4 distinct


On 18/07/2016 07:23, Selva Nalladurai wrote:

Hello guys,
   I would like to know whether we can generate 
n-best translation list in moses(example: if am translating a 
phrase"how are you" from english to french, is it possible to generate 
4-5 phrases in french with meaning similar to "how are you".



Regards,
 Selva Nalladurai


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] help me guys

2016-07-18 Thread Selva Nalladurai

Hello guys,
   I would like to know whether we can generate n-best
translation list in moses(example: if am translating a phrase"how are you"
from english to french, is it possible to generate 4-5 phrases in french
with meaning similar to "how are you".


  Regards,
   Selva Nalladurai
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help regarding languages used for translation

2016-07-14 Thread Tom Hoar

I can't vouch for it. I just remember being surprised someone 
contributed the file a while ago. It but it seems to be a simple 
starting point since it's there.

Tom

On 7/14/2016 6:26 PM, Anoop (അനൂപ്) wrote:

Thanks Tom, I didn't know there was one in Moses for Tamil!

Regards
Anoop.

On Thu, Jul 14, 2016 at 4:17 PM, Tom Hoar <tah...@pttools.net 
<mailto:tah...@pttools.net>> wrote:

Re "Moses does not have tokenizers for Tamil", actually there is a
Tamil nonbreaking prefix file in the folder
scripts/share/nonbreaking_prefixes/nonbreaking_prefix.ta. You
might want to start simple starting with the
scripts/tokenizer/tokenizer.perl file. Then after you see how it
works, escalate to Anoop's suggestions.

Tom

On 7/14/2016 5:14 PM, moses-support-requ...@mit.edu
<mailto:moses-support-requ...@mit.edu> wrote:

Date: Thu, 14 Jul 2016 13:02:57 +0530
From: Anoop (?) <anoop.kunchukut...@gmail.com> 
<mailto:anoop.kunchukut...@gmail.com>
Subject: Re: [Moses-support] help regarding languages used for translation
To: Selva Nalladurai<selva...@gmail.com> <mailto:selva...@gmail.com>
Cc: moses-support<moses-support@mit.edu> <mailto:moses-support@mit.edu>

Hi Selva,

Moses is language-independent, so you can use it for any language pair as
long as you have a parallel corpus. That said, you may have to do language
specific pre and post-processing. For instance,

- Moses does not have tokenizers for Tamil.
- Tamil is agglutinative, so you may want to segment the words to reduce
data sparsity as an additional pre-processing step.

You can the Indic NLP library fo some some simple tokenization as well as
segment the text (http://anoopkunchukuttan.github.io/indic_nlp_library  ).

Since English and Tamil have different word order, you should try syntax
based models (which is implemented in the  Moses package). Another option
way is to pre-order the English sentence to the Tamil word order before
training a phrase based system. You can use this for pre-ordering:
http://www.cfilt.iitb.ac.in/~moses/download/cfilt_preorder/register.html
<http://www.cfilt.iitb.ac.in/%7Emoses/download/cfilt_preorder/register.html>

Regards,
Anoop.

On Thu, Jul 14, 2016 at 11:43 AM, Selva Nalladurai<selva...@gmail.com> 
<mailto:selva...@gmail.com>
wrote:

>Hello Team,
>I am Selva Nalladurai, doing my ME(masters of
>engineering) CSE and am from India. I m doing my research in SMT and would
>like to know whether Moses can be used for translating English to Tamil ( a
>regional language spoken in our country), can i follow the same steps as
>other languages translation for upolading my corpus.
>Thankyou.
>
>Regards,
>Selva Nalladurai
>
>___
>Moses-support mailing list
>Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>http://mailman.mit.edu/mailman/listinfo/moses-support
>
>

___
Moses-support mailing list
Moses-support@mit.edu <mailto:Moses-support@mit.edu>
http://mailman.mit.edu/mailman/listinfo/moses-support

--
I claim to be a simple individual liable to err like any other fellow 
mortal. I own, however, that I have humility enough to confess my 
errors and to retrace my steps.

http://flightsofthought.blogspot.com

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] help regarding languages used for translation

2016-07-14 Thread Selva Nalladurai

Hello Team,
   I am Selva Nalladurai, doing my ME(masters of
engineering) CSE and am from India. I m doing my research in SMT and would
like to know whether Moses can be used for translating English to Tamil ( a
regional language spoken in our country), can i follow the same steps as
other languages translation for upolading my corpus.
   Thankyou.

   Regards,
   Selva Nalladurai
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help

2016-03-25 Thread Lane Schwartz

Well, the error means that the file doesn't exist in that directory. You'll
need to see if you can debug it from there.

On Fri, Mar 25, 2016 at 12:22 AM, Parul gupta  wrote:

> Yes i'm in the same directory and file is existing in that directory.
>
> On Fri, Mar 25, 2016 at 2:41 AM, Lane Schwartz  wrote:
>
>> Make sure you're in the correct directory. Make sure the file actually
>> exists in that directory.
>>
>> On Thu, Mar 24, 2016 at 2:52 PM, Parul gupta  wrote:
>>
>>> I gave this command as follows:
>>>
>>> parul@ubuntu:~$ ~/mosesdecoder/bin/build_binary \
>>> >WMT-News.arpa.en \
>>> >WMT-News.en-hi.blm.hi
>>>
>>> I'm getting this error as follows:
>>>
>>> util/file.cc:76 in int util::OpenReadOrThrow(const char*) threw
>>> ErrnoException because `-1 == (ret = open(name, 00))'.
>>> No such file or directory while opening WMT-News.arpa.en
>>> ERROR
>>>
>>> What does it mean ?
>>>
>>>
>>> I previously gave the same command..at that time it works correctly and
>>> created WMT-News.en-hi.blm.hi file..but when i'm trying to open
>>> WMT-NEws.en-hi.hi file it's showing couln't display...i thought it
>>> might crashed...so, i tried to cretae this file again...but it's showing
>>> this error...what can i do ???
>>>
>>>
>>>
>>> ___
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>
>>
>> --
>> When a place gets crowded enough to require ID's, social collapse is not
>> far away.  It is time to go elsewhere.  The best thing about space travel
>> is that it made it possible to go elsewhere.
>> -- R.A. Heinlein, "Time Enough For Love"
>>
>
>


-- 
When a place gets crowded enough to require ID's, social collapse is not
far away.  It is time to go elsewhere.  The best thing about space travel
is that it made it possible to go elsewhere.
-- R.A. Heinlein, "Time Enough For Love"
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help

2016-03-24 Thread Lane Schwartz

Make sure you're in the correct directory. Make sure the file actually
exists in that directory.

On Thu, Mar 24, 2016 at 2:52 PM, Parul gupta  wrote:

> I gave this command as follows:
>
> parul@ubuntu:~$ ~/mosesdecoder/bin/build_binary \
> >WMT-News.arpa.en \
> >WMT-News.en-hi.blm.hi
>
> I'm getting this error as follows:
>
> util/file.cc:76 in int util::OpenReadOrThrow(const char*) threw
> ErrnoException because `-1 == (ret = open(name, 00))'.
> No such file or directory while opening WMT-News.arpa.en
> ERROR
>
> What does it mean ?
>
>
> I previously gave the same command..at that time it works correctly and
> created WMT-News.en-hi.blm.hi file..but when i'm trying to open
> WMT-NEws.en-hi.hi file it's showing couln't display...i thought it
> might crashed...so, i tried to cretae this file again...but it's showing
> this error...what can i do ???
>
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


-- 
When a place gets crowded enough to require ID's, social collapse is not
far away.  It is time to go elsewhere.  The best thing about space travel
is that it made it possible to go elsewhere.
-- R.A. Heinlein, "Time Enough For Love"
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] help

2016-03-24 Thread Parul gupta

I gave this command as follows:

parul@ubuntu:~$ ~/mosesdecoder/bin/build_binary \
>WMT-News.arpa.en \
>WMT-News.en-hi.blm.hi

I'm getting this error as follows:

util/file.cc:76 in int util::OpenReadOrThrow(const char*) threw
ErrnoException because `-1 == (ret = open(name, 00))'.
No such file or directory while opening WMT-News.arpa.en
ERROR

What does it mean ?


I previously gave the same command..at that time it works correctly and
created WMT-News.en-hi.blm.hi file..but when i'm trying to open
WMT-NEws.en-hi.hi file it's showing couln't display...i thought it
might crashed...so, i tried to cretae this file again...but it's showing
this error...what can i do ???
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help

2016-03-23 Thread Philipp Koehn

Hi,

this looks like a successful build.

It will not have the server functionality where moses runs as a daemon and
responds to tcp/ip requests.

-phi

On Tue, Mar 22, 2016 at 2:43 AM, Parul gupta  wrote:

> warning: No toolsets are configured.
> warning: Configuring default toolset "gcc".
> warning: If the default is wrong, your build may not work correctly.
> warning: Use the "toolset=x" option to override our guess.
> warning: For more configuration options, please consult
> warning:
> http://boost.org/boost-build2/doc/html/bbv2/advanced/configuration.html
> NOT BUILDING MOSES SERVER!
> Performing configuration checks
>
> - Shared Boost : yes (cached)
> - Static Boost : yes (cached)
> ...patience...
> ...patience...
> ...found 4563 targets...
> SUCCESS
>
>
> what does it mean ? is there anything wrong ?
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] help

2016-03-22 Thread Parul gupta

warning: No toolsets are configured.
warning: Configuring default toolset "gcc".
warning: If the default is wrong, your build may not work correctly.
warning: Use the "toolset=x" option to override our guess.
warning: For more configuration options, please consult
warning:
http://boost.org/boost-build2/doc/html/bbv2/advanced/configuration.html
NOT BUILDING MOSES SERVER!
Performing configuration checks

- Shared Boost : yes (cached)
- Static Boost : yes (cached)
...patience...
...patience...
...found 4563 targets...
SUCCESS


what does it mean ? is there anything wrong ?
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help

2016-03-21 Thread Lane Schwartz

Parul,

You'll get better responses if you email the list instead of me. Please
start a new email thread to the mailing list about this topic.

Lane


On Mon, Mar 21, 2016 at 12:53 AM, Parul gupta  wrote:

> This is the command i'm running
>
> echo ’is this an english sentence’ | /home/parul/mosesdecoder/bin/moses -f
> /home/parul/working/mert-work/run1.moses.ini > out
>
> On Fri, Mar 18, 2016 at 6:29 PM, Parul gupta  wrote:
>
>> Hello Sir,
>>
>> Can you tell me how to check and give input to get the translated output ?
>>
>> On Thu, Mar 17, 2016 at 11:52 PM, Parul gupta  wrote:
>>
>>> I am giving this command:
>>>
>>> ~/working$ nohup nice ~/mosesdecoder/scripts/training/train-model.perl
>>> -root-dir train  -corpus ~/corpus/WMT-News.en-hi.clean
>>>  -f en -e hi -alignment grow-diag-final-and -reordering
>>> msd-bidirectional-fe  -lm 0:3:/home/parul/lm/WMT-News.en-hi.blm.hi:8
>>>  -external-bin-dir /home/parul/mosesdecoder/tools>& training.out &
>>>
>>>
>>> On Thu, Mar 17, 2016 at 11:50 PM, Parul gupta  wrote:
>>>
 Use of uninitialized value $_EXTERNAL_BINDIR in concatenation (.) or
 string at /home/parul/mosesdecoder/scripts/training/train-model.perl line
 488.

 ERROR: Cannot find mkcls, GIZA++/mgiza, & snt2cooc.out/snt2cooc in .

 You MUST specify the parameter -external-bin-dir at
 /home/parul/mosesdecoder/scripts/training/train-model.perl line 488.

 I'm getting these errors.

 On Thu, Mar 17, 2016 at 9:24 PM, Parul gupta  wrote:

> But i have some other doubt.
>
>  -lm 0:3:$HOME/lm/news-commentary-v8.fr-en.blm.en:8   
>\
>
>
> Here why are they using 3 ? Is it because of 3-gram model ?
>
> And why is it 8 ?
>
> On Thu, Mar 17, 2016 at 9:15 PM, Parul gupta  wrote:
>
>> I got the solution .
>>
>> On Thu, Mar 17, 2016 at 8:54 PM, Lane Schwartz 
>> wrote:
>>
>>> Is that the exact command you're using? You shouldn't have
>>> backslashes before the tildes.
>>>
>>> Also, what is the error that you're getting?
>>>
>>> On Thu, Mar 17, 2016 at 5:16 AM, Parul gupta 
>>> wrote:
>>>
 ~/mosesdecoder/scripts/training/clean-corpus-n.perl
 \~/corpus/WMT-News.true en hi \~/corpus/WMT-News.clean 1 80

 i'm giving this command

 On Thu, Mar 17, 2016 at 3:33 PM, Parul gupta 
 wrote:

> i got the answer for that.
>
>   ~/corpus/WMT-News.true.en does not exist at
> /home/mosesdecoder/scripts/training/clean-corpus-n.perl line 58.
>
> I'm getting this error. I didn't get what they are trying to say
> in the error. Can you tell me ?
>
> On Wed, Mar 16, 2016 at 8:01 PM, Lane Schwartz  > wrote:
>
>> Can you show the exact command you're running? That script
>> accepts data on standard input, so if you're not giving anything on
>> standard input, it won't produce any output.
>>
>> On Mon, Mar 14, 2016 at 11:40 PM, Parul gupta 
>> wrote:
>>
>>> Sir i created scripts/share/nonbreaking_prefix.hi but when i'm
>>> running the command ..it's showing
>>>
>>> Tokenizer Version 1.1
>>> Language: hi
>>> Number of threads: 1
>>>
>>> and no file has been created of hindi tokenized file.
>>>
>>> ___
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>
>>
>> --
>> When a place gets crowded enough to require ID's, social collapse
>> is not
>> far away.  It is time to go elsewhere.  The best thing about
>> space travel
>> is that it made it possible to go elsewhere.
>> -- R.A. Heinlein, "Time Enough For Love"
>>
>
>

>>>
>>>
>>> --
>>> When a place gets crowded enough to require ID's, social collapse is
>>> not
>>> far away.  It is time to go elsewhere.  The best thing about space
>>> travel
>>> is that it made it possible to go elsewhere.
>>> -- R.A. Heinlein, "Time Enough For Love"
>>>
>>
>>
>

>>>
>>
>


-- 
When a place gets crowded enough to require ID's, social collapse is not
far away.  It is time to go elsewhere.  The best thing about space travel
is that it made it possible to go elsewhere.
-- R.A. Heinlein, "Time Enough For Love"
___
Moses-support mailing list

Re: [Moses-support] help

2016-03-19 Thread Lane Schwartz

Can you show the exact command you're running? That script accepts data on
standard input, so if you're not giving anything on standard input, it
won't produce any output.

On Mon, Mar 14, 2016 at 11:40 PM, Parul gupta  wrote:

> Sir i created scripts/share/nonbreaking_prefix.hi but when i'm running
> the command ..it's showing
>
> Tokenizer Version 1.1
> Language: hi
> Number of threads: 1
>
> and no file has been created of hindi tokenized file.
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


-- 
When a place gets crowded enough to require ID's, social collapse is not
far away.  It is time to go elsewhere.  The best thing about space travel
is that it made it possible to go elsewhere.
-- R.A. Heinlein, "Time Enough For Love"
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] help

2016-03-19 Thread Parul gupta

Sir i created scripts/share/nonbreaking_prefix.hi but when i'm running the
command ..it's showing

Tokenizer Version 1.1
Language: hi
Number of threads: 1

and no file has been created of hindi tokenized file.
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help

2016-03-18 Thread Lane Schwartz

Is that the exact command you're using? You shouldn't have backslashes
before the tildes.

Also, what is the error that you're getting?

On Thu, Mar 17, 2016 at 5:16 AM, Parul gupta  wrote:

> ~/mosesdecoder/scripts/training/clean-corpus-n.perl
> \~/corpus/WMT-News.true en hi \~/corpus/WMT-News.clean 1 80
>
> i'm giving this command
>
> On Thu, Mar 17, 2016 at 3:33 PM, Parul gupta  wrote:
>
>> i got the answer for that.
>>
>>   ~/corpus/WMT-News.true.en does not exist at 
>> /home/mosesdecoder/scripts/training/clean-corpus-n.perl
>> line 58.
>>
>> I'm getting this error. I didn't get what they are trying to say in the
>> error. Can you tell me ?
>>
>> On Wed, Mar 16, 2016 at 8:01 PM, Lane Schwartz 
>> wrote:
>>
>>> Can you show the exact command you're running? That script accepts data
>>> on standard input, so if you're not giving anything on standard input, it
>>> won't produce any output.
>>>
>>> On Mon, Mar 14, 2016 at 11:40 PM, Parul gupta  wrote:
>>>
 Sir i created scripts/share/nonbreaking_prefix.hi but when i'm running
 the command ..it's showing

 Tokenizer Version 1.1
 Language: hi
 Number of threads: 1

 and no file has been created of hindi tokenized file.

 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support


>>>
>>>
>>> --
>>> When a place gets crowded enough to require ID's, social collapse is not
>>> far away.  It is time to go elsewhere.  The best thing about space travel
>>> is that it made it possible to go elsewhere.
>>> -- R.A. Heinlein, "Time Enough For Love"
>>>
>>
>>
>


-- 
When a place gets crowded enough to require ID's, social collapse is not
far away.  It is time to go elsewhere.  The best thing about space travel
is that it made it possible to go elsewhere.
-- R.A. Heinlein, "Time Enough For Love"
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help on pipeline ....

2015-09-17 Thread Vincent Nguyen



Well in fact, it comes to how to handle xml mark up in text to translate 
and obviously this is not trivial and no good solution.


If I am not mistaken this issue has been tackled in various circumstances :
- casmacat project / matecat_util project
- web translation in the contrib / web translate.cgi script (which 
handles html tags)


I found old posts from Philipp saying the exact location of mark ups 
could be determined with phrase and word alignment .


well if I am not mistaken :

let's take this example :
Les banques de la zone euro sont soumises :

only the word "euro" is tagged g2

if "zone euro" is phrase translated to "euro zone" then Moses will only 
send back the phrase alignment. So we lose the information

and we cannot tag g2 the word "euro".

I guess to prevent this one did try to get the Word alignment using 
ibm1align.py
in this 
https://github.com/casmacat/moses-mt-server/tree/master/code/tags4moses


BUT it requires the lex vocabulary files which is to me not a good 
solution since it must temper a lot the performance.


In the end, I don't really see any good solution without making Moses 
xml aware .


big debate ?




Le 16/09/2015 17:30, Vincent Nguyen a écrit :


I am struggling with a pipeline .

Here is the text1.txt file I would like to translate from FR to EN
Les banques de la zone euro sont soumises :
au ratio de capital lié à la détention d’actifs risqués 
(nous nous intéressons ici au crédit) ;
au ratio de levier, qui détermine le capital règlementaire à 
partir de la taille du bilan de la banque ;
au ratio de liquidité, qui impose aux banques de détenir en 
particulier des portefeuilles importants de titres publics.


I am running the following properly :

/home/moses/mosesdecoder/scripts/tokenizer/normalize-punctuation.perl 
fr < text1.txt > text2.txt
/home/moses/matecat/matecat_util/code/tokenizer/deescape-special-chars.perl 
< text2.txt > text3.txt
/home/moses/matecat/matecat_util/code/tokenizer/tokenizer.perl -X -a 
-l fr  < text3.txt > text4.txt
/home/moses/mosesdecoder/scripts/recaser/truecase.perl --model 
/home/moses/working/truecaser/truecase-model.1.fr < text4.txt > text5.txt
/home/moses/mosesdecoder/bin/moses -f 
/home/moses/working/tuning/moses.tuned.ini.1 < text5.txt > text6.txt


then in my text6.txt I have

 banks in the euro zone are subject :
 ratio of capital linked to the detention of risky assets ( 
we are here to credit ; )
 the leverage ratio , which determines the regulatory 
capital from the size of the balance sheet of the bank ;
 ratio of liquidity , which requires banks to hold 
especially important portfolios of securities . public


but then neither the detokenizer nor the detruecaser will give me the 
correct output.

"banks" will not get the uppercase B


I also tried to look at this 
https://github.com/christianbuck/matecat_util/tree/master/python_server
or this 
https://github.com/christianbuck/matecat_util/tree/master/code/tags4moses


but no luck.




___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Help on pipeline ....

2015-09-16 Thread Vincent Nguyen



I am struggling with a pipeline .

Here is the text1.txt file I would like to translate from FR to EN
Les banques de la zone euro sont soumises :
au ratio de capital lié à la détention d’actifs risqués (nous 
nous intéressons ici au crédit) ;
au ratio de levier, qui détermine le capital règlementaire à 
partir de la taille du bilan de la banque ;
au ratio de liquidité, qui impose aux banques de détenir en 
particulier des portefeuilles importants de titres publics.


I am running the following properly :

/home/moses/mosesdecoder/scripts/tokenizer/normalize-punctuation.perl fr 
< text1.txt > text2.txt
/home/moses/matecat/matecat_util/code/tokenizer/deescape-special-chars.perl 
< text2.txt > text3.txt
/home/moses/matecat/matecat_util/code/tokenizer/tokenizer.perl -X -a -l 
fr  < text3.txt > text4.txt
/home/moses/mosesdecoder/scripts/recaser/truecase.perl --model 
/home/moses/working/truecaser/truecase-model.1.fr < text4.txt > text5.txt
/home/moses/mosesdecoder/bin/moses -f 
/home/moses/working/tuning/moses.tuned.ini.1 < text5.txt > text6.txt


then in my text6.txt I have

 banks in the euro zone are subject :
 ratio of capital linked to the detention of risky assets ( we 
are here to credit ; )
 the leverage ratio , which determines the regulatory capital 
from the size of the balance sheet of the bank ;
 ratio of liquidity , which requires banks to hold especially 
important portfolios of securities . public


but then neither the detokenizer nor the detruecaser will give me the 
correct output.

"banks" will not get the uppercase B


I also tried to look at this 
https://github.com/christianbuck/matecat_util/tree/master/python_server
or this 
https://github.com/christianbuck/matecat_util/tree/master/code/tags4moses


but no luck.


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help

2014-11-11 Thread Philipp Koehn

Hi,

these are very small corpus sizes to train statistical machine
translation system. Typically, folks use at least 100 times
as many words.

You should be able to track down, why some translations
deteriorate - I would expect that coverage increases and
you would get better translations.

-phi

On Sun, Nov 9, 2014 at 7:50 AM, Maria Marpaung
maria_marpa...@yahoo.co.id wrote:
 Hello please help me,
 I want to ask, I have can run the translation using Moses. First, I using
 1000 words. next I add a document to 2000 words. But, results generated
 translation worse than the 1000 document words.
 Can you give advice, what should I do?

 Best regards,

 Maria Marpaung


 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Help

2014-11-09 Thread Maria Marpaung

Hello please help me,
I want to ask, I have can run the translation using Moses. First, I using 1000 
words. next I add a document to 2000 words. But, results generated translation 
worse than the 1000 document words.
 Can you give advice, what should I do?
Best regards,
Maria Marpaung
 ___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help in Moses Incremental Retraing

2014-10-24 Thread Sandipan Dandapat

Hi Ulrich,
Sorry for sending the doubts to you directly. I will keep in mind to post
my future queries to moses-support.

Thanks a lot for the clarification. Let me play with the MERT.

Thanks and regards,
sandipan

On 24 October 2014 01:51, Ulrich Germann ulrich.germ...@gmail.com wrote:

Hi Sandipan,

first, please post Moses-related questions to moses-support@mit.edu, not
individual contributors.

second, the current seven features used by Mmsapt /
PhraseDictionaryBitextSampling are (for details, see my recent paper on
this phrase table implementation:
https://www.researchgate.net/publication/267270863_Dynamic_Phrase_Tables_for_Machine_Translation_in_an_Interactive_Post-editing_Scenario
)

THE STANDARD SET OF FEATURES MAY CHANGE AT ANY TIME, as this is still work
in progress.

- forward and backward lexically smoothed phrase scores (2 scores; same as
standard features)
- rarity penalty (1/(x+1)), where x is the number of phrase pair
occurrences in the corpus/sample (1 score)
- the lower bound on forward and backward phrase-level probabilities, with
confidence level .99 (2 scores)
- 2 provenance features (x/(x+1)), where x is the number of phrase pair
occurrences in the (static) background and (dynamic) foreground corpus (2
scores)

third, you need to retrain the feature weights for good performance with
any of the standard techniques, but with the I usually use MERT. The
executable simulate-pe allows you to feed in references and word
aligmnents one sentence at a time; there are additional parameters
--spe-src, --spe-trg, --spe-aln to specify source, target, and alignment
(symal output format). Source and target files are one sentence per line,
tokenized. Michael Denkowski is currently in the process of integrating
online tuning into Moses, but I'm not sure whether that's ready to be
deployed yet.

Regards - Uli

On Thu, Oct 23, 2014 at 1:47 AM, Sandipan Dandapat
sandipandanda...@gmail.com wrote:

Dear Ulrich,
I got your reference from Prashanta Mathur. I am a postdoctoral
researcher in CNGL, DCU and I am working with Moses incremental
retraining. It will be great if you help me to understand couple of doubts:

1. I found there are 7 weights to define for PT0 (PT0 is the Mmsapt name)
i.e.

Mmsapt name=PT0 output-factor=0 num-features=7
base=/home/sandipan/inc_retrain/MT_sys/En-Fr/dgt/50_i/mmsa_pt/train. L1=en
L2=fr
[weight]
PT0= 0.1 0.2 0.3 0.4 0.5 0.6 0.7

num-featues in PBSMT model is 4 which does not work with Mmsapt. What are
these 7 weights? Can I use uniform weights for all 7 features? Or how do I
adjust these values? Or, how to adjust these weights?

2. I found there is significant difference in BLEU score when I am using
standard PBSMT model and when I am using MMST based model. Is this because
of the weights I am using or am I doing something wrong?

It will be real great help, if you help me to understand the above issue.
Thanking you.

Regards,
sandipan

--
Ulrich Germann
Research Associate
School of Informatics
University of Edinburgh

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help in Moses Incremental Retraing

2014-10-23 Thread Ulrich Germann

Hi Sandipan,

first, please post Moses-related questions to moses-support@mit.edu, not
individual contributors.

THE STANDARD SET OF FEATURES MAY CHANGE AT ANY TIME, as this is still work
in progress.

Regards - Uli

On Thu, Oct 23, 2014 at 1:47 AM, Sandipan Dandapat
sandipandanda...@gmail.com wrote:

Dear Ulrich,
I got your reference from Prashanta Mathur. I am a postdoctoral researcher
in CNGL, DCU and I am working with Moses incremental retraining. It will
be great if you help me to understand couple of doubts:

1. I found there are 7 weights to define for PT0 (PT0 is the Mmsapt name)
i.e.

Mmsapt name=PT0 output-factor=0 num-features=7
base=/home/sandipan/inc_retrain/MT_sys/En-Fr/dgt/50_i/mmsa_pt/train. L1=en
L2=fr
[weight]
PT0= 0.1 0.2 0.3 0.4 0.5 0.6 0.7

It will be real great help, if you help me to understand the above issue.
Thanking you.

Regards,
sandipan

--
Ulrich Germann
Research Associate
School of Informatics
University of Edinburgh
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help

2014-09-26 Thread Hieu Hoang

the decoder is waiting for the input. You can type your input in, or
provide the decoder with the input file  to translate:
   /moses -f moses.ini -i input-file
or
   /moses -f moses.ini  input-file
Don't forget - the input must have been tokenized and cased in the same way
as the training data

On 25 September 2014 09:31, Maria Marpaung maria_marpa...@yahoo.co.id
wrote:

 Hi,
 I have following your instruction here:
 http://www.statmt.org/moses/?n=Moses.Baseline
 and I kind of stuck in the Testing step.

 I want running a translation, but loading is so long and also the
 translation not out.
 My command is this:

 maria@maria-Aspire-E1-471:~$ /home/maria/mosesdecoder/bin/moses -f
 /home/maria/working/binarised-model/moses.ini

 Defined parameters (per moses.ini or switch):
 config: /home/maria/working/binarised-model/moses.ini
 distortion-limit: 6
 feature: UnknownWordPenalty WordPenalty PhrasePenalty
 PhraseDictionaryBinary name=TranslationModel0 num-features=4
 path=/home/maria/working/binarised-model/phrase-table input-factor=0
 output-factor=0 LexicalReordering name=LexicalReordering0 num-features=6
 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
 path=/home/maria/working/binarised-model/reordering-table Distortion KENLM
 lazyken=0 name=LM0 factor=0
 path=/home/maria/lm/news-commentary-v8.fr-en.blm.en order=3
 .
 .
 .
 .
 Loading UnknownWordPenalty0
 Loading WordPenalty0
 Loading PhrasePenalty0
 Loading LexicalReordering0
 binary file loaded, default OFF_T: -1
 Loading Distortion0
 Loading LM0
 Loading TranslationModel0
 IO from STDOUT/STDIN
 Created input-output object : [0.907] seconds

 I just stop here. Translation does not exit
 What should I do?
 please help me for my thesis

 Thank you,

 Maria

 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support




-- 
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Help

2014-09-25 Thread Maria Marpaung

Hi,
I have following your instruction here: 
http://www.statmt.org/moses/?n=Moses.Baseline
and I kind of stuck in the Testing step.

I want running a translation, but loading is so long and also the translation 
not out.
My command is this:

maria@maria-Aspire-E1-471:~$ /home/maria/mosesdecoder/bin/moses -f 
/home/maria/working/binarised-model/moses.ini

Defined parameters (per moses.ini or switch):
config: /home/maria/working/binarised-model/moses.ini 
distortion-limit: 6 
feature: UnknownWordPenalty WordPenalty PhrasePenalty 
PhraseDictionaryBinary name=TranslationModel0 num-features=4 
path=/home/maria/working/binarised-model/phrase-table input-factor=0 
output-factor=0 LexicalReordering name=LexicalReordering0 num-features=6 
type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 
path=/home/maria/working/binarised-model/reordering-table Distortion KENLM 
lazyken=0 name=LM0 factor=0 path=/home/maria/lm/news-commentary-v8.fr-en.blm.en 
order=3 
.
.
.
.
Loading UnknownWordPenalty0
Loading WordPenalty0
Loading PhrasePenalty0
Loading LexicalReordering0
binary file loaded, default OFF_T: -1
Loading Distortion0
Loading LM0
Loading TranslationModel0
IO from STDOUT/STDIN
Created input-output object : [0.907] seconds


I just stop here. Translation does not exit

What should I do?
please help me for my thesis

Thank you,

Maria
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Help

2014-09-23 Thread Maria Marpaung

Hi,

I want to ask.
I've Testing step.
Please tell me how make Set the path of the PhraseDictionary feature to point 
to $HOME/working/binarised- model/phrase-table and  Set the path of the 
LexicalReordering feature to point to $HOME/working/binarised- 
model/reordering-table?

Idont know how do it, please help me

Thank you,

Maria Marpaung ___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help

2014-09-23 Thread Hieu Hoang

hi maria

On 23 September 2014 10:49, Maria Marpaung maria_marpa...@yahoo.co.id
wrote:

 Hi,

 I want to ask.
 I've Testing step.
 Please tell me how make Set the path of the PhraseDictionary feature to
 point to $HOME/working/binarised- model/phrase-table

In the moses.ini file, there is a line
   PhraseDictionaryMemory path=/home/.
Change it to
   PhraseDictionaryBinary
path=/home/username/working/binarised-model/phrase-table


 and Set the path of the LexicalReordering feature to point to
 $HOME/working/binarised- model/reordering-table?

The moses.ini should also contain a line
 LexicalReordering path=/home/
change it to
   LexicalReordering
path=/home/username/working/binarised-model/reordering-table

Idont know how do it, please help me

 Thank you,

 Maria Marpaung


 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support




-- 
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] help

2014-09-01 Thread Mulu Gebreegziabher

I am getting an error ...failed gcc.link 
/home/.../tools/moses/bin/mosesserver... during moses installation when using 
the following command:
 ./bjam --with-srilm=/home/mulu/moses/tools/srilm/ -a 

So I do again 
./bjam --with-srilm=/home/mulu/moses/tools/srilm/ -a --debug-configuration -d2 
| gzip build.log.gz
and attached herewith the log file.

I am running a 64bit machine with Debian wheezy and I have installed boost1.56 
with the command:
./bootstrap.sh
sudo ./b2 --prefix=/usr/local/include/boost --libdir =$PWD/lib64 
--layout=tagged link=static,shared threading=multi,single install || echo 
FAILURE
I have also export PATH=$PATH\:/usr/local/include:/usr/local/include/boost

I have also installed packages that are needed by boost like libbz2-dev 
zlib1g-dev tk-dev tcl-dev ,etc.

I have attached the error log.

I have tried all options but still getting 


build.log.gz
Description: application/gzip
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help

2014-09-01 Thread Barry Haddow


Hi Mulu

Is it the same as this problem?
https://www.mail-archive.com/moses-support@mit.edu/msg10584.html

Note that you probably don't need to build Moses with srilm. By default, 
Moses includes KenLM, which can handle ARPA files, as well as its own 
compiled format for faster loading.


cheers - Barry

On 01/09/14 10:27, Mulu Gebreegziabher wrote:
I am getting an error ...failed gcc.link 
/home/.../tools/moses/bin/mosesserver... during moses installation 
when using the following command:

 ./bjam --with-srilm=/home/mulu/moses/tools/srilm/ -a

So I do again
./bjam --with-srilm=/home/mulu/moses/tools/srilm/ -a 
--debug-configuration -d2 | gzip build.log.gz

and attached herewith the log file.

I am running a 64bit machine with Debian wheezy and I have installed 
boost1.56 with the command:

./bootstrap.sh
sudo ./b2 --prefix=/usr/local/include/boost --libdir =$PWD/lib64 
--layout=tagged link=static,shared threading=multi,single install || 
echo FAILURE

I have also export PATH=$PATH\:/usr/local/include:/usr/local/include/boost

I have also installed packages that are needed by boost like 
libbz2-dev zlib1g-dev tk-dev tcl-dev ,etc.


I have attached the error log.

I have tried all options but still getting


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help

2014-06-20 Thread Nicola Bertoldi

Hi Maria
first  I would like to ask (for next time)
to write to the IRSTLM mailing list  (user-irs...@list.fbk.eu)
to any problem related to IRSTLM only

 
Now I am going to reply you privately.


Nicola



On Jun 20, 2014, at 6:47 AM, Maria Marpaung wrote:

 Hello, I need help.
 I'm working on my thesis for form translation system using MOSES MT. I came 
 to the Language model Training. I am having some problems. I have followed 
 some of the steps like following the Moses Baseline. including the following:
 
 1. The language model (LM) is used to ensure fluent output, so it is built
 with the target language (i.e Indonesia language in this case). The IRSTLM
 documentation gives a full explanation of the command-line options, but the
 following will build an appropriate 3-gram language model, removing
 singletons, smoothing with improved Kneser-Ney, and adding sentence
 boundary symbols:
 
  mkdir ~/lm
  cd ~/lm
  ~/irstlm/bin/add-start-end.sh \
 ~/corpus/news-commentary-v8.fr-en.true.en \
 news-commentary-v8.fr-en.sb.en
  export IRSTLM=$HOME/irstlm; ~/irstlm/bin/build-lm.sh \
-i news-commentary-v8.fr-en.sb.en  \
-t ./tmp  -p -s improved-kneser-ney -o news-commentary-v8.fr-en.lm.en
  ~/irstlm/bin/compile-lm --text news-commentary-v8.fr-en.lm.en.gz \
news-commentary-v8.fr-en.arpa.en
 
 First until four commands were executed successfuly. The last one failed.
 Here is the result after entering the following command line:
 
 maria@maria-Aspire-E1-471:~/lm$ ~/moses/irstlm/bin/compile-lm --text
 news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en
 
 inpfile: news-commentary-v8.fr-en.lm.en.gz
 outfile: news-commentary-v8.fr-en.arpa.en
 loading up to the LM level 1000 (if any)
 dub: 1000
 Failed to open news-commentary-v8.fr-en.lm.en.gz!
 
 
 2. I add command yes like this:
 maria@maria-Aspire-E1-471:~/lm$ ~/moses/irstlm/bin/compile-lm –text yes
 news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en
 
 Warning: Too many arguments
 
 
 
 please help me, what should I do?
 
 
 Best regards!
 
 Maria Marpaung
 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help about Using Giza++ for comparable Corpora

2014-06-20 Thread Philipp Koehn

Hi,

GIZA++ is designed for parallel sentence pairs, but it is tolerant to
some degree of noise. If you feed comparable sentence pairs,
which I assume you mean to be sentence pairs which are only
partially (or not at all) translationally equivalent, then you will
hopefully get word alignments for the relevant parts, but also
alignments for the mismatched parts. Given a large amount
of such data, you may be able to learn useful word translation
tables - but it depends on the noisiness of the data.

If you add parallel sentence pairs, you will likely get better
alignments.

-phi


On Thu, Jun 19, 2014 at 7:20 AM, alireza tabebordbar
ar.tabebord...@gmail.com wrote:
 Hi all
 I am Master degree Student and I'm New to SMT.
 I extracted some Comparable sentences and I want to use Giza++ for
 word alignment.I Think Giza Use EM algorithm for aligning words,
 However I don't know that I can feed comparable sentences directly to
 Giza++ or I have combine them with some Parallel sentences.
 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Help about Using Giza++ for comparable Corpora

2014-06-19 Thread alireza tabebordbar

Hi all
I am Master degree Student and I'm New to SMT.
I extracted some Comparable sentences and I want to use Giza++ for
word alignment.I Think Giza Use EM algorithm for aligning words,
However I don't know that I can feed comparable sentences directly to
Giza++ or I have combine them with some Parallel sentences.
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] help

2014-06-19 Thread Maria Marpaung

 
Hello, I need help. 
I'm working on my thesis for form translation system using MOSES MT. I came to 
the Language model Training. I am having some problems. I have followed some of 
the steps like following the Moses Baseline. including the following: 1. The 
language model (LM) is used to ensure fluent output, so it is built
with the target language (i.e Indonesia language in this case). The IRSTLM
documentation gives a full explanation of the command-line options, but the
following will build an appropriate 3-gram language model, removing
singletons, smoothing with improved Kneser-Ney, and adding sentence
boundary symbols: mkdir ~/lm cd ~/lm ~/irstlm/bin/add-start-end.sh  
   \  ~/corpus/news-commentary-v8.fr-en.true.en \  
news-commentary-v8.fr-en.sb.en export IRSTLM=$HOME/irstlm; 
~/irstlm/bin/build-lm.sh \ -i news-commentary-v8.fr-en.sb.en  \ 
-t ./tmp  -p -s improved-kneser-ney -o news-commentary-v8.fr-en.lm.en 
~/irstlm/bin/compile-lm --text news-commentary-v8.fr-en.lm.en.gz \ 
news-commentary-v8.fr-en.arpa.en First until four commands were executed 
successfuly. The last one failed.
Here is the result after entering the following command line: 
maria@maria-Aspire-E1-471:~/lm$ ~/moses/irstlm/bin/compile-lm --text
news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en inpfile: 
news-commentary-v8.fr-en.lm.en.gz
outfile: news-commentary-v8.fr-en.arpa.en
loading up to the LM level 1000 (if any)
dub: 1000
Failed to open news-commentary-v8.fr-en.lm.en.gz! 


2. I add command yes like this:
maria@maria-Aspire-E1-471:~/lm$ ~/moses/irstlm/bin/compile-lm –text yes
news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en

Warning: Too many arguments



please help me, what should I do?


Best regards!

Maria Marpaung___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Help with installing Moses

2014-06-10 Thread bassel ahmad

Dear Sir,
Please find attached build.log.zip file I was asked to send you.
I used the command:
./bjam --with-boost=~/boost_1_55_01 -j8
to generate the error. 

Can you please help me to correct this?

 
Best Regards,


Bassel Ahmad

build.log.gz
Description: application/gzip
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] HELP

2014-05-04 Thread Richa Bansal

Hello,

While compiling moses i got the following error, moses/LM/IRST.cpp:27:25:
fatal error: lmContainer.h: No such file or directory.

Also the debug file is attached for your refrence.

Please do the needfull


Regards
Richa Bansal


build.log1.gz
Description: GNU Zip compressed data
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Help: Issues while installing Moses and Boost

2014-03-23 Thread Apurv Nagvenkar

Hello Sir,

I am trying to install moses on Fedora 18 64-bit system.
Following are the commands or instructions which I followed from numerous
tutorials.

***-Build SRILM-**
cd srilm
MAKE_PIC=1
export SRILM=$PWD
export MACHINE_TYPE=i686-m64
export PATH=${PATH}:$SRILM/bin/$MACHINE_TYPE
export PATH=${PATH}:$SRILM/bin
export MANPATH=$SRILM/man
make World
make test
make cleanest

**-Installing external directories--**
PREFIX=$HOME/usr
LIBDIR=$PREFIX/lib64
export PATH=$PREFIX/bin${PATH:+:$PATH}
export LD_LIBRARY_PATH=$LIBDIR${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
export LIBRARY_PATH=$LIBDIR${LIBRARY_PATH:+:$LIBRARY_PATH}
export CPATH=$PREFIX/include${CPATH:+:$CPATH}


#For autotools packages e.g. xmlrpc-c and zlib

./configure --prefix=$PREFIX --libdir=$LIBDIR [other options here]

*Installing Boost
cd boost_1_55_0
./bootstrap.sh
./b2 --prefix=$PWD --libdir=$PWD/lib64 --layout=tagged link=static,shared
threading=multi,single install || echo FAILURE
./b2 install --prefix=/usr/share/boost-build

export BOOST_ROOT=$PWD
export BOOST_BUILD_PATH=/usr/share/boost-build


In mosesdecoder I entered following command

 ./bjam --with-boost=$BOOST_ROOT --with-srilm=$SRILM

The above commands lead to some error. I am attaching a build.log file.

Kindly go through it and please try to fix it

Regards,
Apurv Nagvenkar
DCST, Goa University
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help. First request to MosesServer very slow

2014-03-11 Thread Barry Haddow

HI Marcos

For command-line Moses, it has to output the sentences in the same order 
as they are input. So if, say, you have 4 threads and provide Moses with 
a large file of input sentences, it will give 1 sentence to each of the 
4 threads. But no output will be produced until sentence 1 is 
translated, so if all sentences are about the same length then there 
will be a delay for the first one, then all others will then be output 
very quickly.

The MTMonkey paper would have used an older version of Moses, which had 
a single global translation options cache, and this could explain the 
difference between your observed behaviour and their reported 
performance. The change in the cache happened some time in 2013, and 
unfortunately seems to have had a negative impact on Moses server. As 
Kenneth has pointed out though, it is something that needs to be fixed 
in Moses server.

cheers - Barry

On 10/03/14 09:22, Marcos Fernandez wrote:
 Hi Marcos

 I think the problem is that the rules (or phrase pairs) are now cached
 on a per thread basis. This is good for command-line Moses as it uses a
 pool of threads, and having per-thread caches means that there is no
 locking on the caches, as there used to be.
 Barry, I am not sure that that is the cause, because in that case my 20
 short sentences file would get translated much faster with command-line
 Moses, however it takes 4.5 seconds to translate it. This is, 2 seconds
 (measured) for loading tables in memory, and the same 2.5 seconds than with
 MosesServer to translate the file. And the behaviour is also the same as
 before, the first sentence in the file takes much longer than the rest.

 However, what you say perhaps could be the cause of the difference in time
 between using a different xmlrpc ServerProxy object for each request
 (probably, in this case xmlrpc executes each request in a different thread),
 or reusing one only ServerProxy for all the requests (where there would be
 only one thread, so it could take advantage of the cache).

 What I understand then, is that the cache stores information of the
 previously translated sentences to accelerate the translation of the next
 ones. But that does not eliminate the problem of the first slower request.
 As you can see, I am finding that issue even in command-line Moses (not with
 the first request, but with the first sentence in a file).

 I am thinking that perhaps I have no problem, and this is just the usual way
 in which Moses works. Just to make sure:

 1. would you say that a time of 2-3 seconds for translating (spa-eng) a
 single sentence (~15-20 words) could be a normal response time (discounting
 the time of loading tables)? (Intel Xeon with 32GB RAM)
 2. if now you write 3 similar sentences in a file, and execute Moses
 (command-line, serial) over this file, would you expect it to take a much
 shorter time (perhaps the half) than the sum of the times for the 3 single
 sentences?

 If the answer is yes to both (specially the second), then I am probably
 worrying in vain. My worries started when I read the MTMonkey paper:
 http://ufal.mff.cuni.cz/~pecina/files/pbml-2013.pdf

 Here the authors use the approach of creating a new ServerProxy instance
 each time a request is sent to MosesServer (the worst case scenario for me),
 and they get great results, so I thought they were not experiencing that
 overhead for every request. But perhaps they just used sentences that get
 translated very fast even with that overhead.

 Well, in the case that this is the usual way of working for Moses, if what
 Kenneth suggests is possible, that would eliminate the overhead almost
 completely for MosesServer. I mean, there would still be an overhead for the
 first request that each thread serves after the server is just launched, but
 never more, because the caches would be filled with useful information from
 that point on. I think that this would be extremely interesting for web
 translating services, or to do web-page translations on the fly.

 However, I don't see a way to avoid that overhead in command-line Moses, as
 it dies after each execution.

 Marcos.


 mosesserver, afaik, creates a new thread for each connection, so it
 can't take advantage of the cache. This is done in the xmlrpc-c library
 so we don't have much control over it. If you dig around in the xmlrpc-c
 documentation (or code!) you might find a way to control the threading
 policy.

 I just spoke to Marcin about the problem, and we're not sure if loading
 the compact phrase table into memory would help, as you still would need
 the higher level cache (in PhraseDictionary). But you could try this anyway.

 cheers - Barry

 On 06/03/14 17:20, Marcos Fernandez wrote:
 Hi, I am having an issue with MosesServer.

 I am using compact phrase and reordering table, and KENLM.

 The problem is this (I'll explain with an example):

 - I have one file with 20 very short sentences. I split and tokenize
 them and send one XMLPRC request per sentence to

Re: [Moses-support] Help. First request to MosesServer very slow

2014-03-10 Thread Marcos Fernandez

 Hi Marcos
 
 I think the problem is that the rules (or phrase pairs) are now cached 
 on a per thread basis. This is good for command-line Moses as it uses a 
 pool of threads, and having per-thread caches means that there is no 
 locking on the caches, as there used to be.

Barry, I am not sure that that is the cause, because in that case my 20
short sentences file would get translated much faster with command-line
Moses, however it takes 4.5 seconds to translate it. This is, 2 seconds
(measured) for loading tables in memory, and the same 2.5 seconds than with
MosesServer to translate the file. And the behaviour is also the same as
before, the first sentence in the file takes much longer than the rest.

However, what you say perhaps could be the cause of the difference in time
between using a different xmlrpc ServerProxy object for each request
(probably, in this case xmlrpc executes each request in a different thread),
or reusing one only ServerProxy for all the requests (where there would be
only one thread, so it could take advantage of the cache).

What I understand then, is that the cache stores information of the
previously translated sentences to accelerate the translation of the next
ones. But that does not eliminate the problem of the first slower request.
As you can see, I am finding that issue even in command-line Moses (not with
the first request, but with the first sentence in a file).

I am thinking that perhaps I have no problem, and this is just the usual way
in which Moses works. Just to make sure:

1. would you say that a time of 2-3 seconds for translating (spa-eng) a
single sentence (~15-20 words) could be a normal response time (discounting
the time of loading tables)? (Intel Xeon with 32GB RAM)
2. if now you write 3 similar sentences in a file, and execute Moses
(command-line, serial) over this file, would you expect it to take a much
shorter time (perhaps the half) than the sum of the times for the 3 single
sentences?

If the answer is yes to both (specially the second), then I am probably
worrying in vain. My worries started when I read the MTMonkey paper:
http://ufal.mff.cuni.cz/~pecina/files/pbml-2013.pdf

Here the authors use the approach of creating a new ServerProxy instance
each time a request is sent to MosesServer (the worst case scenario for me),
and they get great results, so I thought they were not experiencing that
overhead for every request. But perhaps they just used sentences that get
translated very fast even with that overhead.

Well, in the case that this is the usual way of working for Moses, if what
Kenneth suggests is possible, that would eliminate the overhead almost
completely for MosesServer. I mean, there would still be an overhead for the
first request that each thread serves after the server is just launched, but
never more, because the caches would be filled with useful information from
that point on. I think that this would be extremely interesting for web
translating services, or to do web-page translations on the fly.

However, I don't see a way to avoid that overhead in command-line Moses, as
it dies after each execution.

Marcos.


 mosesserver, afaik, creates a new thread for each connection, so it 
 can't take advantage of the cache. This is done in the xmlrpc-c library 
 so we don't have much control over it. If you dig around in the xmlrpc-c 
 documentation (or code!) you might find a way to control the threading 
 policy.
 
 I just spoke to Marcin about the problem, and we're not sure if loading 
 the compact phrase table into memory would help, as you still would need 
 the higher level cache (in PhraseDictionary). But you could try this anyway.
 
 cheers - Barry
 
 On 06/03/14 17:20, Marcos Fernandez wrote:
  Hi, I am having an issue with MosesServer.
 
  I am using compact phrase and reordering table, and KENLM.
 
  The problem is this (I'll explain with an example):
 
  - I have one file with 20 very short sentences. I split and tokenize
  them and send one XMLPRC request per sentence to MosesServer
  - If I create just one XMLRPC ServerProxy instance and I use it to send
  all the requests through it, all the sentences get translated in approx
  2.5 sec. The problem is that the first sentence takes almost 2 seconds
  to get translated, while the other 19 are much faster
  - If I create one ServerProxy instance per request, the translation time
  rises to 30 sec (now every sentence takes almost 2 sec)
 
  I don't understand the reason of that delay for the first request. I
  have followed the source of this delay to the function:
 
  GetTargetPhraseCollectionLEGACY(const Phrase src)
 
  in the file: ...TranslationModel/PhraseDictionary.cpp
 
  It seems that for the first request it's needed  to look for something
  in the phrase table, while for subsequent requests it can be retrieved
  (most of the times) from a cache.
 
  But, as the sentences in my file are not related one to another in any
  way, the information on this cache can not

Re: [Moses-support] Help. First request to MosesServer very slow

2014-03-10 Thread Marcin Junczys-Dowmunt

Hi Marcos,
the command line tool behaviour you are describing is actually 
consistent with what Barry said. You would see this lag before the first 
couple of translations and later it would disappear. Usually you do not 
use the command line tool for only 4 sentences, but for a couple of 
hundreds or thousands. For instance if you would create a file that 
contains the same sentences say 10 times you would only see the lag for 
the first. Your problem is, that MosesSever basically destroys the cache 
after each request, so for each new request it has to fill it again. 
Hence the delay.
Best,
Marcin

W dniu 10.03.2014 09:22, Marcos Fernandez pisze:
 Hi Marcos

 I think the problem is that the rules (or phrase pairs) are now cached
 on a per thread basis. This is good for command-line Moses as it uses a
 pool of threads, and having per-thread caches means that there is no
 locking on the caches, as there used to be.
 Barry, I am not sure that that is the cause, because in that case my 20
 short sentences file would get translated much faster with command-line
 Moses, however it takes 4.5 seconds to translate it. This is, 2 seconds
 (measured) for loading tables in memory, and the same 2.5 seconds than with
 MosesServer to translate the file. And the behaviour is also the same as
 before, the first sentence in the file takes much longer than the rest.

 However, what you say perhaps could be the cause of the difference in time
 between using a different xmlrpc ServerProxy object for each request
 (probably, in this case xmlrpc executes each request in a different thread),
 or reusing one only ServerProxy for all the requests (where there would be
 only one thread, so it could take advantage of the cache).

 What I understand then, is that the cache stores information of the
 previously translated sentences to accelerate the translation of the next
 ones. But that does not eliminate the problem of the first slower request.
 As you can see, I am finding that issue even in command-line Moses (not with
 the first request, but with the first sentence in a file).

 I am thinking that perhaps I have no problem, and this is just the usual way
 in which Moses works. Just to make sure:

 1. would you say that a time of 2-3 seconds for translating (spa-eng) a
 single sentence (~15-20 words) could be a normal response time (discounting
 the time of loading tables)? (Intel Xeon with 32GB RAM)
 2. if now you write 3 similar sentences in a file, and execute Moses
 (command-line, serial) over this file, would you expect it to take a much
 shorter time (perhaps the half) than the sum of the times for the 3 single
 sentences?

 If the answer is yes to both (specially the second), then I am probably
 worrying in vain. My worries started when I read the MTMonkey paper:
 http://ufal.mff.cuni.cz/~pecina/files/pbml-2013.pdf

 Here the authors use the approach of creating a new ServerProxy instance
 each time a request is sent to MosesServer (the worst case scenario for me),
 and they get great results, so I thought they were not experiencing that
 overhead for every request. But perhaps they just used sentences that get
 translated very fast even with that overhead.

 Well, in the case that this is the usual way of working for Moses, if what
 Kenneth suggests is possible, that would eliminate the overhead almost
 completely for MosesServer. I mean, there would still be an overhead for the
 first request that each thread serves after the server is just launched, but
 never more, because the caches would be filled with useful information from
 that point on. I think that this would be extremely interesting for web
 translating services, or to do web-page translations on the fly.

 However, I don't see a way to avoid that overhead in command-line Moses, as
 it dies after each execution.

 Marcos.


 mosesserver, afaik, creates a new thread for each connection, so it
 can't take advantage of the cache. This is done in the xmlrpc-c library
 so we don't have much control over it. If you dig around in the xmlrpc-c
 documentation (or code!) you might find a way to control the threading
 policy.

 I just spoke to Marcin about the problem, and we're not sure if loading
 the compact phrase table into memory would help, as you still would need
 the higher level cache (in PhraseDictionary). But you could try this anyway.

 cheers - Barry

 On 06/03/14 17:20, Marcos Fernandez wrote:
 Hi, I am having an issue with MosesServer.

 I am using compact phrase and reordering table, and KENLM.

 The problem is this (I'll explain with an example):

 - I have one file with 20 very short sentences. I split and tokenize
 them and send one XMLPRC request per sentence to MosesServer
 - If I create just one XMLRPC ServerProxy instance and I use it to send
 all the requests through it, all the sentences get translated in approx
 2.5 sec. The problem is that the first sentence takes almost 2 seconds
 to get translated, while the other 19 are much faster
 - If

[Moses-support] Help. First request to MosesServer very slow

2014-03-06 Thread Marcos Fernandez

Hi, I am having an issue with MosesServer.

I am using compact phrase and reordering table, and KENLM.

The problem is this (I'll explain with an example):

- I have one file with 20 very short sentences. I split and tokenize 
them and send one XMLPRC request per sentence to MosesServer
- If I create just one XMLRPC ServerProxy instance and I use it to send 
all the requests through it, all the sentences get translated in approx 
2.5 sec. The problem is that the first sentence takes almost 2 seconds 
to get translated, while the other 19 are much faster
- If I create one ServerProxy instance per request, the translation time 
rises to 30 sec (now every sentence takes almost 2 sec)

I don't understand the reason of that delay for the first request. I 
have followed the source of this delay to the function:

GetTargetPhraseCollectionLEGACY(const Phrase src)

in the file: ...TranslationModel/PhraseDictionary.cpp

It seems that for the first request it's needed  to look for something 
in the phrase table, while for subsequent requests it can be retrieved 
(most of the times) from a cache.

But, as the sentences in my file are not related one to another in any 
way, the information on this cache can not be sentence-dependent, so why 
wouldn't it be possible for the cache to be preloaded with the 
information needed?

I think that perhaps I have something misconfigured, because I have seen 
other people using the approach of creating one ServerProxy object for 
each XMLRPC request (which would facilitate things a lot for me), so I 
don't think they are experiencing this overhead. Perhaps using the 
compact formats can have something to do with it?

Any help would be much appreciated. I paste below my moses.ini, if that 
helps:

Thanks :)

### MOSES CONFIG FILE ###
###

# input factors
[input-factors]
0

# mapping steps
[mapping]
0 T 0

# translation tables: table type (hierarchical(0), textual (0), binary 
(1)), source-factors, target-factors, number of scores, file
# OLD FORMAT is still handled for back-compatibility
# OLD FORMAT translation tables: source-factors, target-factors, number 
of scores, file
# OLD FORMAT a binary table type (1) is assumed
[ttable-file]
12 0 0 5 /opt/moses-compiling/modelos/es-en/phrase-model/phrase-table

# no generation models, no generation-file section

# language models: type(srilm/irstlm), factors, order, file
[lmodel-file]
8 0 5 
/opt/moses-compiling/modelos/es-en/lm/13-19-03gen_intec_head8m_sb5LM.kenlm


# limit on how many phrase translations e for each phrase f are loaded
# 0 = all elements loaded
[ttable-limit]
10

# distortion (reordering) files
[distortion-file]
0-0 wbe-msd-bidirectional-fe-allff 6 
/opt/moses-compiling/modelos/es-en/phrase-model/reordering-table

# distortion (reordering) weight
[weight-d]
0.097107
0.150373
-0.0551767
-0.0307787
0.114613
0.214587
0.0467398

# language model weights
[weight-l]
0.0442748


# translation model weights
[weight-t]
0.00370888
0.0425665
0.0719956
0.0202699
0.071147

# no generation models, no weight-generation section

# word penalty
[weight-w]
0.0366626

[distortion-limit]
6

[v]
0


-- 
Marcos Fernández López
Project Technician
Centro de Investigación en Tecnoloxías da Información (CITIUS)
Universidade de Santiago de Compostela
15782 Santiago de Compostela, Spain

e-mail: marcos.fernandez.lo...@usc.es
Phone:(+34) 881 816444

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help. First request to MosesServer very slow

2014-03-06 Thread Marcin Junczys-Dowmunt

Hi Marcos,
What you can do, if you want to rule out the compact data structures are 
the issue, is load the tables at start-up into memory (if you have 
enough memory):

That would be the following additional options to your ini-file:

[minphr-memory]
1

[minlexr-memory]
1

However your problem looks client-side related. Why should the creation 
of client-side requests affect translation time on the side of the 
server? I understand the same server instance is running all the time?

Best,
Marcin

W dniu 06.03.2014 17:20, Marcos Fernandez pisze:
 Hi, I am having an issue with MosesServer.

 I am using compact phrase and reordering table, and KENLM.

 The problem is this (I'll explain with an example):

 - I have one file with 20 very short sentences. I split and tokenize
 them and send one XMLPRC request per sentence to MosesServer
 - If I create just one XMLRPC ServerProxy instance and I use it to send
 all the requests through it, all the sentences get translated in approx
 2.5 sec. The problem is that the first sentence takes almost 2 seconds
 to get translated, while the other 19 are much faster
 - If I create one ServerProxy instance per request, the translation time
 rises to 30 sec (now every sentence takes almost 2 sec)

 I don't understand the reason of that delay for the first request. I
 have followed the source of this delay to the function:

 GetTargetPhraseCollectionLEGACY(const Phrase src)

 in the file: ...TranslationModel/PhraseDictionary.cpp

 It seems that for the first request it's needed  to look for something
 in the phrase table, while for subsequent requests it can be retrieved
 (most of the times) from a cache.

 But, as the sentences in my file are not related one to another in any
 way, the information on this cache can not be sentence-dependent, so why
 wouldn't it be possible for the cache to be preloaded with the
 information needed?

 I think that perhaps I have something misconfigured, because I have seen
 other people using the approach of creating one ServerProxy object for
 each XMLRPC request (which would facilitate things a lot for me), so I
 don't think they are experiencing this overhead. Perhaps using the
 compact formats can have something to do with it?

 Any help would be much appreciated. I paste below my moses.ini, if that
 helps:

 Thanks :)

 ### MOSES CONFIG FILE ###
 ###

 # input factors
 [input-factors]
 0

 # mapping steps
 [mapping]
 0 T 0

 # translation tables: table type (hierarchical(0), textual (0), binary
 (1)), source-factors, target-factors, number of scores, file
 # OLD FORMAT is still handled for back-compatibility
 # OLD FORMAT translation tables: source-factors, target-factors, number
 of scores, file
 # OLD FORMAT a binary table type (1) is assumed
 [ttable-file]
 12 0 0 5 /opt/moses-compiling/modelos/es-en/phrase-model/phrase-table

 # no generation models, no generation-file section

 # language models: type(srilm/irstlm), factors, order, file
 [lmodel-file]
 8 0 5
 /opt/moses-compiling/modelos/es-en/lm/13-19-03gen_intec_head8m_sb5LM.kenlm


 # limit on how many phrase translations e for each phrase f are loaded
 # 0 = all elements loaded
 [ttable-limit]
 10

 # distortion (reordering) files
 [distortion-file]
 0-0 wbe-msd-bidirectional-fe-allff 6
 /opt/moses-compiling/modelos/es-en/phrase-model/reordering-table

 # distortion (reordering) weight
 [weight-d]
 0.097107
 0.150373
 -0.0551767
 -0.0307787
 0.114613
 0.214587
 0.0467398

 # language model weights
 [weight-l]
 0.0442748


 # translation model weights
 [weight-t]
 0.00370888
 0.0425665
 0.0719956
 0.0202699
 0.071147

 # no generation models, no weight-generation section

 # word penalty
 [weight-w]
 0.0366626

 [distortion-limit]
 6

 [v]
 0



___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help. First request to MosesServer very slow

2014-03-06 Thread Kenneth Heafield

Hi,

In my view, the threading design of the server is a bug.  How about a
producer-consumer queue with multiple producers (a client connection)
and consumers (decoding threads).  Each client connection owns a
producer-consumer queue as a return channel so that decoder threads can
return their result.  Or we could use boost futures.

Kenneth

On 03/06/14 10:30, Barry Haddow wrote:
 Hi Marcos
 
 I think the problem is that the rules (or phrase pairs) are now cached 
 on a per thread basis. This is good for command-line Moses as it uses a 
 pool of threads, and having per-thread caches means that there is no 
 locking on the caches, as there used to be.
 
 mosesserver, afaik, creates a new thread for each connection, so it 
 can't take advantage of the cache. This is done in the xmlrpc-c library 
 so we don't have much control over it. If you dig around in the xmlrpc-c 
 documentation (or code!) you might find a way to control the threading 
 policy.
 
 I just spoke to Marcin about the problem, and we're not sure if loading 
 the compact phrase table into memory would help, as you still would need 
 the higher level cache (in PhraseDictionary). But you could try this anyway.
 
 cheers - Barry
 
 On 06/03/14 17:20, Marcos Fernandez wrote:
 Hi, I am having an issue with MosesServer.

 I am using compact phrase and reordering table, and KENLM.

 The problem is this (I'll explain with an example):

 - I have one file with 20 very short sentences. I split and tokenize
 them and send one XMLPRC request per sentence to MosesServer
 - If I create just one XMLRPC ServerProxy instance and I use it to send
 all the requests through it, all the sentences get translated in approx
 2.5 sec. The problem is that the first sentence takes almost 2 seconds
 to get translated, while the other 19 are much faster
 - If I create one ServerProxy instance per request, the translation time
 rises to 30 sec (now every sentence takes almost 2 sec)

 I don't understand the reason of that delay for the first request. I
 have followed the source of this delay to the function:

 GetTargetPhraseCollectionLEGACY(const Phrase src)

 in the file: ...TranslationModel/PhraseDictionary.cpp

 It seems that for the first request it's needed  to look for something
 in the phrase table, while for subsequent requests it can be retrieved
 (most of the times) from a cache.

 But, as the sentences in my file are not related one to another in any
 way, the information on this cache can not be sentence-dependent, so why
 wouldn't it be possible for the cache to be preloaded with the
 information needed?

 I think that perhaps I have something misconfigured, because I have seen
 other people using the approach of creating one ServerProxy object for
 each XMLRPC request (which would facilitate things a lot for me), so I
 don't think they are experiencing this overhead. Perhaps using the
 compact formats can have something to do with it?

 Any help would be much appreciated. I paste below my moses.ini, if that
 helps:

 Thanks :)

 ### MOSES CONFIG FILE ###
 ###

 # input factors
 [input-factors]
 0

 # mapping steps
 [mapping]
 0 T 0

 # translation tables: table type (hierarchical(0), textual (0), binary
 (1)), source-factors, target-factors, number of scores, file
 # OLD FORMAT is still handled for back-compatibility
 # OLD FORMAT translation tables: source-factors, target-factors, number
 of scores, file
 # OLD FORMAT a binary table type (1) is assumed
 [ttable-file]
 12 0 0 5 /opt/moses-compiling/modelos/es-en/phrase-model/phrase-table

 # no generation models, no generation-file section

 # language models: type(srilm/irstlm), factors, order, file
 [lmodel-file]
 8 0 5
 /opt/moses-compiling/modelos/es-en/lm/13-19-03gen_intec_head8m_sb5LM.kenlm


 # limit on how many phrase translations e for each phrase f are loaded
 # 0 = all elements loaded
 [ttable-limit]
 10

 # distortion (reordering) files
 [distortion-file]
 0-0 wbe-msd-bidirectional-fe-allff 6
 /opt/moses-compiling/modelos/es-en/phrase-model/reordering-table

 # distortion (reordering) weight
 [weight-d]
 0.097107
 0.150373
 -0.0551767
 -0.0307787
 0.114613
 0.214587
 0.0467398

 # language model weights
 [weight-l]
 0.0442748


 # translation model weights
 [weight-t]
 0.00370888
 0.0425665
 0.0719956
 0.0202699
 0.071147

 # no generation models, no weight-generation section

 # word penalty
 [weight-w]
 0.0366626

 [distortion-limit]
 6

 [v]
 0


 
 
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help. First request to MosesServer very slow

2014-03-06 Thread Barry Haddow

Hi Marcos

I think the problem is that the rules (or phrase pairs) are now cached 
on a per thread basis. This is good for command-line Moses as it uses a 
pool of threads, and having per-thread caches means that there is no 
locking on the caches, as there used to be.

mosesserver, afaik, creates a new thread for each connection, so it 
can't take advantage of the cache. This is done in the xmlrpc-c library 
so we don't have much control over it. If you dig around in the xmlrpc-c 
documentation (or code!) you might find a way to control the threading 
policy.

I just spoke to Marcin about the problem, and we're not sure if loading 
the compact phrase table into memory would help, as you still would need 
the higher level cache (in PhraseDictionary). But you could try this anyway.

cheers - Barry

On 06/03/14 17:20, Marcos Fernandez wrote:
 Hi, I am having an issue with MosesServer.

 I am using compact phrase and reordering table, and KENLM.

 The problem is this (I'll explain with an example):

 - I have one file with 20 very short sentences. I split and tokenize
 them and send one XMLPRC request per sentence to MosesServer
 - If I create just one XMLRPC ServerProxy instance and I use it to send
 all the requests through it, all the sentences get translated in approx
 2.5 sec. The problem is that the first sentence takes almost 2 seconds
 to get translated, while the other 19 are much faster
 - If I create one ServerProxy instance per request, the translation time
 rises to 30 sec (now every sentence takes almost 2 sec)

 I don't understand the reason of that delay for the first request. I
 have followed the source of this delay to the function:

 GetTargetPhraseCollectionLEGACY(const Phrase src)

 in the file: ...TranslationModel/PhraseDictionary.cpp

 It seems that for the first request it's needed  to look for something
 in the phrase table, while for subsequent requests it can be retrieved
 (most of the times) from a cache.

 But, as the sentences in my file are not related one to another in any
 way, the information on this cache can not be sentence-dependent, so why
 wouldn't it be possible for the cache to be preloaded with the
 information needed?

 I think that perhaps I have something misconfigured, because I have seen
 other people using the approach of creating one ServerProxy object for
 each XMLRPC request (which would facilitate things a lot for me), so I
 don't think they are experiencing this overhead. Perhaps using the
 compact formats can have something to do with it?

 Any help would be much appreciated. I paste below my moses.ini, if that
 helps:

 Thanks :)

 ### MOSES CONFIG FILE ###
 ###

 # input factors
 [input-factors]
 0

 # mapping steps
 [mapping]
 0 T 0

 # translation tables: table type (hierarchical(0), textual (0), binary
 (1)), source-factors, target-factors, number of scores, file
 # OLD FORMAT is still handled for back-compatibility
 # OLD FORMAT translation tables: source-factors, target-factors, number
 of scores, file
 # OLD FORMAT a binary table type (1) is assumed
 [ttable-file]
 12 0 0 5 /opt/moses-compiling/modelos/es-en/phrase-model/phrase-table

 # no generation models, no generation-file section

 # language models: type(srilm/irstlm), factors, order, file
 [lmodel-file]
 8 0 5
 /opt/moses-compiling/modelos/es-en/lm/13-19-03gen_intec_head8m_sb5LM.kenlm


 # limit on how many phrase translations e for each phrase f are loaded
 # 0 = all elements loaded
 [ttable-limit]
 10

 # distortion (reordering) files
 [distortion-file]
 0-0 wbe-msd-bidirectional-fe-allff 6
 /opt/moses-compiling/modelos/es-en/phrase-model/reordering-table

 # distortion (reordering) weight
 [weight-d]
 0.097107
 0.150373
 -0.0551767
 -0.0307787
 0.114613
 0.214587
 0.0467398

 # language model weights
 [weight-l]
 0.0442748


 # translation model weights
 [weight-t]
 0.00370888
 0.0425665
 0.0719956
 0.0202699
 0.071147

 # no generation models, no weight-generation section

 # word penalty
 [weight-w]
 0.0366626

 [distortion-limit]
 6

 [v]
 0




-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help. First request to MosesServer very slow

2014-03-06 Thread Barry Haddow

Hi

I agree that it's a bad design, but I'm not clear if it can be fixed 
without changing to a different xmlrpc library

cheers - Barry

On 06/03/14 18:40, Kenneth Heafield wrote:
 Hi,

   In my view, the threading design of the server is a bug.  How about a
 producer-consumer queue with multiple producers (a client connection)
 and consumers (decoding threads).  Each client connection owns a
 producer-consumer queue as a return channel so that decoder threads can
 return their result.  Or we could use boost futures.

 Kenneth

 On 03/06/14 10:30, Barry Haddow wrote:
 Hi Marcos

 I think the problem is that the rules (or phrase pairs) are now cached
 on a per thread basis. This is good for command-line Moses as it uses a
 pool of threads, and having per-thread caches means that there is no
 locking on the caches, as there used to be.

 mosesserver, afaik, creates a new thread for each connection, so it
 can't take advantage of the cache. This is done in the xmlrpc-c library
 so we don't have much control over it. If you dig around in the xmlrpc-c
 documentation (or code!) you might find a way to control the threading
 policy.

 I just spoke to Marcin about the problem, and we're not sure if loading
 the compact phrase table into memory would help, as you still would need
 the higher level cache (in PhraseDictionary). But you could try this anyway.

 cheers - Barry

 On 06/03/14 17:20, Marcos Fernandez wrote:
 Hi, I am having an issue with MosesServer.

 I am using compact phrase and reordering table, and KENLM.

 The problem is this (I'll explain with an example):

 - I have one file with 20 very short sentences. I split and tokenize
 them and send one XMLPRC request per sentence to MosesServer
 - If I create just one XMLRPC ServerProxy instance and I use it to send
 all the requests through it, all the sentences get translated in approx
 2.5 sec. The problem is that the first sentence takes almost 2 seconds
 to get translated, while the other 19 are much faster
 - If I create one ServerProxy instance per request, the translation time
 rises to 30 sec (now every sentence takes almost 2 sec)

 I don't understand the reason of that delay for the first request. I
 have followed the source of this delay to the function:

 GetTargetPhraseCollectionLEGACY(const Phrase src)

 in the file: ...TranslationModel/PhraseDictionary.cpp

 It seems that for the first request it's needed  to look for something
 in the phrase table, while for subsequent requests it can be retrieved
 (most of the times) from a cache.

 But, as the sentences in my file are not related one to another in any
 way, the information on this cache can not be sentence-dependent, so why
 wouldn't it be possible for the cache to be preloaded with the
 information needed?

 I think that perhaps I have something misconfigured, because I have seen
 other people using the approach of creating one ServerProxy object for
 each XMLRPC request (which would facilitate things a lot for me), so I
 don't think they are experiencing this overhead. Perhaps using the
 compact formats can have something to do with it?

 Any help would be much appreciated. I paste below my moses.ini, if that
 helps:

 Thanks :)

 ### MOSES CONFIG FILE ###
 ###

 # input factors
 [input-factors]
 0

 # mapping steps
 [mapping]
 0 T 0

 # translation tables: table type (hierarchical(0), textual (0), binary
 (1)), source-factors, target-factors, number of scores, file
 # OLD FORMAT is still handled for back-compatibility
 # OLD FORMAT translation tables: source-factors, target-factors, number
 of scores, file
 # OLD FORMAT a binary table type (1) is assumed
 [ttable-file]
 12 0 0 5 /opt/moses-compiling/modelos/es-en/phrase-model/phrase-table

 # no generation models, no generation-file section

 # language models: type(srilm/irstlm), factors, order, file
 [lmodel-file]
 8 0 5
 /opt/moses-compiling/modelos/es-en/lm/13-19-03gen_intec_head8m_sb5LM.kenlm


 # limit on how many phrase translations e for each phrase f are loaded
 # 0 = all elements loaded
 [ttable-limit]
 10

 # distortion (reordering) files
 [distortion-file]
 0-0 wbe-msd-bidirectional-fe-allff 6
 /opt/moses-compiling/modelos/es-en/phrase-model/reordering-table

 # distortion (reordering) weight
 [weight-d]
 0.097107
 0.150373
 -0.0551767
 -0.0307787
 0.114613
 0.214587
 0.0467398

 # language model weights
 [weight-l]
 0.0442748


 # translation model weights
 [weight-t]
 0.00370888
 0.0425665
 0.0719956
 0.0202699
 0.071147

 # no generation models, no weight-generation section

 # word penalty
 [weight-w]
 0.0366626

 [distortion-limit]
 6

 [v]
 0



 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support



-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

___
Moses-support mailing list

Re: [Moses-support] Help. First request to MosesServer very slow

2014-03-06 Thread Kenneth Heafield

Hi,

The issue is that the xmlrpc library launches a thread for each client
connection.  Nothing prevents these threads from then contacting a
central pool of decoder threads.

Kenneth

On 03/06/14 11:09, Barry Haddow wrote:
 Hi
 
 I agree that it's a bad design, but I'm not clear if it can be fixed
 without changing to a different xmlrpc library
 
 cheers - Barry
 
 On 06/03/14 18:40, Kenneth Heafield wrote:
 Hi,

 In my view, the threading design of the server is a bug.  How about a
 producer-consumer queue with multiple producers (a client connection)
 and consumers (decoding threads).  Each client connection owns a
 producer-consumer queue as a return channel so that decoder threads can
 return their result.  Or we could use boost futures.

 Kenneth

 On 03/06/14 10:30, Barry Haddow wrote:
 Hi Marcos

 I think the problem is that the rules (or phrase pairs) are now cached
 on a per thread basis. This is good for command-line Moses as it uses a
 pool of threads, and having per-thread caches means that there is no
 locking on the caches, as there used to be.

 mosesserver, afaik, creates a new thread for each connection, so it
 can't take advantage of the cache. This is done in the xmlrpc-c library
 so we don't have much control over it. If you dig around in the xmlrpc-c
 documentation (or code!) you might find a way to control the threading
 policy.

 I just spoke to Marcin about the problem, and we're not sure if loading
 the compact phrase table into memory would help, as you still would need
 the higher level cache (in PhraseDictionary). But you could try this
 anyway.

 cheers - Barry

 On 06/03/14 17:20, Marcos Fernandez wrote:
 Hi, I am having an issue with MosesServer.

 I am using compact phrase and reordering table, and KENLM.

 The problem is this (I'll explain with an example):

 - I have one file with 20 very short sentences. I split and tokenize
 them and send one XMLPRC request per sentence to MosesServer
 - If I create just one XMLRPC ServerProxy instance and I use it to send
 all the requests through it, all the sentences get translated in approx
 2.5 sec. The problem is that the first sentence takes almost 2 seconds
 to get translated, while the other 19 are much faster
 - If I create one ServerProxy instance per request, the translation
 time
 rises to 30 sec (now every sentence takes almost 2 sec)

 I don't understand the reason of that delay for the first request. I
 have followed the source of this delay to the function:

 GetTargetPhraseCollectionLEGACY(const Phrase src)

 in the file: ...TranslationModel/PhraseDictionary.cpp

 It seems that for the first request it's needed  to look for something
 in the phrase table, while for subsequent requests it can be retrieved
 (most of the times) from a cache.

 But, as the sentences in my file are not related one to another in any
 way, the information on this cache can not be sentence-dependent, so
 why
 wouldn't it be possible for the cache to be preloaded with the
 information needed?

 I think that perhaps I have something misconfigured, because I have
 seen
 other people using the approach of creating one ServerProxy object for
 each XMLRPC request (which would facilitate things a lot for me), so I
 don't think they are experiencing this overhead. Perhaps using the
 compact formats can have something to do with it?

 Any help would be much appreciated. I paste below my moses.ini, if that
 helps:

 Thanks :)

 ### MOSES CONFIG FILE ###
 ###

 # input factors
 [input-factors]
 0

 # mapping steps
 [mapping]
 0 T 0

 # translation tables: table type (hierarchical(0), textual (0), binary
 (1)), source-factors, target-factors, number of scores, file
 # OLD FORMAT is still handled for back-compatibility
 # OLD FORMAT translation tables: source-factors, target-factors, number
 of scores, file
 # OLD FORMAT a binary table type (1) is assumed
 [ttable-file]
 12 0 0 5 /opt/moses-compiling/modelos/es-en/phrase-model/phrase-table

 # no generation models, no generation-file section

 # language models: type(srilm/irstlm), factors, order, file
 [lmodel-file]
 8 0 5
 /opt/moses-compiling/modelos/es-en/lm/13-19-03gen_intec_head8m_sb5LM.kenlm



 # limit on how many phrase translations e for each phrase f are loaded
 # 0 = all elements loaded
 [ttable-limit]
 10

 # distortion (reordering) files
 [distortion-file]
 0-0 wbe-msd-bidirectional-fe-allff 6
 /opt/moses-compiling/modelos/es-en/phrase-model/reordering-table

 # distortion (reordering) weight
 [weight-d]
 0.097107
 0.150373
 -0.0551767
 -0.0307787
 0.114613
 0.214587
 0.0467398

 # language model weights
 [weight-l]
 0.0442748


 # translation model weights
 [weight-t]
 0.00370888
 0.0425665
 0.0719956
 0.0202699
 0.071147

 # no generation models, no weight-generation section

 # word penalty
 [weight-w]
 0.0366626

 [distortion-limit]
 6

 [v]
 0



 ___
 Moses-support mailing list
 Moses-support@mit.edu

[Moses-support] HELP

2014-02-12 Thread Mohadeseh Sheikhoveisi

Hi,

I tried to install srilm via srilm_1.5.6-1nlp1~0gutsy1_i386.deb with
this command on ubuntu 7.10:

#dpkg -i srilm_1.5.6-1nlp1~0gutsy1_i386.deb

I see this error in terminal:

(Reading database ... 90886 files and directories currently installed.)
Preparing to replace srilm 1.5.6-1nlp1~0gutsy1 (using
srilm_1.5.6-1nlp1~0gutsy1_i386.deb) ...
Unpacking replacement srilm ...
dpkg: dependency problems prevent configuration of srilm:
 srilm depends on tcl8.3 (= 8.3.5); however:
  Package tcl8.3 is not installed.
 srilm depends on csh; however:
  Package csh is not installed.
 srilm depends on gawk; however:
  Package gawk is not installed.
 srilm depends on tcl8.3; however:
  Package tcl8.3 is not installed.
dpkg: error processing srilm (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 srilm

Then I try run apt-get install build-essential subversion zlib1g-dev
autoconf automake g++ make gawk gzip tcl8.4 tcl8.4-dev tcsh tcl-dev
tclx8.4 tclx8.4-dev libtool libboost-dev libboost-graph-dev
libboost-iostreams-dev libboost-program-options-dev gcc-4.1 g++-4.1
git-core
and following package could not install :
subversion
zlib1g.dev
autoconf
automake
gawk
tcl8.4
tcl8.4-dev
tcsh
tcl-dev
tclx8.4
tclx8.4-dev libtool
libboost-dev
libboost-graph-dev
libboost-iostreams-dev
libboost-program-options-dev
git-core

Finally, I try to add lines to /etc/apt/sources.list and try again to
run above command and see that error again.

please help me.

Best Regards,
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] HELP

2014-02-12 Thread Germán Sanchis Trilles


Hi Mohadeseh,

have you tried to

apt-get update  # update the apt-get repository with the info in the 
sources.list file
apt-get upgrade # upgrade the system with the packages available in the 
apt repository
apt-get -f install  # fix the installation, removing broken packages and 
cleaning dependencies

It looks like your dpkg system is somewhat broken, so those commands 
should clean-up your dpkg.


Cheers,

Germán



On Wed, 12 Feb 2014, Mohadeseh Sheikhoveisi wrote:


Hi,

I tried to install srilm via srilm_1.5.6-1nlp1~0gutsy1_i386.deb with
this command on ubuntu 7.10:

#dpkg -i srilm_1.5.6-1nlp1~0gutsy1_i386.deb

I see this error in terminal:

(Reading database ... 90886 files and directories currently installed.)
Preparing to replace srilm 1.5.6-1nlp1~0gutsy1 (using
srilm_1.5.6-1nlp1~0gutsy1_i386.deb) ...
Unpacking replacement srilm ...
dpkg: dependency problems prevent configuration of srilm:
srilm depends on tcl8.3 (= 8.3.5); however:
 Package tcl8.3 is not installed.
srilm depends on csh; however:
 Package csh is not installed.
srilm depends on gawk; however:
 Package gawk is not installed.
srilm depends on tcl8.3; however:
 Package tcl8.3 is not installed.
dpkg: error processing srilm (--install):
dependency problems - leaving unconfigured
Errors were encountered while processing:
srilm

Then I try run apt-get install build-essential subversion zlib1g-dev
autoconf automake g++ make gawk gzip tcl8.4 tcl8.4-dev tcsh tcl-dev
tclx8.4 tclx8.4-dev libtool libboost-dev libboost-graph-dev
libboost-iostreams-dev libboost-program-options-dev gcc-4.1 g++-4.1
git-core
and following package could not install :
subversion
zlib1g.dev
autoconf
automake
gawk
tcl8.4
tcl8.4-dev
tcsh
tcl-dev
tclx8.4
tclx8.4-dev libtool
libboost-dev
libboost-graph-dev
libboost-iostreams-dev
libboost-program-options-dev
git-core

Finally, I try to add lines to /etc/apt/sources.list and try again to
run above command and see that error again.

please help me.

Best Regards,
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] help

2014-01-31 Thread Leila Tavakoli

HI,
I want to tune my development set and i want to use -mertdir. so i use
mert-moses-new.pl . i use this command:

./mert-moses-new.pl /mnt/hgfs/H/dev/en/en.txt /mnt/hgfs/H/dev/ref/
/home/msho/Desktop/Shafagh/moses-default/bin/moses
/mnt/hgfs/H/Train150/model/moses.ini -working-dir
/mnt/hgfs/H/Train150/model/mert -rootdir
/home/msho/Desktop/Shafagh/moses-default/scripts -mertdir
/home/msho/Desktop/Shafagh/moses-default/bin /home/msho/Desktop/log.out
2/home/msho/Desktop/log.out

terminal shows error. this error printed in log.out.
log.out file attached
please help me

best regards,

-- 

Leila Tavakoli

Natural Language Processing lab,

School of Electrical and Computer Eng,

Tehran University

http://ece.ut.ac.ir/nlp


log.out
Description: Binary data
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help

2014-01-31 Thread Rico Sennrich

Leila Tavakoli leila.tavakoliii@... writes:

 
 
 Hi,
 I need to tune my English-Persian development set, and i use this command: 
 
 
 root at
ubuntu:/home/msho/Desktop/Shafagh/moses-default/scripts/training#
./mert-moses.pl /mnt/hgfs/H/dev/en/en.txt /mnt/hgfs/H/dev/ref/
/home/msho/Desktop/Shafagh/moses-default/bin/moses
/mnt/hgfs/H/Train150/model/moses.ini -working-dir
/mnt/hgfs/H/Train150/model/mert -rootdir
/home/msho/Desktop/Shafagh/moses-default/scripts -mertdir
/home/msho/Desktop/Shafagh/moses-default/bin
 
 
 
 run1 is incomplete and terminal displays this error:
 
 
 .../moses-default/bin/moses: 3: Syntax error: Unterminated quoted string
 Exit code: 2
 
 Failed to run moses with the config filtered/moses.ini at
./mert-moses.pl line 1271.
 

Hi Leila,

the error looks like something you get from a bash script, not from a binary
program. Can you please confirm that you successfully compiled moses with
bjam, and that /home/msho/Desktop/Shafagh/moses-default/bin/moses is
correctly recognized as a binary? You can run the 'file' command to test
this. Here's what I get:

file /data/tools/mosesdecoder/bin/moses
/data/tools/mosesdecoder/bin/moses: ELF 64-bit LSB executable, x86-64,
version 1 (GNU/Linux), dynamically linked (uses shared libs), for GNU/Linux
2.6.24, BuildID[sha1]=0x3aab40a7b36b5041b0a57e08f015265e28fb6006, not stripped

best wishes,
Rico


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] help

2014-01-30 Thread Leila Tavakoli

Hi,
I need to tune my English-Persian development set, and i use this command:

root@ubuntu:/home/msho/Desktop/Shafagh/moses-default/scripts/training# ./
mert-moses.pl /mnt/hgfs/H/dev/en/en.txt /mnt/hgfs/H/dev/ref/
/home/msho/Desktop/Shafagh/moses-default/bin/moses
/mnt/hgfs/H/Train150/model/moses.ini -working-dir
/mnt/hgfs/H/Train150/model/mert -rootdir
/home/msho/Desktop/Shafagh/moses-default/scripts -mertdir
/home/msho/Desktop/Shafagh/moses-default/bin

run1 is incomplete and terminal displays this error:

.../moses-default/bin/moses: 3: Syntax error: Unterminated quoted string
Exit code: 2
Failed to run moses with the config filtered/moses.ini at ./mert-moses.pl line
1271.


please help me.

best regards,

-- 

Leila Tavakoli

Natural Language Processing lab,

School of Electrical and Computer Eng,

Tehran University

http://ece.ut.ac.ir/nlp
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help

2014-01-30 Thread Tom Hoar

Can you send the error log when you run the Moses binary with this 
moses.ini file, and include the moses.ini file? The command line would 
look like this:


  $ /home/msho/Desktop/Shafagh/moses-default/bin/moses -f 
/mnt/hgfs/H/Train150/model/moses.ini




On 01/30/2014 07:11 PM, Leila Tavakoli wrote:

Hi,
I need to tune my English-Persian development set, and i use this 
command:


root@ubuntu:/home/msho/Desktop/Shafagh/moses-default/scripts/training# 
./mert-moses.pl http://mert-moses.pl/ /mnt/hgfs/H/dev/en/en.txt 
/mnt/hgfs/H/dev/ref/ 
/home/msho/Desktop/Shafagh/moses-default/bin/moses 
/mnt/hgfs/H/Train150/model/moses.ini -working-dir 
/mnt/hgfs/H/Train150/model/mert -rootdir 
/home/msho/Desktop/Shafagh/moses-default/scripts -mertdir 
/home/msho/Desktop/Shafagh/moses-default/bin


run1 is incomplete and terminal displays this error:

.../moses-default/bin/moses: 3: Syntax error: Unterminated quoted string
Exit code: 2
Failed to run moses with the config filtered/moses.ini at 
./mert-moses.pl http://mert-moses.pl/ line 1271.



please help me.

best regards,

--

Leila Tavakoli

Natural Language Processing lab,

School of Electrical and Computer Eng,

Tehran University

http://ece.ut.ac.ir/nlp



___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Help

2014-01-29 Thread Mohadeseh Sheikhoveisi

Hi,
I need to tune my English-Persian development set and I use following
command:
root@ubuntu:/home/msho/Desktop/Shafagh/moses-default/scripts/training# ./
mert-moses.pl /mnt/hgfs/H/dev/en/en.txt /mnt/hgfs/H/dev/ref/
/home/msho/Desktop/Shafagh/moses-default/bin/moses
/mnt/hgfs/H/Train150/model/moses.ini -working-dir
/mnt/hgfs/H/Train150/model/mert -rootdir
/home/msho/Desktop/Shafagh/moses-default/scripts
-mertdir /home/msho/Desktop/Shafagh/moses-default/bin

I get this Error:

.../moses-default/bin/moses: 3: Syntax error: Unterminated quoted string
Exit code: 2
Failed to run moses with the config filtered/moses.ini at ./mert-moses.pl line
1271.

please help me.

Best Regards,
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Help Needed in Developing Machine Translation System

2013-12-09 Thread Asad A.Malik

Hi All,
The reason I am emailing is that I want help and guidance in developing Machine 
Translation
system.

 
I am currently enrolled
in MS. And for my semester project I have to develop one
of the Machine Translation system. I have already taken time in submitting the
proposal just because I in a situation that I don't know how and from where 
should
I start. I have studied literature of the basic Machine Translation systems
(Rule Based, Statistical and Example Based) and wanted to develop the system
using these mentioned techniques. 
 
Now the problem is that
I don't know any thing about the tools via which I will have to develop it.
I've uses MOSES a little bit but can't understand on it.Also I don't know what
resources will I be required in the development. Some times I think that the
development of Rule Based will be easy and some times think that statistical
will be easy. But actual I want (any one of three mentioned techniques)
that is easy to develop and don't require much of human effort. 
 
Kindly help me in
selecting the technique and show me the right path. I will really appreciate
your help.
  
Regards 
 
Asad A.Malik
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help in installing moses

2013-09-27 Thread Kenneth Heafield

Hi,

Try recompiling without spaces in the directory name.  If that fixes
it, at least we'll know where the issue is.

Kenneth

On 09/27/13 11:41, subhash ganeshan wrote:
 Hello,
 Recently while installing moses on my system Ubuntu12.04, I ran
 into some installation issues.
 I am attaching a log file of the same with this mail.
 so, kindly go through the same and help me in installing the same.
 
 The command I used while installing was ./bjam -j4.
 
 Thanks and regards,
 Subhash
 
 http://sigads.rediff.com/RealMedia/ads/click_nx.ads/www.rediffmail.com/signatureline.htm@Middle?
 *Ganesha offers* Company email  website (*FREE*) at your own domain
 (*FREE*) - *KNOW MORE *
 http://track.rediff.com/click?url=___http://businessemail.rediff.com/company-email-hosting-services?sc_cid=sign-060913___cmp=hostlnk=sign-060913nsrv1=host
 
 
 
 
 
 
 
 
 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support
 
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Help Request

2013-09-05 Thread zhengshun999999

Thank you for helping diagnose the building problem!notice: found boost-build.jam at /usr/share/boost-build/boost-build.jam
notice: loading Boost.Build from /usr/share/boost-build/kernel
notice: Searching /etc /home/zs /usr/share/boost-build/kernel 
/usr/share/boost-build /usr/share/boost-build/kernel 
/usr/share/boost-build/util /usr/share/boost-build/build 
/usr/share/boost-build/tools /usr/share/boost-build/contrib 
/usr/share/boost-build/. for site-config configuration file site-config.jam .
notice: Configuration file site-config.jam not found in /etc /home/zs 
/usr/share/boost-build/kernel /usr/share/boost-build 
/usr/share/boost-build/kernel /usr/share/boost-build/util 
/usr/share/boost-build/build /usr/share/boost-build/tools 
/usr/share/boost-build/contrib /usr/share/boost-build/. .
notice: Searching /home/zs /usr/share/boost-build/kernel /usr/share/boost-build 
/usr/share/boost-build/kernel /usr/share/boost-build/util 
/usr/share/boost-build/build /usr/share/boost-build/tools 
/usr/share/boost-build/contrib /usr/share/boost-build/. for user-config 
configuration file user-config.jam .
notice: Configuration file user-config.jam not found in /home/zs 
/usr/share/boost-build/kernel /usr/share/boost-build 
/usr/share/boost-build/kernel /usr/share/boost-build/util 
/usr/share/boost-build/build /usr/share/boost-build/tools 
/usr/share/boost-build/contrib /usr/share/boost-build/. .

file /home/zs/mosesdecoder/previous.sh
#!/bin/sh
/usr/bin/bjam --help --debug-configuration -d2


bash -c g++ -lSegFault -x c++ - 'int main() {}' -o 
/home/zs/mosesdecoder/dummy /dev/null 2/dev/null  rm 
/home/zs/mosesdecoder/dummy 2/dev/null
 0
bash -c g++  -static -lSegFault -x c++ - 'int main() {}' -o 
/home/zs/mosesdecoder/dummy /dev/null 2/dev/null  rm 
/home/zs/mosesdecoder/dummy 2/dev/null
 1
bash -c g++  -lboost_program_options-1_46_1 -x c++ - 'int main() {}' -o 
/home/zs/mosesdecoder/dummy /dev/null 2/dev/null  rm 
/home/zs/mosesdecoder/dummy 2/dev/null
 1
bash -c g++  -static -lboost_program_options -x c++ - 'int main() {}' -o 
/home/zs/mosesdecoder/dummy /dev/null 2/dev/null  rm 
/home/zs/mosesdecoder/dummy 2/dev/null
 0
bash -c g++  -lboost_system-mt  -static -x c++ - 'int main() {}' -o 
/home/zs/mosesdecoder/dummy /dev/null 2/dev/null  rm 
/home/zs/mosesdecoder/dummy 2/dev/null
 0
bash -c g++  -lboost_thread-mt  -static -x c++ - 'int main() {}' -o 
/home/zs/mosesdecoder/dummy /dev/null 2/dev/null  rm 
/home/zs/mosesdecoder/dummy 2/dev/null
 0
bash -c g++  -lboost_program_options-mt  -static -x c++ - 'int main() {}' 
-o /home/zs/mosesdecoder/dummy /dev/null 2/dev/null  rm 
/home/zs/mosesdecoder/dummy 2/dev/null
 0
bash -c g++  -lboost_unit_test_framework-mt  -DBOOST_TEST_MODULE=CompileTest  
-include boost/test/unit_test.hpp  -static -x c++ - 
'BOOST_AUTO_TEST_CASE(foo) {}' -o /home/zs/mosesdecoder/dummy /dev/null 
2/dev/null  rm /home/zs/mosesdecoder/dummy 2/dev/null
 0
bash -c g++  -lboost_iostreams-mt  -static -x c++ - 'int main() {}' -o 
/home/zs/mosesdecoder/dummy /dev/null 2/dev/null  rm 
/home/zs/mosesdecoder/dummy 2/dev/null
 0
bash -c g++  -lboost_filesystem-mt  -static -x c++ - 'int main() {}' -o 
/home/zs/mosesdecoder/dummy /dev/null 2/dev/null  rm 
/home/zs/mosesdecoder/dummy 2/dev/null
 0
bash -c g++  -static -lz -x c++ - 'int main() {}' -o 
/home/zs/mosesdecoder/dummy /dev/null 2/dev/null  rm 
/home/zs/mosesdecoder/dummy 2/dev/null
 0
bash -c g++ -ltcmalloc_minimal -x c++ - 'int main() {}' -o 
/home/zs/mosesdecoder/dummy /dev/null 2/dev/null  rm 
/home/zs/mosesdecoder/dummy 2/dev/null
 1
Tip: install tcmalloc for faster threading.  See BUILD-INSTRUCTIONS.txt for 
more information.
warning: No toolsets are configured.
warning: Configuring default toolset gcc.
warning: If the default is wrong, your build may not work correctly.
warning: Use the toolset=x option to override our guess.
warning: For more configuration options, please consult
warning: http://boost.org/boost-build2/doc/html/bbv2/advanced/configuration.html
notice: will use 'g++' for gcc, condition toolsetgcc-4.6
notice: using gcc libraries :: toolsetgcc-4.6 :: /usr/bin /usr/lib /usr/lib32 
/usr/lib64
notice: using gcc archiver :: toolsetgcc-4.6 :: /usr/bin/ar
notice: using gcc ranlib :: toolsetgcc-4.6 :: /usr/bin/ranlib
warning: toolset gcc initialization: can not find tool windres
warning: initialized from
notice: using rc compiler :: toolsetgcc-4.6 :: /usr/bin/as
bash -c g++ -lbz2 -x c++ - 'int main() {}' -o /home/zs/mosesdecoder/dummy 
/dev/null 2/dev/null  rm /home/zs/mosesdecoder/dummy 2/dev/null
 0
bash -c g++ -include bzlib.h -x c++ - 'int main() {}' -o 
/home/zs/mosesdecoder/dummy /dev/null 2/dev/null  rm 
/home/zs/mosesdecoder/dummy 2/dev/null
 0
bash -c g++  -static -lbz2 -x c++ - 'int main() {}' -o 
/home/zs/mosesdecoder/dummy /dev/null 2/dev/null  rm 
/home/zs/mosesdecoder/dummy 2/dev/null
 0
bash -c g++ -llzma -x c++ - 'int main() {}' -o /home/zs/mosesdecoder/dummy 
/dev/null 2/dev/null  rm /home/zs/mosesdecoder/dummy

Re: [Moses-support] help with installing Moses for the first time

2013-07-08 Thread Hieu Hoang


I recommend you follow this guide to get started with Moses:
http://www.statmt.org/moses/?n=Moses.Baseline
Once you understand it, then you need to collect some arabic-english 
data to create a system


On 07/07/2013 22:16, Khetam sh wrote:


Dear,


I am so interested in using Moses tool on Arabic English Combination, 
but I am having trouble accessing it, but I would also appreciate some 
insights into how to use it once I have access. I am wondering if you 
will help me?


 Best Regards,


Khetam AS




___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help with Moses installation

2013-07-07 Thread Bill_Lang(Gmail)

Hi Khetam,
 Maybe you can refer this link:
http://www.statmt.org/moses/?n=Moses.Baseline
 It is very helpful.

Best Regards,
LANG Jun


On Sun, Jul 7, 2013 at 2:19 AM, Khetam sh khetam.alsha...@hotmail.comwrote:

 Dear,



 I am so interested in using Moses tool on Arabic English Combination,
 but I am having trouble accessing it, but I would also appreciate some
 insights into how to use it once I have access. I am wondering if you will
 help me? 



 Your help would be greatly appreciated.

 Best Regards,

 Khetam



 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] help with Moses installation

2013-07-06 Thread Khetam sh

Dear,

 

I am so
interested in using Moses tool on Arabic English Combination, but I am
having trouble accessing it, but I would also appreciate some insights into how
to use it once I have access. I am wondering if you will help me? 

 

Your
help would be greatly appreciated.

Best
Regards,

Khetam

  ___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help me!

2013-05-07 Thread Thanh-Le Ha

Hi,

The Vietnamese National Project on Vietnamese Language and Speech
Processing (VLSP) has been working to compile some resources and
fundamental tools, including 10-sentence bilingual English-Vietnamese
corpus. You might have to fill a simple form to get that corpus from them:
http://vlsp.vietlp.org:8080/demo/?page=resources

Best regards,
Le.


On Mon, May 6, 2013 at 11:30 PM, Hieu Hoang hieuho...@gmail.com wrote:

  hi ha

 if you want to translate english to vietnamese, you need an
 english-vietnamese corpus. You can find here:
http://opus.lingfil.uu.se/
 other people on the moses mailing list might have other data they can give
 you.

 You can also use pivoting methods and triangulation to translate
 english-to-french-to-vietnamese. Eg.
en.wikipedia.org/wiki/*Pivot*_language

 On 06/05/2013 17:14, Ha Nguyen Tien wrote:

  Hi!
 My name is Ha
 I live in Viet Nam
 I'm researching the statistical machine translation models for Vietnamese.
 I have setup moses in my computer and i have used English-French corpus to
 translate French into English, now i want to use English-Vietnamese corpus
 to translate Vietnamese into English but i don't know that I have to build
 a English-Vietnamese corpus (its construct look like?) to use released
 toolkits to train my English-Vietnamese corpus.
 Please to help me.
 Thanks.

 -
  Nguyen Tien Ha
  Phone: 0985900997



 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help me!

2013-05-06 Thread Hieu Hoang


hi ha

if you want to translate english to vietnamese, you need an 
english-vietnamese corpus. You can find here:

http://opus.lingfil.uu.se/
other people on the moses mailing list might have other data they can 
give you.


You can also use pivoting methods and triangulation to translate 
english-to-french-to-vietnamese. Eg.

en.wikipedia.org/wiki/*Pivot*_language

On 06/05/2013 17:14, Ha Nguyen Tien wrote:

Hi!
My name is Ha
I live in Viet Nam
I'm researching the statistical machine translation models for 
Vietnamese. I have setup moses in my computer and i have used 
English-French corpus to translate French into English, now i want to 
use English-Vietnamese corpus to translate Vietnamese into English but 
i don't know that I have to build a English-Vietnamese corpus (its 
construct look like?) to use released toolkits to train my 
English-Vietnamese corpus.

Please to help me.
Thanks.

-
Nguyen Tien Ha
Phone: 0985900997


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Help on Moses Installation

2013-04-17 Thread Nadhii Dagaga

I have been trying to install moses on ubuntu 12.10 by following the
instructions on the manual. first i have installed both boost 1.52 and
gcc-4.7 and i have downloaded and extracted moses tool. The  i have been
trying to install it by using the following command:

./bjam --with-irstlm= /home/nadhii/moses/irstlm/

and it is displaying a warning and it is not creating any executable files
in bin directory:

warning: No toolsets are configured.
warning: Configuring default toolset gcc.
warning: If the default is wrong, your build may not work correctly.
warning: Use the toolset=x option to override our guess.
warning: For more configuration options, please consult
warning:
http://boost.org/boost-build2/doc/html/bbv2/advanced/configuration.html
notice: could not find main target /home/nadhii/moses/irstlm/
notice: assuming it is a name of file to create.
...found 1 target...

i have also tried to run bjam toolset=gcc-4.7  and it creates some
executable files in /moses/bin directory, but there is no executable file
with a name moses which i guess is the main decoder.

Please will u tell me the solution for this problem?
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help on Moses Installation

2013-04-17 Thread Kenneth Heafield

Please follow the directions on

http://www.statmt.org/moses/?n=Moses.MailingLists

There should be no space after the equals sign.

./bjam --with-irstlm=/home/nadhii/moses/irstlm/

Kenneth

On 04/17/13 10:02, Nadhii Dagaga wrote:
 I have been trying to install moses on ubuntu 12.10 by following the
 instructions on the manual. first i have installed both boost 1.52 and
 gcc-4.7 and i have downloaded and extracted moses tool. The  i have been
 trying to install it by using the following command:

 ./bjam --with-irstlm= /home/nadhii/moses/irstlm/

 and it is displaying a warning and it is not creating any executable
 files in bin directory:

 warning: No toolsets are configured.
 warning: Configuring default toolset gcc.
 warning: If the default is wrong, your build may not work correctly.
 warning: Use the toolset=x option to override our guess.
 warning: For more configuration options, please consult
 warning:
 http://boost.org/boost-build2/doc/html/bbv2/advanced/configuration.html
 notice: could not find main target /home/nadhii/moses/irstlm/
 notice: assuming it is a name of file to create.
 ...found 1 target...

 i have also tried to run bjam toolset=gcc-4.7  and it creates some
 executable files in /moses/bin directory, but there is no executable
 file with a name moses which i guess is the main decoder.

 Please will u tell me the solution for this problem?


 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help me to find and download IWSLT08 and NIST08 datasets

2013-04-17 Thread Philipp Koehn

Hi,

the NIST evaluation sets are distributed through the LDC.

The IWSLT evaluation sets are available if you register to participate
in their evaluation campaign - I am not sure if they are available
otherwise, but you should get in touch with the organizers.

-phi

On Mon, Apr 15, 2013 at 7:39 AM, Saeed Farzi saeedfa...@gmail.com wrote:
 Dear members,

 I gonna run and test our machine translation system on the well-known
 datasets (IWSLT08, NIST08) . Unfortunately, I could not find them on the
 web. I wonder if anybody could help me to find and download.

 Tnx in advance

 --
S.Farzi, Ph.D. Student
 Natural Language Processing Lab,
   School of Electrical and Computer Eng.,
Tehran University
  Tel: +9821-6111-9719


 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Help on Language Model Training

2013-04-13 Thread Jabesa Daba

when i have modified the path and run the following command:

~/irstlm/bin/add-start-end.sh  
~/corpus/news-commentary-v7.fr-en.true.en 
news-commentary-v7.fr-en.sb.en
 export IRSTLM=$HOME/irstlm; ~/irstlm/bin/build-lm.sh -i
news-commentary-v7.fr-en.sb.en  -t ./tmp  -p  \
-s improved-kneser-ney -o news-commentary-v7.fr-en.lm.en
 ~/irstlm/bin/compile-lm --text yes news-commentary-v7.fr-en.lm.en.gz
news-commentary-v7.fr-en.arpa.en


it displays

inpfile: news-commentary-v7.fr-en.lm.en.gz
loading up to the LM level 1000 (if any)
dub: 1000
Failed to open news-commentary-v7.fr-en.lm.en.gz!

what do you think is the problem?

regards,
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help on Language Model Training

2013-04-13 Thread Philipp Koehn

Hi,

please use full paths for your corpus files.

-phi

On Sat, Apr 13, 2013 at 1:44 PM, Jabesa Daba jappy...@gmail.com wrote:
 when i have modified the path and run the following command:

 ~/irstlm/bin/add-start-end.sh   ~/corpus/news-commentary-v7.fr-en.true.en 
 news-commentary-v7.fr-en.sb.en
  export IRSTLM=$HOME/irstlm; ~/irstlm/bin/build-lm.sh -i
 news-commentary-v7.fr-en.sb.en  -t ./tmp  -p  \
 -s improved-kneser-ney -o news-commentary-v7.fr-en.lm.en
  ~/irstlm/bin/compile-lm --text yes news-commentary-v7.fr-en.lm.en.gz
 news-commentary-v7.fr-en.arpa.en


 it displays

 inpfile: news-commentary-v7.fr-en.lm.en.gz
 loading up to the LM level 1000 (if any)
 dub: 1000
 Failed to open news-commentary-v7.fr-en.lm.en.gz!

 what do you think is the problem?

 regards,
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help on Language Model Training

2013-04-12 Thread Kenneth Heafield

Hi,

It looks like you're writing to news-commentary-v7.fr-en.lm.en without 
a .gz then trying to read news-commentary-v7.fr-en.lm.en.gz .  If that 
doesn't fix it, subscribe to the IRSTLM mailing list at 
https://list.fbk.eu/sympa/subscribe/user-irstlm and direct your message 
there.  Also, you probably want to specify an n-gram order.

Shameless self-promotion:

http://kheafield.com/code/kenlm/estimation/

Kenneth

On 04/12/13 09:10, Jabesa Daba wrote:
 when i run the following command:

 ~/irstlm/bin/add-start-end.sh   ~/corpus/news-commentary-v7.fr-en.true.en  
 news-commentary-v7.fr-en.sb.en
   export IRSTLM=$HOME/irstlm; ~/irstlm/bin/build-lm.sh -i  
 news-commentary-v7.fr-en.sb.en  -t ./tmp  -p  \
  -s improved-kneser-ney -o news-commentary-v7.fr-en.lm.en
   ~/irstlm/bin/compile-lm --text yes news-commentary-v7.fr-en.lm.en.gz 
 news-commentary-v7.fr-en.arpa.en


 it displays

 inpfile: news-commentary-v7.fr-en.lm.en.gz
 loading up to the LM level 1000 (if any)
 dub: 1000
 Failed to open news-commentary-v7.fr-en.lm.en.gz!

 what do you think is the problem?

 regards,


 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help with the concept

2013-03-24 Thread Philipp Koehn

Hi,

to use Moses, you will need a very large amount of translated text for training,
in the order of millions of words. If you have that, please consult the Moses
web site on details on how to train a system.

-phi

On Sat, Mar 23, 2013 at 5:13 AM, Eleni Teshome tesh.el...@gmail.com wrote:
 Dear all,
 I wanted to calculate the real concept on Moses...I'm translating from
 English to Amharic (my native language)...Example..given the following
 sentences
 Abebe ate beso = አበበ በሶ በላ in which Abebe maps to አበበ, ate maps to በላ, beso
 maps to በሶ
 Kebede drunk  beso=  ከበደ በሶ ጠጣ in which Kebede maps to ከበደ, drunk maps to
 ጠጣ, beso maps to በሶ

 and i wanted to translate:
 Abebe drunk beso which is አበበ በሶ ጠጣ

 how can i calculate the language model and translation model???

 Regards,
 Eleni










 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Help with the concept

2013-03-22 Thread Eleni Teshome

Dear all,
I wanted to calculate the real concept on Moses...I'm translating from
English to Amharic (my native language)...Example..given the following
sentences
Abebe ate beso = አበበ በሶ በላ in which Abebe maps to አበበ, ate maps to በላ,
beso maps to በሶ
Kebede drunk  beso=  ከበደ በሶ ጠጣ in which Kebede maps to ከበደ, drunk maps to ጠጣ,
beso maps to በሶ

and i wanted to translate:
Abebe drunk beso which is አበበ በሶ ጠጣ

how can i calculate the language model and translation model???

Regards,
Eleni
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help for reusing moses-chart in a different (dependency tree - to - string) MT system

2012-11-09 Thread Fabien Cromières

Hi, Hieu

Thank you for your quick reply :-).

But then, if the decoder can handle any number of non-terminal, it means I
still don't understand why the rules I am generating do not seem to be used
properly by the decoder. I think I will try to make a minimal example of
the problem I have and come back for more wisdom...

Thanks again for the answers,
Fabien


On Fri, Nov 9, 2012 at 12:15 AM, Hieu Hoang hieuho...@gmail.com wrote:

  hi fabien


 On 08/11/2012 01:25, Fabien Cromières wrote:

 Dear Moses community,

  First, thanks for providing the Research community with such a nice
 open-source tool.

  Second, I have some rather involved questions about the moses-chart
 decoder for syntax-based MT. Hopefully, someone having a good familiarity
 with it can answer me. Thanks in advance!

   Short version of the questions:  

  * Is there any support for Synchronized Tree Substitution Grammar in
 moses-chart? (or any trick to use such a grammar with the decoder).

 no, moses-chart only does scfg.
 The decoder accepts tree input for tree-to-string decoding. This can
 approximate stsg decoding.

  * Can moses-chart handle rules with more than 2 Non Terminals?

 yes, the decoding algorithm is CKY+. It can handle arbitrary number of
 non-terms.

  * Can moses-chart be directly given (in any way) a compact
 representation of a set of parse of an input sentence?  (and then just do
 the remaining work of selecting the best parse)

 i believe so, but i've never tried it. However, you can't assign
 probabilities of the input parse though. It wasn't really designed for
 forest decoding, so i'm not sure if it will be any good.


   Longer, more detailed, version: 

  I am trying to improve a tree-to-string MT system (it is actually
 tree-to-tree, but it will be easier for me to describe it as a
 tree-to-string system). And I was hoping I could somehow re-use part of the
 Moses toolchain. Basically, my system uses a dependency tree representation
 of the input sentence.

  For example: he -is - (a-boy)  (not easy to represent trees with
 strings in a readable way;hopefully, this notation is intuitive enough).

  I then have some Synchronized Tree Substitution rules, eg. for
 English-French (although unlike normal TSG, here the target side will be  a
 flat string).

  R1: Y- he | il
 R2: X- Y-is-Z| Y est Z
 R3: Z-a - boy| un garcon
 R4: Z-a -boy| un enfant
  R5: Z- V-boy| V garcon
  R6: V-a |un

  The rules are already extracted, selected and mapped to the input by
 my system (in other words, the source side parsing is already done). This
 mean I have a somehow compact representation of every possible alternatives
 derivations like this:
 R1-R2-(R3|R4|(R5-R6))

  Each derivation gives a different target sentence. From there, the
 problem is extracting the best derivation/translation according to language
 model and other features. It is essentially possible to do that with cube
 pruning or other beam-search approaches. However it would be interesting
 for me to re-use the work done for the moses-chart implementation.

  My first idea was to try and convert my TSG rules to the rule format
 expected by moses-chart. Unfortunately, this way, I cannot tell moses-chart
 in any way that the source side has already been parsed. However, I was
 hoping that I could make it easy for moses-chart to do the source-side
 parsing.

  Here is how I did it. For each input sentence, I use  the position of
 the words instead of the words (to reduce parsing ambiguity).
 For example, the input sentence:
 he is a boy
 becomes:
 1 2 3 4
 when I feed it to moses-chart.

  Then I generate a rule file by adapting my rules accordingly (note that
 each Non Terminal encode the source position it should match in its name,
 ensuring the possible derivations that can be found by moses-chart is the
 same as those given by the parsing done by my system):
 1 [X][X1] ||| il [X][X1] ||| ||| ...
 [X][X1] 2 [X][X4] [X][TOP] ||| [X][X1] est [X][X4] [X][TOP] ||| 0-0 2-2 |||
 3 4 [X][X4] ||| un garcon [X][X4] ||| ||| ...
  3 4 [X][X4] ||| un enfant [X][X4] ||| ||| ...
  [X][X3] 4 [X][X4] ||| [X][X3] garcon [X][X4] ||| 0-0 ||| ...
  3 [X][X3] ||| un [X][X1] ||| ||| ...

  However, I quickly found this was not giving the expected result
 (actually, no translation is found most of the time). After analyzing a bit
 more, I think I found out the problem is that I am generating some rules
 with more than 2 non-terminals. moses-chart do not complain about them, but
 it does not seem to be able to use them properly.

  I know hiero-style parsing will not work well with rules having more
 than 2 NT, so I guess this is to be expected. However, since it is not
 explicitly mentioned anywhere, I wanted to confirm that.

  Also, has anyone any suggestion for doing what I want to do with
 moses-chart? (since I have already the source-side parsing figured out, I
 don't think there is theoretical problems with having more than 2 NT per
 rule in my case; however,

[Moses-support] help for reusing moses-chart in a different (dependency tree - to - string) MT system

2012-11-08 Thread Fabien Cromières

Dear Moses community,

First, thanks for providing the Research community with such a nice
open-source tool.

Second, I have some rather involved questions about the moses-chart decoder
for syntax-based MT. Hopefully, someone having a good familiarity with it
can answer me. Thanks in advance!

 Short version of the questions:  

* Is there any support for Synchronized Tree Substitution Grammar in
moses-chart? (or any trick to use such a grammar with the decoder).
* Can moses-chart handle rules with more than 2 Non Terminals?
* Can moses-chart be directly given (in any way) a compact representation
of a set of parse of an input sentence?  (and then just do the remaining
work of selecting the best parse)

 Longer, more detailed, version: 

I am trying to improve a tree-to-string MT system (it is actually
tree-to-tree, but it will be easier for me to describe it as a
tree-to-string system). And I was hoping I could somehow re-use part of the
Moses toolchain. Basically, my system uses a dependency tree representation
of the input sentence.

For example: he -is - (a-boy)  (not easy to represent trees with strings
in a readable way;hopefully, this notation is intuitive enough).

I then have some Synchronized Tree Substitution rules, eg. for
English-French (although unlike normal TSG, here the target side will be  a
flat string).

R1: Y- he | il
R2: X- Y-is-Z| Y est Z
R3: Z-a - boy| un garcon
R4: Z-a -boy| un enfant
R5: Z- V-boy| V garcon
R6: V-a |un

The rules are already extracted, selected and mapped to the input by my
system (in other words, the source side parsing is already done). This mean
I have a somehow compact representation of every possible alternatives
derivations like this:
R1-R2-(R3|R4|(R5-R6))

Each derivation gives a different target sentence. From there, the problem
is extracting the best derivation/translation according to language model
and other features. It is essentially possible to do that with cube pruning
or other beam-search approaches. However it would be interesting for me to
re-use the work done for the moses-chart implementation.

My first idea was to try and convert my TSG rules to the rule format
expected by moses-chart. Unfortunately, this way, I cannot tell moses-chart
in any way that the source side has already been parsed. However, I was
hoping that I could make it easy for moses-chart to do the source-side
parsing.

Here is how I did it. For each input sentence, I use  the position of the
words instead of the words (to reduce parsing ambiguity).
For example, the input sentence:
he is a boy
becomes:
1 2 3 4
when I feed it to moses-chart.

Then I generate a rule file by adapting my rules accordingly (note that
each Non Terminal encode the source position it should match in its name,
ensuring the possible derivations that can be found by moses-chart is the
same as those given by the parsing done by my system):
1 [X][X1] ||| il [X][X1] ||| ||| ...
[X][X1] 2 [X][X4] [X][TOP] ||| [X][X1] est [X][X4] [X][TOP] ||| 0-0 2-2 |||
3 4 [X][X4] ||| un garcon [X][X4] ||| ||| ...
3 4 [X][X4] ||| un enfant [X][X4] ||| ||| ...
[X][X3] 4 [X][X4] ||| [X][X3] garcon [X][X4] ||| 0-0 ||| ...
3 [X][X3] ||| un [X][X1] ||| ||| ...

However, I quickly found this was not giving the expected result (actually,
no translation is found most of the time). After analyzing a bit more, I
think I found out the problem is that I am generating some rules with more
than 2 non-terminals. moses-chart do not complain about them, but it does
not seem to be able to use them properly.

I know hiero-style parsing will not work well with rules having more than 2
NT, so I guess this is to be expected. However, since it is not explicitly
mentioned anywhere, I wanted to confirm that.

Also, has anyone any suggestion for doing what I want to do with
moses-chart? (since I have already the source-side parsing figured out, I
don't think there is theoretical problems with having more than 2 NT per
rule in my case; however, I probably need to somehow provide the set of
possible parses to moses-chart)

I could also consider hacking into the moses-cmd code and try to replace
the source-parsing component by my own, but re-use some of the machinery.
Does that make sense or am I better off giving up on the idea of  re-using
moses code?). Or is anybody aware of another open source decoder that would
be more adapted for my case?

Thanks a lot to anyone that took the time to read my long explanations,
anyway :-)

Fabien
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help for reusing moses-chart in a different (dependency tree - to - string) MT system

2012-11-08 Thread Hieu Hoang


hi fabien

On 08/11/2012 01:25, Fabien Cromières wrote:

Dear Moses community,

First, thanks for providing the Research community with such a nice 
open-source tool.


Second, I have some rather involved questions about the moses-chart 
decoder for syntax-based MT. Hopefully, someone having a good 
familiarity with it can answer me. Thanks in advance!


 Short version of the questions:  

* Is there any support for Synchronized Tree Substitution Grammar in 
moses-chart? (or any trick to use such a grammar with the decoder).

no, moses-chart only does scfg.
The decoder accepts tree input for tree-to-string decoding. This can 
approximate stsg decoding.

* Can moses-chart handle rules with more than 2 Non Terminals?
yes, the decoding algorithm is CKY+. It can handle arbitrary number of 
non-terms.
* Can moses-chart be directly given (in any way) a compact 
representation of a set of parse of an input sentence?  (and then just 
do the remaining work of selecting the best parse)
i believe so, but i've never tried it. However, you can't assign 
probabilities of the input parse though. It wasn't really designed for 
forest decoding, so i'm not sure if it will be any good.


 Longer, more detailed, version: 

I am trying to improve a tree-to-string MT system (it is actually 
tree-to-tree, but it will be easier for me to describe it as a 
tree-to-string system). And I was hoping I could somehow re-use part 
of the Moses toolchain. Basically, my system uses a dependency tree 
representation of the input sentence.


For example: he -is - (a-boy)  (not easy to represent trees with 
strings in a readable way;hopefully, this notation is intuitive enough).


I then have some Synchronized Tree Substitution rules, eg. for 
English-French (although unlike normal TSG, here the target side will 
be  a flat string).


R1: Y- he | il
R2: X- Y-is-Z| Y est Z
R3: Z-a - boy| un garcon
R4: Z-a -boy| un enfant
R5: Z- V-boy| V garcon
R6: V-a |un

The rules are already extracted, selected and mapped to the input by 
my system (in other words, the source side parsing is already done). 
This mean I have a somehow compact representation of every possible 
alternatives derivations like this:

R1-R2-(R3|R4|(R5-R6))

Each derivation gives a different target sentence. From there, the 
problem is extracting the best derivation/translation according to 
language model and other features. It is essentially possible to do 
that with cube pruning or other beam-search approaches. However it 
would be interesting for me to re-use the work done for the 
moses-chart implementation.


My first idea was to try and convert my TSG rules to the rule format 
expected by moses-chart. Unfortunately, this way, I cannot tell 
moses-chart in any way that the source side has already been parsed. 
However, I was hoping that I could make it easy for moses-chart to do 
the source-side parsing.


Here is how I did it. For each input sentence, I use  the position of 
the words instead of the words (to reduce parsing ambiguity).

For example, the input sentence:
he is a boy
becomes:
1 2 3 4
when I feed it to moses-chart.

Then I generate a rule file by adapting my rules accordingly (note 
that each Non Terminal encode the source position it should match in 
its name, ensuring the possible derivations that can be found by 
moses-chart is the same as those given by the parsing done by my system):

1 [X][X1] ||| il [X][X1] ||| ||| ...
[X][X1] 2 [X][X4] [X][TOP] ||| [X][X1] est [X][X4] [X][TOP] ||| 0-0 
2-2 |||

3 4 [X][X4] ||| un garcon [X][X4] ||| ||| ...
3 4 [X][X4] ||| un enfant [X][X4] ||| ||| ...
[X][X3] 4 [X][X4] ||| [X][X3] garcon [X][X4] ||| 0-0 ||| ...
3 [X][X3] ||| un [X][X1] ||| ||| ...

However, I quickly found this was not giving the expected result 
(actually, no translation is found most of the time). After analyzing 
a bit more, I think I found out the problem is that I am generating 
some rules with more than 2 non-terminals. moses-chart do not complain 
about them, but it does not seem to be able to use them properly.


I know hiero-style parsing will not work well with rules having more 
than 2 NT, so I guess this is to be expected. However, since it is not 
explicitly mentioned anywhere, I wanted to confirm that.


Also, has anyone any suggestion for doing what I want to do with 
moses-chart? (since I have already the source-side parsing figured 
out, I don't think there is theoretical problems with having more than 
2 NT per rule in my case; however, I probably need to somehow provide 
the set of possible parses to moses-chart)


I could also consider hacking into the moses-cmd code and try to 
replace the source-parsing component by my own, but re-use some of the 
machinery. Does that make sense or am I better off giving up on the 
idea of  re-using moses code?). Or is anybody aware of another open 
source decoder that would be more adapted for my case?


Thanks a lot to anyone that took the time to read my long 
explanations,

[Moses-support] Help - Error while running Moses for the WTM 2010

2012-11-04 Thread Cuong Hoang

Hi all,
I was stuck with the error suddenly Killed when I run Moses on the WMT
2010 Task which contains 22520400 pairs of sentences.
The error comes when I run the sentences from line 21298000 to line
21299000:
*Exit code: 137*
*ERROR at /home/cuongh/STATMT.BIG/scripts/training/train-model.perl line
1130.*
I also attach the screenshot on the error I was stuck
Please help me overcome this problem.
Tks and best regards.
C. Hoang

-- 
Hoàng Cường
SMTNerd
attachment: Screenshot from 2012-11-04 14:38:02.png___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help - Error while running Moses for the WTM 2010

2012-11-04 Thread Cuong Hoang

Thanks,
I own you the second time, Hieu Hoang!
On Sun, Nov 4, 2012 at 3:20 PM, Hieu Hoang hieuho...@gmail.com wrote:

  hi cuong

 i think you're running out of memory when it's trying to run word
 alignment.

 You should try and verify this by running the word alignment step by
 itself and see if it gives the same error.

 You should run on another machine with more memory. Or follow this
 instruction:
http://www.statmt.org/moses/?n=Moses.Optimize#ntoc8

 Also, if you're on a 32-bit system, or on cygwin. You will not be able to
 use a binary language model bigger than about 2GB. Remember that when you
 get to the tuning stage


 On 03/11/2012 23:58, Cuong Hoang wrote:

 Hi all,
 I was stuck with the error suddenly Killed when I run Moses on the WMT
 2010 Task which contains 22520400 pairs of sentences.
 The error comes when I run the sentences from line 21298000 to line
 21299000:
  *Exit code: 137*
 *ERROR at /home/cuongh/STATMT.BIG/scripts/training/train-model.perl line
 1130.*
 I also attach the screenshot on the error I was stuck
 Please help me overcome this problem.
  Tks and best regards.
 C. Hoang

  --
 Hoàng Cường
 SMTNerd



 ___
 Moses-support mailing 
 listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support



 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support




-- 
Hoàng Cường
SMTNerd
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help - Decoding error for training Moses with a very large bilingual corpus

2012-10-29 Thread Cuong Hoang

Hi all,
I re-check the problem again. It just happens when I run Moses on a large
training corpus.
When I run on a smaller training corpus, no problem is happened.
I don't think there is any problem of server since I run on a 24-cores and
130G RAM server.
Everything is absolutely done including deploying two advantages of
Moses: Binary Phrase Tables with On-demand Loading and Binary Reordering
Tables with On-demand Loading.
Please help me,
Cheers,
C. Hoang

On Mon, Oct 29, 2012 at 12:52 PM, Cuong Hoang hoangcuong2...@gmail.comwrote:

 Hi all,
 When i run Moses which trains from a sub-part of WMT10. The bilingual
 corpus contains nearly 10 million pairs of sentences.
 Everything's okie. However, I was stuck with the error for the decoding
 process:

 *Check factors.size()==m_output.size() failed in moses/src/PDTAimp.h:273*
 and hence:
 *Aborted (core dumped)*
 *
 *
 I run Moses with the smaller sub-part of WMT10 (200K pairs of sentences),
 everything  is oke.

 Please let me know how to overcome with this bug.
 Thanks and best regards,
 C. Hoang




-- 
Hoàng Cường
SMTNerd
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help - Decoding error for training Moses with a very large bilingual corpus

2012-10-29 Thread Hieu Hoang

Hi Cuong

I think you have a | character in your data. You need to escape this before
training. You can use the script
  *escape-special-chars.perl*
*To do this*

Hieu
Sent from my flying horse

On 29 Oct 2012, at 03:21 AM, Cuong Hoang hoangcuong2...@gmail.com wrote:

Hi all,
I re-check the problem again. It just happens when I run Moses on a large
training corpus.
When I run on a smaller training corpus, no problem is happened.
I don't think there is any problem of server since I run on a 24-cores and
130G RAM server.
Everything is absolutely done including deploying two advantages of
Moses: Binary Phrase Tables with On-demand Loading and Binary Reordering
Tables with On-demand Loading.
Please help me,
Cheers,
C. Hoang

On Mon, Oct 29, 2012 at 12:52 PM, Cuong Hoang hoangcuong2...@gmail.comwrote:

 Hi all,
 When i run Moses which trains from a sub-part of WMT10. The bilingual
 corpus contains nearly 10 million pairs of sentences.
 Everything's okie. However, I was stuck with the error for the decoding
 process:

 *Check factors.size()==m_output.size() failed in moses/src/PDTAimp.h:273*
 and hence:
 *Aborted (core dumped)*
 *
 *
 I run Moses with the smaller sub-part of WMT10 (200K pairs of sentences),
 everything  is oke.

 Please let me know how to overcome with this bug.
 Thanks and best regards,
 C. Hoang




-- 
Hoàng Cường
SMTNerd

 ___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Help - Decoding error for training Moses with a very large bilingual corpus

2012-10-28 Thread Cuong Hoang

Hi all,
When i run Moses which trains from a sub-part of WMT10. The bilingual
corpus contains nearly 10 million pairs of sentences.
Everything's okie. However, I was stuck with the error for the decoding
process:

*Check factors.size()==m_output.size() failed in moses/src/PDTAimp.h:273*
and hence:
*Aborted (core dumped)*
*
*
I run Moses with the smaller sub-part of WMT10 (200K pairs of sentences),
everything  is oke.

Please let me know how to overcome with this bug.
Thanks and best regards,
C. Hoang
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Help with Moses compiling using bjam

2012-10-02 Thread Anis Zaman

Hi,
I am using moses for my senior year thesis. After I downloaded the moses
code from the web, when I try to compile by doing the following:

*./bjam --with-srilm=/Users/anis/demo/tools/srilm
--with-irstlm=/Users/anis/demo/tools/irstlm
--with-giza=/Users/anis/demo/tools/bin -j2*

I get bunch of these errors:

*...failed darwin.link
mert/bin/darwin-4.2.1/release/debug-symbols-on/link-static/threading-multi/singleton_test...
*
*...skipped
pmert/bin/darwin-4.2.1/release/debug-symbols-on/link-static/threading-multisingleton_test.passed
for lack of
pmert/bin/darwin-4.2.1/release/debug-symbols-on/link-static/threading-multisingleton_test...
*
*darwin.link
mert/bin/darwin-4.2.1/release/debug-symbols-on/link-static/threading-multi/timer_test
*
*ld: library not found for -lboost_unit_test_framework*
*collect2: ld returned 1 exit status*
...
...
...
...
*...failed updating 43 targets...*
*...skipped 68 targets...*
*...updated 13 targets...*
*
*
*
*
And there is not

   - dist/bin/processPhraseTable
   - dist/bin/processLexicalTable
   - dist/bin/moses

directories created. So at this point I can not even build a model or test
the success of my installation. I am using Mac OSX Mountain Lion. Any help
will be highly appreciated.

Thanks,

On Mon, Oct 1, 2012 at 9:55 PM, Ehsan alef.s...@yahoo.com wrote:

 Hi,

 Now that I have the result of Moses training, I am stuck in Tuning step.
 the training seemed to be done and I got the moses.ini file. But when I
 try to do the tuning I get
 this Error:
 ERROR:File
 .../working/mert-work/filtered/reordering-table.wbe-msd-bidirectional-fe
 does not exist

 does it mean that the training wasn't successful and I have to redo it?

 thanks,

   - Forwarded Message -
 *From:* Ehsan alef.s...@yahoo.com
 *To:* moses-support@mit.edu moses-support@mit.edu
 *Sent:* Monday, October 1, 2012 2:52:09 PM
 *Subject:* an inquiry regarding moses.ini

 Hi,

 I am following your instructions here :
 http://www.statmt.org/moses/?n=Moses.Baseline
 and I kind of stuck in the Training the Translation System step.

 I am working on my laptop and do not have enough processing resource to
 this step.
 Is there any chance that you have the moses.ini file of this step,
 uploaded somewhere?

 or if you don't, could you approximate how long does it take till it
 finishes this step on a
 corei7 laptop with 4GB of RAM?

 thanks,
 Alef



 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support




-- 
Anis Zaman
Bard College
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] help

2012-08-25 Thread Eleni Teshome

dear all,
i've been using the moses baseline site and the commands seem to do just
fine but i just couldn't pass the following step:
export IRSTLM=/home/eleni/irstlm; ~/irstlm/bin/build-lm.sh
-inews-commentary-v7.fr-en.sb.en -t ./tmp -p -s improved-kneser-ney -o
news-commentary-v7.fr-en.lm.en
when i try to run the above command, it displays the following:
Temporary directory ./tmp not found

creating ./tmp

Extracting dictionary from training corpus

Splitting dictionary into 3 lists

Extracting n-gram statistics for each word list

Important: dictionary must be ordered according to order of appearance of
words in data

used to generate n-gram blocks, so that sub language model blocks results
ordered too
dict.*

Estimating language models for each word list

ls: cannot access ./tmp/dict.*: No such file or directory (is there
supposed to be a dict.* in the tmp directory?)

Merging language models into news-commentary-v7.fr-en.lm.en

Cleaning temporary directory ./tmp

Removing temporary directory ./tmp

then i ignored the above and tried to run the next command:

~/irstlm/bin/compile-lm --text yes news-commentary-v7.fr-en.lm.en.gz
news-commentary-v7.fr-en.arpa.en
then it displayed the following:

inpfile: news-commentary-v7.fr-en.lm.en.gz
loading up to the LM level 1000 (if any)
dub: 1000
Failed to open news-commentary-v7.fr-en.lm.en.gz!

is there any solution as to what the problem could be? than you in advance
for all your help.

regards,

Eleni
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help

2012-08-13 Thread Barry Haddow

Hi Mohadeseh

You're using a very old version of Moses, so I'd suggest you update to 
the latest github version.

As to the Giza error, does this answer your question?
http://comments.gmane.org/gmane.comp.nlp.moses.user/7231

cheers - Barry

cheers - Barry

On 12/08/12 09:32, Mohadeseh Sheikhoveisi wrote:
 Hi,
 I want to get word-to-word alignment (IBM-Model-1) using giza++ while I run
 ./train-model.perl -scripts-root-dir /opt/tools/scripts-20120707-0326
 -root-dir /home/msho/Parallel-Corpus/Now/ -corpus
 /home/msho/Parallel-Corpus/clean/Clean-Corpus -f en -e fa -alignment
 grow-diag-final-and -reordering msd-bidirectional-fe -lm
 0:3:/home/msho/Parallel-Corpus/LM/lm/Data.fa.lm -parallel -giza-option
 m1=5,m2=0,m3=0,m4=0,mh=0,t1=5,nodumps=0  I get this error:

 ERROR: Giza did not produce the output file
 /home/msho/Parallel-Corpus/New/giza.fa-en/fa-en.A3.final. Is your corpus
 clean (reasonably-sized sentences)? at ./train-model.perl line 958.

 how can i get word-to-word alignment?
 please help me.

 Best Regards,



 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help

2012-08-13 Thread Barry Haddow

Hi Mohadeseh

If you use the giza option 'nodumps=0' then you should also get the 
model 1 alignments. Check the GIZA++ help message to be sure.

I meant that your Moses version was out-of-date, not your GIZA++ version,

cheers - Barry

On 13/08/12 11:19, Mohadeseh Sheikhoveisi wrote:
 Hi Barry

 thanks for your help. I use giza-pp-v1.0.7 (last version of giza++), and
 I apply your suggestion and get aligned.grow-diag-final-and but I need
 word-alignment (IBM-Model-1) while that file based on higher IBM-Models.

 Best Regards,

 On Mon, Aug 13, 2012 at 12:53 PM, Barry Haddow
 bhad...@staffmail.ed.ac.ukwrote:

 Hi Mohadeseh

 You're using a very old version of Moses, so I'd suggest you update to the
 latest github version.

 As to the Giza error, does this answer your question?
 http://comments.gmane.org/**gmane.comp.nlp.moses.user/7231http://comments.gmane.org/gmane.comp.nlp.moses.user/7231

 cheers - Barry

 cheers - Barry


 On 12/08/12 09:32, Mohadeseh Sheikhoveisi wrote:

 Hi,
 I want to get word-to-word alignment (IBM-Model-1) using giza++ while I
 run
 ./train-model.perl -scripts-root-dir /opt/tools/scripts-20120707-**0326
 -root-dir /home/msho/Parallel-Corpus/**Now/ -corpus
 /home/msho/Parallel-Corpus/**clean/Clean-Corpus -f en -e fa -alignment
 grow-diag-final-and -reordering msd-bidirectional-fe -lm
 0:3:/home/msho/Parallel-**Corpus/LM/lm/Data.fa.lm -parallel -giza-option
 m1=5,m2=0,m3=0,m4=0,mh=0,t1=5,**nodumps=0  I get this error:

 ERROR: Giza did not produce the output file
 /home/msho/Parallel-Corpus/**New/giza.fa-en/fa-en.A3.final. Is your
 corpus
 clean (reasonably-sized sentences)? at ./train-model.perl line 958.

 how can i get word-to-word alignment?
 please help me.

 Best Regards,



 __**_
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/**mailman/listinfo/moses-supporthttp://mailman.mit.edu/mailman/listinfo/moses-support


 --
 The University of Edinburgh is a charitable body, registered in
 Scotland, with registration number SC005336.




-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] help

2012-08-12 Thread Mohadeseh Sheikhoveisi

Hi,
I want to get word-to-word alignment (IBM-Model-1) using giza++ while I run
./train-model.perl -scripts-root-dir /opt/tools/scripts-20120707-0326
-root-dir /home/msho/Parallel-Corpus/Now/ -corpus
/home/msho/Parallel-Corpus/clean/Clean-Corpus -f en -e fa -alignment
grow-diag-final-and -reordering msd-bidirectional-fe -lm
0:3:/home/msho/Parallel-Corpus/LM/lm/Data.fa.lm -parallel -giza-option
m1=5,m2=0,m3=0,m4=0,mh=0,t1=5,nodumps=0  I get this error:

ERROR: Giza did not produce the output file
/home/msho/Parallel-Corpus/New/giza.fa-en/fa-en.A3.final. Is your corpus
clean (reasonably-sized sentences)? at ./train-model.perl line 958.

how can i get word-to-word alignment?
please help me.

Best Regards,
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help with filtering the noise enties in phrase-table.

2012-08-03 Thread Barry Haddow


Hi Jun

It's normal for the phrase table to contain a lot of noise, and 
generally the decoder can cope with this since it looks at various 
sources of information, ie different phrase scores, language model, 
reordering model. Moses also applies a cutoff when it loads translation 
options, so most of them won't be considered.


You can also try pruning the phrase table using the technique of Johnson 
et al

http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc17
This makes the phrase table much smaller, but doesn't usually change 
translation quality much,


cheers - Barry

On 03/08/12 02:57, jun@emc.com wrote:


Hi all,

I'm using Moses as the decoder. After the creation of translation 
model, i checked with the phrase-table. There are a lot of meaningless 
entries, looks l like below:


Can i remove these entries from the phrase-table? Will it have impact 
on the translation quality? If so, how about i delete all the 
punctuations before the creation of the translation model?


! kadov-  ?? ||| ! kadov- gt; Network ||| 1 0.341024 1 0.114004 
2.718 ||| ||| 4 4


! kadov-   ||| ! kadov- gt; Network parameters ||| 1 0.32747 1 
0.0420637 2.718 ||| ||| 2 2


! kadov-  ? ||| ! kadov- gt; Network parameters and ||| 1 
0.19872 1 0.0395799 2.718 ||| ||| 2 2


! kadov-  ??? ||| ! kadov- gt; Network parameters and security 
||| 1 0.129761 1 0.0179069 2.718 ||| ||| 2 2


! kadov-   ||| ! kadov- gt; Network Time ||| 1 0.225683 1 
0.00977698 2.718 ||| ||| 2 2


! kadov-  ?? ( ||| ! kadov- gt; Network Time Protocol ( ||| 1 
0.0968096 1 0.00137715 2.718 ||| ||| 2 2


! kadov-  ?? ||| ! kadov- gt; Network Time Protocol ||| 1 
0.208681 1 0.00154011 2.718 ||| ||| 2 2


! kadov-  ?? LUN ||| ! kadov- gt; Offline LUN ||| 1 0.307783 1 
0.03884 2.718 ||| ||| 1 1


! kadov-  ?? LUN ?? ||| ! kadov- gt; Offline LUN information ||| 1 
0.289121 1 0.0252108 2.718 ||| ||| 1 1


! kadov-  ?? LUN ??? ||| ! kadov- gt; Offline LUN information . ||| 
1 0.280973 1 0.0243627 2.718 ||| ||| 1 1


! kadov-  ?? ||| ! kadov- gt; Offline ||| 1 0.313173 1 0.0645311 
2.718 ||| ||| 2 2


! kadov-   ||| ! kadov- gt; Offline state ||| 1 0.266374 1 
0.0156563 2.718 ||| ||| 1 1


! kadov-  ?? ||| ! kadov- gt; Auto ||| 1 0.507327 0.2 0.0200052 
2.718 ||| ||| 1 5


!  ||| !  ||| 0.0185185 0.00821238 1 0.00102938 2.718 ||| ||| 54 1

! # % ' * + - ||| ! # % apos; * + - ||| 1 0.00122135 1 0.239671 2.718 
||| ||| 2 2


! # % ' * + ||| ! # % apos; * + ||| 1 0.00255049 1 0.750455 2.718 ||| 
||| 2 2


! # % ' * ||| ! # % apos; * ||| 1 0.00801901 1 0.773186 2.718 ||| ||| 2 2

! # % ' ||| ! # % apos; ||| 1 0.0121474 1 0.784364 2.718 ||| ||| 2 2

! # % ||| ! # % ||| 1 0.0426557 1 0.897899 2.718 ||| ||| 2 2

! # ||| ! # ||| 1 0.0550974 1 0.908339 2.718 ||| ||| 2 2

! #$ % ^* ||| ! # $ % ^ amp; * ||| 1 5.14084e-07 1 0.00855668 2.718 
||| ||| 2 2


! #$ % ||| ! # $ % ||| 1 6.54489e-05 1 0.23103 2.718 ||| ||| 2 2

! #$ ||| ! # $ ||| 1 8.45389e-05 1 0.233717 2.718 ||| |



___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help with filtering the noise enties in phrase-table.

2012-08-03 Thread Jun.Tan

Hi Barry,

Thank you very much!
I’ll do a test with this tool.
Btw, the download URL link on Moses website is unreachable. The project site 
should be:
http://projectile.sv.cmu.edu/research/public/tools/salm/salm.htm#update

From: Barry Haddow [mailto:bhad...@staffmail.ed.ac.uk]
Sent: Friday, August 03, 2012 4:03 PM
To: Tan, Jun
Cc: moses-support@mit.edu
Subject: Re: [Moses-support] help with filtering the noise enties in 
phrase-table.

Hi Jun

It's normal for the phrase table to contain a lot of noise, and generally the 
decoder can cope with this since it looks at various sources of information, ie 
different phrase scores, language model, reordering model. Moses also applies a 
cutoff when it loads translation options, so most of them won't be considered.

You can also try pruning the phrase table using the technique of Johnson et al
http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc17
This makes the phrase table much smaller, but doesn't usually change 
translation quality much,

cheers - Barry

On 03/08/12 02:57, jun@emc.commailto:jun@emc.com wrote:

Hi all,

I'm using Moses as the decoder. After the creation of translation model, i 
checked with the phrase-table. There are a lot of meaningless entries, looks l 
like below:

Can i remove these entries from the phrase-table? Will it have impact on the 
translation quality? If so, how about i delete all the punctuations before the 
creation of the translation model?




! kadov-  网络 ||| ! kadov- gt; Network ||| 1 0.341024 1 0.114004 2.718 ||| ||| 
4 4
! kadov-  网络 参数 ||| ! kadov- gt; Network parameters ||| 1 0.32747 1 0.0420637 
2.718 ||| ||| 2 2
! kadov-  网络 参数 和 ||| ! kadov- gt; Network parameters and ||| 1 0.19872 1 
0.0395799 2.718 ||| ||| 2 2
! kadov-  网络 参数 和 安全 ||| ! kadov- gt; Network parameters and security ||| 1 
0.129761 1 0.0179069 2.718 ||| ||| 2 2
! kadov-  网络 时间 ||| ! kadov- gt; Network Time ||| 1 0.225683 1 0.00977698 
2.718 ||| ||| 2 2
! kadov-  网络 时间 协议 ( ||| ! kadov- gt; Network Time Protocol ( ||| 1 0.0968096 
1 0.00137715 2.718 ||| ||| 2 2
! kadov-  网络 时间 协议 ||| ! kadov- gt; Network Time Protocol ||| 1 0.208681 1 
0.00154011 2.718 ||| ||| 2 2
! kadov-  脱机 LUN ||| ! kadov- gt; Offline LUN ||| 1 0.307783 1 0.03884 2.718 
||| ||| 1 1
! kadov-  脱机 LUN 信息 ||| ! kadov- gt; Offline LUN information ||| 1 0.289121 1 
0.0252108 2.718 ||| ||| 1 1
! kadov-  脱机 LUN 信息 。 ||| ! kadov- gt; Offline LUN information . ||| 1 
0.280973 1 0.0243627 2.718 ||| ||| 1 1
! kadov-  脱机 ||| ! kadov- gt; Offline ||| 1 0.313173 1 0.0645311 2.718 ||| 
||| 2 2
! kadov-  脱机 状态 ||| ! kadov- gt; Offline state ||| 1 0.266374 1 0.0156563 
2.718 ||| ||| 1 1
! kadov-  自动 ||| ! kadov- gt; Auto ||| 1 0.507327 0.2 0.0200052 2.718 ||| ||| 
1 5
!  ||| ! ” ||| 0.0185185 0.00821238 1 0.00102938 2.718 ||| ||| 54 1
! # % ' * + - ||| ! # % apos; * + - ||| 1 0.00122135 1 0.239671 2.718 ||| ||| 
2 2
! # % ' * + ||| ! # % apos; * + ||| 1 0.00255049 1 0.750455 2.718 ||| ||| 2 2
! # % ' * ||| ! # % apos; * ||| 1 0.00801901 1 0.773186 2.718 ||| ||| 2 2
! # % ' ||| ! # % apos; ||| 1 0.0121474 1 0.784364 2.718 ||| ||| 2 2
! # % ||| ! # % ||| 1 0.0426557 1 0.897899 2.718 ||| ||| 2 2
! # ||| ! # ||| 1 0.0550974 1 0.908339 2.718 ||| ||| 2 2
! #$ % ^* ||| ! # $ % ^ amp; * ||| 1 5.14084e-07 1 0.00855668 2.718 ||| ||| 2 
2
! #$ % ||| ! # $ % ||| 1 6.54489e-05 1 0.23103 2.718 ||| ||| 2 2
! #$ ||| ! # $ ||| 1 8.45389e-05 1 0.233717 2.718 ||| |






___

Moses-support mailing list

Moses-support@mit.edumailto:Moses-support@mit.edu

http://mailman.mit.edu/mailman/listinfo/moses-support

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Help

2012-08-02 Thread Mohadeseh Sheikhoveisi

Hi,
I'm running following command:

./train-model.perl -scripts-root-dir /opt/tools/scripts-20120707-0326
-root-dir /home/msho/Parallel-Corpus/New -corpus
/home/msho/Parallel-Corpus/clean/Clean-Corpus -f en -e fa -alignment
grow-diag-final-and -reordering msd-bidirectional-fe -lm
0:3:/home/msho/Parallel-Corpus/LM/lm/Data.fa.lm -parallel -giza-option
m1=5,m2=0,m3=0,m4=0,mh=0,t1=5,nodumps=0

and I get this error:

Giza did not produce the output file
/home/msho/Parallel-Corpus/New/giza.fa-en/fa-en.A3.final. Is your corpus
clean (reasonably-sized sentences)? at ./train-model.perl line 958.

My corpus is cleaned using clean-corpus-n.perl and limited to 70 tokens.
giza produced some files (in ./giza.fa-en) such as:
 fa-en.A1.5,
fa-en.cooc
fa-en.Decoder.config
fa-en.gizacfg
fa-en.perp
fa-en.t1.5
fa-en.trn.src.vcb
fa-en.trn.trg.vcb
fa-en.tst.src.vcb
fa-en.tst.trg.vcb

Also I attached log file.


log
Description: Binary data
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help

2012-08-02 Thread Barry Haddow

Hi Mohadeseh

I think the problem is that you added this:

-giza-option
m1=5,m2=0,m3=0,m4=0,mh=0,t1=5,nodumps=0

You are telling GIZA not to run the higher IBM models, yet 
train-model.perl is looking for their output. Try removing the 
giza-option and it should work,

cheers - Barry

On 02/08/12 08:54, Mohadeseh Sheikhoveisi wrote:
 Hi,
 I'm running following command:

 ./train-model.perl -scripts-root-dir /opt/tools/scripts-20120707-0326
 -root-dir /home/msho/Parallel-Corpus/New -corpus
 /home/msho/Parallel-Corpus/clean/Clean-Corpus -f en -e fa -alignment
 grow-diag-final-and -reordering msd-bidirectional-fe -lm
 0:3:/home/msho/Parallel-Corpus/LM/lm/Data.fa.lm -parallel -giza-option
 m1=5,m2=0,m3=0,m4=0,mh=0,t1=5,nodumps=0

 and I get this error:

 Giza did not produce the output file
 /home/msho/Parallel-Corpus/New/giza.fa-en/fa-en.A3.final. Is your corpus
 clean (reasonably-sized sentences)? at ./train-model.perl line 958.

 My corpus is cleaned using clean-corpus-n.perl and limited to 70 tokens.
 giza produced some files (in ./giza.fa-en) such as:
   fa-en.A1.5,
 fa-en.cooc
 fa-en.Decoder.config
 fa-en.gizacfg
 fa-en.perp
 fa-en.t1.5
 fa-en.trn.src.vcb
 fa-en.trn.trg.vcb
 fa-en.tst.src.vcb
 fa-en.tst.trg.vcb

 Also I attached log file.



 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help with filtering the noise enties in phrase-table.

2012-08-02 Thread Tom Hoar



All entries in a phrase table are meaningless. There are merely a
mappings with probabilities (et al) between groupings of tokens in your
your source  target data. I suggest better corpus preparation and
management is a better way to manage the content in the tables rather
than directly editing them. 

Tom 

On Thu, 2 Aug 2012 21:57:50 -0400, 
wrote:  

Hi all, 

I'm using Moses as the decoder. After the creation
of translation model, i checked with the phrase-table. There are a lot
of meaningless entries, looks l like below: 

Can i remove these entries
from the phrase-table? Will it have impact on the translation quality?
If so, how about i delete all the punctuations before the creation of
the translation model?  

! kadov-  网络 ||| ! kadov-  Network ||| 1
0.341024 1 0.114004 2.718 ||| ||| 4 4 

! kadov-  网络 参数 ||| ! kadov- 
Network parameters ||| 1 0.32747 1 0.0420637 2.718 ||| ||| 2 2 

!
kadov-  网络 参数 和 ||| ! kadov-  Network parameters and ||| 1 0.19872 1
0.0395799 2.718 ||| ||| 2 2 

! kadov-  网络 参数 和 安全 ||| ! kadov- 
Network parameters and security ||| 1 0.129761 1 0.0179069 2.718 ||| |||
2 2 

! kadov-  网络 时间 ||| ! kadov-  Network Time ||| 1 0.225683 1
0.00977698 2.718 ||| ||| 2 2 

! kadov-  网络 时间 协议 ( ||| ! kadov- 
Network Time Protocol ( ||| 1 0.0968096 1 0.00137715 2.718 ||| ||| 2 2


! kadov-  网络 时间 协议 ||| ! kadov-  Network Time Protocol ||| 1
0.208681 1 0.00154011 2.718 ||| ||| 2 2 

! kadov-  脱机 LUN ||| ! kadov-
 Offline LUN ||| 1 0.307783 1 0.03884 2.718 ||| ||| 1 1 

! kadov-  脱机
LUN 信息 ||| ! kadov-  Offline LUN information ||| 1 0.289121 1 0.0252108
2.718 ||| ||| 1 1 

! kadov-  脱机 LUN 信息 。 ||| ! kadov-  Offline LUN
information . ||| 1 0.280973 1 0.0243627 2.718 ||| ||| 1 1 

! kadov- 
脱机 ||| ! kadov-  Offline ||| 1 0.313173 1 0.0645311 2.718 ||| ||| 2 2


! kadov-  脱机 状态 ||| ! kadov-  Offline state ||| 1 0.266374 1
0.0156563 2.718 ||| ||| 1 1 

! kadov-  自动 ||| ! kadov-  Auto ||| 1
0.507327 0.2 0.0200052 2.718 ||| ||| 1 5 

!  ||| !  ||| 0.0185185
0.00821238 1 0.00102938 2.718 ||| ||| 54 1 

! # % ' * + - ||| ! # % ' *
+ - ||| 1 0.00122135 1 0.239671 2.718 ||| ||| 2 2 

! # % ' * + ||| ! #
% ' * + ||| 1 0.00255049 1 0.750455 2.718 ||| ||| 2 2 

! # % ' * ||| !
# % ' * ||| 1 0.00801901 1 0.773186 2.718 ||| ||| 2 2 

! # % ' ||| ! #
% ' ||| 1 0.0121474 1 0.784364 2.718 ||| ||| 2 2 

! # % ||| ! # % ||| 1
0.0426557 1 0.897899 2.718 ||| ||| 2 2 

! # ||| ! # ||| 1 0.0550974 1
0.908339 2.718 ||| ||| 2 2 

! #$ % ^* ||| ! # $ % ^  * ||| 1
5.14084e-07 1 0.00855668 2.718 ||| ||| 2 2 

! #$ % ||| ! # $ % ||| 1
6.54489e-05 1 0.23103 2.718 ||| ||| 2 2 

! #$ ||| ! # $ ||| 1
8.45389e-05 1 0.233717 2.718 ||| | 

   ___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help with filtering the noise enties in phrase-table.

2012-08-02 Thread Jun.Tan

I think I got your meaning. The phrase table just creates a relationship 
between source and target. It doesn’t have an impact on the translation quality.

Actually, the corpus data determines the translation quality entirely. Due to 
all of corpus data may be from various channels, it’s critical to have a tool 
can control the quality of these data. I have checked the resource on Moses 
website, found nothing related with this topic.

How can I improve the quality of training data? Especially, when the data size 
is very big, and it’s impossible to be done manually.
Any suggestion is welcome.

Thanks,
Jun


From: Tom Hoar [mailto:tah...@precisiontranslationtools.com]
Sent: Friday, August 03, 2012 10:08 AM
To: Tan, Jun
Cc: moses-support@mit.edu
Subject: Re: [Moses-support] help with filtering the noise enties in 
phrase-table.


All entries in a phrase table are meaningless. There are merely a mappings 
with probabilities (et al) between groupings of tokens in your your source  
target data. I suggest better corpus preparation and management is a better way 
to manage the content in the tables rather than directly editing them.

Tom



On Thu, 2 Aug 2012 21:57:50 -0400, jun@emc.commailto:jun@emc.com 
wrote:

Hi all,

I'm using Moses as the decoder. After the creation of translation model, i 
checked with the phrase-table. There are a lot of meaningless entries, looks l 
like below:

Can i remove these entries from the phrase-table? Will it have impact on the 
translation quality? If so, how about i delete all the punctuations before the 
creation of the translation model?




! kadov-  网络 ||| ! kadov-  Network ||| 1 0.341024 1 0.114004 2.718 ||| ||| 4 4
! kadov-  网络 参数 ||| ! kadov-  Network parameters ||| 1 0.32747 1 0.0420637 
2.718 ||| ||| 2 2
! kadov-  网络 参数 和 ||| ! kadov-  Network parameters and ||| 1 0.19872 1 
0.0395799 2.718 ||| ||| 2 2
! kadov-  网络 参数 和 安全 ||| ! kadov-  Network parameters and security ||| 1 
0.129761 1 0.0179069 2.718 ||| ||| 2 2
! kadov-  网络 时间 ||| ! kadov-  Network Time ||| 1 0.225683 1 0.00977698 2.718 
||| ||| 2 2
! kadov-  网络 时间 协议 ( ||| ! kadov-  Network Time Protocol ( ||| 1 0.0968096 1 
0.00137715 2.718 ||| ||| 2 2
! kadov-  网络 时间 协议 ||| ! kadov-  Network Time Protocol ||| 1 0.208681 1 
0.00154011 2.718 ||| ||| 2 2
! kadov-  脱机 LUN ||| ! kadov-  Offline LUN ||| 1 0.307783 1 0.03884 2.718 ||| 
||| 1 1
! kadov-  脱机 LUN 信息 ||| ! kadov-  Offline LUN information ||| 1 0.289121 1 
0.0252108 2.718 ||| ||| 1 1
! kadov-  脱机 LUN 信息 。 ||| ! kadov-  Offline LUN information . ||| 1 0.280973 
1 0.0243627 2.718 ||| ||| 1 1
! kadov-  脱机 ||| ! kadov-  Offline ||| 1 0.313173 1 0.0645311 2.718 ||| ||| 2 
2
! kadov-  脱机 状态 ||| ! kadov-  Offline state ||| 1 0.266374 1 0.0156563 2.718 
||| ||| 1 1
! kadov-  自动 ||| ! kadov-  Auto ||| 1 0.507327 0.2 0.0200052 2.718 ||| ||| 1 5
!  ||| ! ” ||| 0.0185185 0.00821238 1 0.00102938 2.718 ||| ||| 54 1
! # % ' * + - ||| ! # % ' * + - ||| 1 0.00122135 1 0.239671 2.718 ||| ||| 2 2
! # % ' * + ||| ! # % ' * + ||| 1 0.00255049 1 0.750455 2.718 ||| ||| 2 2
! # % ' * ||| ! # % ' * ||| 1 0.00801901 1 0.773186 2.718 ||| ||| 2 2
! # % ' ||| ! # % ' ||| 1 0.0121474 1 0.784364 2.718 ||| ||| 2 2
! # % ||| ! # % ||| 1 0.0426557 1 0.897899 2.718 ||| ||| 2 2
! # ||| ! # ||| 1 0.0550974 1 0.908339 2.718 ||| ||| 2 2
! #$ % ^* ||| ! # $ % ^  * ||| 1 5.14084e-07 1 0.00855668 2.718 ||| ||| 2 2
! #$ % ||| ! # $ % ||| 1 6.54489e-05 1 0.23103 2.718 ||| ||| 2 2
! #$ ||| ! # $ ||| 1 8.45389e-05 1 0.233717 2.718 ||| |


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] help with TXT cleanup tool

2012-07-31 Thread Jun.Tan

Hi all,

As Moses can only support plain text to do training or tuning, does anybody 
have some good TXT cleanup tool?

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] help in multi-threaded moses

2012-02-20 Thread shweta porwal

Hi im building an smt web translation system using your given guidelines on
moses home.

It worked fine when I configured moses normally without multi-threading.

But my project requires more faster output, so I configured moses uing





./configure --enable-threads --with-srilm=/opt/tools/srilm
--with-boost=/usr/local/include --with-boost-thread=boost_thread



 It does not give any errors, I could even complete all the further steps
and training of the model.

But in the final echo command it shows segemtation fault as given below:





ssp@ubuntu:/opt/tools/moses-scripts/scripts-20120220-2011/training$ cp
/home/ssp/smt/etof/model/moses.ini /home/ssp/smt/etof/model/moses-bin.ini



ssp@ubuntu:/opt/tools/moses-scripts/scripts-20120220-2011/training$ echo
today is monday | TMP=/tmp /opt/tools/moses/moses-cmd/src/moses -f
/home/ssp/smt/etof/model/moses.ini

Defined parameters (per moses.ini or switch):

  config: /home/ssp/smt/etof/model/moses.ini

  distortion-file: 0-0 wbe-msd-bidirectional-fe-allff 6
/home/ssp/smt/etof/model/reordering-table.wbe-msd-bidirectional-fe.gz

  distortion-limit: 6

  input-factors: 0

  lmodel-file: 8 0 3 /home/ssp/smt/etof/lm/europarl-v6.lm

  mapping: 0 T 0

  ttable-file: 0 0 0 5 /home/ssp/smt/etof/model/phrase-table.gz

  ttable-limit: 20

  weight-d: 0.3 0.3 0.3 0.3 0.3 0.3 0.3

  weight-l: 0.5000

  weight-t: 0.2 0.2 0.2 0.2 0.2

  weight-w: -1

Segmentation fault





This is a copy of my moses.ini file:







#

### MOSES CONFIG FILE ###

#



# input factors

[input-factors]

0



# mapping steps

[mapping]

0 T 0



# translation tables: table type (hierarchical(0), textual (0), binary
(1)), source-factors, target-factors, number of scores, file

# OLD FORMAT is still handled for back-compatibility

# OLD FORMAT translation tables: source-factors, target-factors, number of
scores, file

# OLD FORMAT a binary table type (1) is assumed

[ttable-file]

1 0 0 5 /home/ssp/smt/etof/model/phrase-table.gz



# no generation models, no generation-file section



# language models: type(srilm/irstlm), factors, order, file

[lmodel-file]

0 0 3 /home/ssp/smt/etof/lm/europarl-v6.lm





# limit on how many phrase translations e for each phrase f are loaded

# 0 = all elements loaded

[ttable-limit]

20



# distortion (reordering) files

[distortion-file]

0-0 wbe-msd-bidirectional-fe-allff 6
/home/ssp/smt/etof/model/reordering-table.wbe-msd-bidirectional-fe.gz



# distortion (reordering) weight

[weight-d]

0.3

0.3

0.3

0.3

0.3

0.3

0.3



# language model weights

[weight-l]

0.5000





# translation model weights

[weight-t]

0.2

0.2

0.2

0.2

0.2



# no generation models, no weight-generation section



# word penalty

[weight-w]

-1



[distortion-limit]

6



 please i would appreciate your help!!!

As an alternative way to do web translation should i try creating a
moses-server.

I tried the instructions given in Advanced Features of moses, but could not
implement it.


Thank you in advance.
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] help in multi-threaded moses

2012-02-20 Thread Barry Haddow

Hi Shweta

It looks like you're using an out-of-date version of moses. Try downloading 
the latest from github

https://github.com/moses-smt/mosesdecoder

cheers  - Barry

On Monday 20 Feb 2012 17:39:39 shweta porwal wrote:
 Hi im building an smt web translation system using your given guidelines on
 moses home.
 
 It worked fine when I configured moses normally without multi-threading.
 
 But my project requires more faster output, so I configured moses uing
 
 
 
 
 
 ./configure --enable-threads --with-srilm=/opt/tools/srilm
 --with-boost=/usr/local/include --with-boost-thread=boost_thread
 
 
 
  It does not give any errors, I could even complete all the further steps
 and training of the model.
 
 But in the final echo command it shows segemtation fault as given below:
 
 
 
 
 
 ssp@ubuntu:/opt/tools/moses-scripts/scripts-20120220-2011/training$ cp
 /home/ssp/smt/etof/model/moses.ini /home/ssp/smt/etof/model/moses-bin.ini
 
 
 
 ssp@ubuntu:/opt/tools/moses-scripts/scripts-20120220-2011/training$ echo
 today is monday | TMP=/tmp /opt/tools/moses/moses-cmd/src/moses -f
 /home/ssp/smt/etof/model/moses.ini
 
 Defined parameters (per moses.ini or switch):
 
   config: /home/ssp/smt/etof/model/moses.ini
 
   distortion-file: 0-0 wbe-msd-bidirectional-fe-allff 6
 /home/ssp/smt/etof/model/reordering-table.wbe-msd-bidirectional-fe.gz
 
   distortion-limit: 6
 
   input-factors: 0
 
   lmodel-file: 8 0 3 /home/ssp/smt/etof/lm/europarl-v6.lm
 
   mapping: 0 T 0
 
   ttable-file: 0 0 0 5 /home/ssp/smt/etof/model/phrase-table.gz
 
   ttable-limit: 20
 
   weight-d: 0.3 0.3 0.3 0.3 0.3 0.3 0.3
 
   weight-l: 0.5000
 
   weight-t: 0.2 0.2 0.2 0.2 0.2
 
   weight-w: -1
 
 Segmentation fault
 
 
 
 
 
 This is a copy of my moses.ini file:
 
 
 
 
 
 
 
 #
 
 ### MOSES CONFIG FILE ###
 
 #
 
 
 
 # input factors
 
 [input-factors]
 
 0
 
 
 
 # mapping steps
 
 [mapping]
 
 0 T 0
 
 
 
 # translation tables: table type (hierarchical(0), textual (0), binary
 (1)), source-factors, target-factors, number of scores, file
 
 # OLD FORMAT is still handled for back-compatibility
 
 # OLD FORMAT translation tables: source-factors, target-factors, number of
 scores, file
 
 # OLD FORMAT a binary table type (1) is assumed
 
 [ttable-file]
 
 1 0 0 5 /home/ssp/smt/etof/model/phrase-table.gz
 
 
 
 # no generation models, no generation-file section
 
 
 
 # language models: type(srilm/irstlm), factors, order, file
 
 [lmodel-file]
 
 0 0 3 /home/ssp/smt/etof/lm/europarl-v6.lm
 
 
 
 
 
 # limit on how many phrase translations e for each phrase f are loaded
 
 # 0 = all elements loaded
 
 [ttable-limit]
 
 20
 
 
 
 # distortion (reordering) files
 
 [distortion-file]
 
 0-0 wbe-msd-bidirectional-fe-allff 6
 /home/ssp/smt/etof/model/reordering-table.wbe-msd-bidirectional-fe.gz
 
 
 
 # distortion (reordering) weight
 
 [weight-d]
 
 0.3
 
 0.3
 
 0.3
 
 0.3
 
 0.3
 
 0.3
 
 0.3
 
 
 
 # language model weights
 
 [weight-l]
 
 0.5000
 
 
 
 
 
 # translation model weights
 
 [weight-t]
 
 0.2
 
 0.2
 
 0.2
 
 0.2
 
 0.2
 
 
 
 # no generation models, no weight-generation section
 
 
 
 # word penalty
 
 [weight-w]
 
 -1
 
 
 
 [distortion-limit]
 
 6
 
 
 
  please i would appreciate your help!!!
 
 As an alternative way to do web translation should i try creating a
 moses-server.
 
 I tried the instructions given in Advanced Features of moses, but could not
 implement it.
 
 
 Thank you in advance.
 
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Help!!!Unkown parameter DPR-file in moses!!!

2011-12-09 Thread Philipp Koehn

Hi,

how did you create the configuration file moses.ini in the
first place? DPR-file is not a standard parameter.

-phi

2011/12/6 李惠 lh9...@126.com:
 Dear Mr./Ms.

 When I was compling the Moses and tested the phrase-model, some problems
 arose. I quoted it as follows.
 lilian@munich:~/bin/sample-models/phrase-model$
 /home/lilian/bin/moses/moses-cmd/src/moses -f moses.iniinout
  Defined parameters (per moses.ini or switch):
     DPR-file: DPR-file.txt
     config: moses.ini
     input-factors: 0
     lmodel-file: 0 0 3 ../lm/europarl.srilm.gz
     mapping: T 0
     n-best-list: nbest.txt 100
     ttable-file: 0 0 0 1 phrase-table
     ttable-limit: 10
     weight-DPR: 3
     weight-d: 1
    ! weight-l: 1
     weight-t: 1
     weight-w: 0
 ERROR:Unknown parameter DPR-file
 ERROR:Unknown parameter weight-DPR
 I don't know how it happens and how to solve it. Thank you for your
 attention, if you want any further information, please donot hesitate to
 contact me.

 Best Regards,
 Hui LI
 Tsinghua University, China.





 ___
 Moses-support mailing list
 Moses-support@mit.edu
 http://mailman.mit.edu/mailman/listinfo/moses-support


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Help me about including new library

2010-09-05 Thread hungnv54

Hi all,
I used new shared library libWSD.so. So, I want to include it in moses 
project. With 2 projects: moses and moses-cmd, I don't know how to add it in 
eclipse. How to build 2 projects: moses and moses-cmd in eclipse.
thanks a lot!
 
   

Thư được gửi từ http://mail.zing.vn 
 8643200x___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

1 2 >

1 - 100 of 115 matches

Mail list logo