Re: [Moses-support] Reading phrase table taking very long

[Hieu Hoang]

you're running out of memory. Binarize your phrase table

On 09/09/2013 22:34, rohit dholakia wrote:

Hi,

  Alright, my phrase table is just 100MB compressed. And after taking 
quite long to read the phrase table, i got an exception bad_alloc. I 
have quite a bit of RAM, so, i am not sure what I am doing wrong .


Start loading text SCFG phrase table. Moses  format : [51.000] seconds
Reading /global/scratch/rd/Thesis/TestingMoses/train/model/phrase-table.gz
5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100

Exception: std::bad_alloc

Are the changes in the master branch mostly related to an updated 
moses file format ?  Are there any changes in the algorithms(mert/ibm 
models) that warrant a re-run of tests done using the previous version ?





On Mon, Sep 9, 2013 at 1:24 PM, rohit dholakia > wrote:


Hi

  I have recently managed to install moses from the master branch
on github.  The file format has changed. The training and mert has
gone without incident. But, i have observed that reading the
phrase table is taking very very long compared to the previous
moses. From mert.out and from translating test data log,

/Loading table into memory...done./
/Start loading text SCFG phrase table. Moses  format : [51.000]
seconds/
/Reading
/global/scratch/rd/Thesis/TestingMoses/train/model/phrase-table.gz/

/5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100/


Now, this step, this step is taking really long. Am I the only one
who is observing this ? My phrase table is trained on only 110K
sentences, quite noisy ones at that. Has only 4.8 million rules.

/[rd@seawolf1 TestingMoses]$ zcat train/model/phrase-table.gz | wc -l/
/4838810/

To explain the time taken, its been more than 5 minutes and not
even a single star has come up below that last line (  5 ---
10 ) .

Thanks.





___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] No moses.ini file is created !!!

[Hieu Hoang]

look in mert.out. Is there an error?


On 09/09/2013 13:40, Deeptiman Pattnaik wrote:


  Hello ,


 I am in an end of my building my sample language 
model of two parallel languages. But i don't know the problem , while 
trying to execute following commands, i can't able to create the 
moses.ini file.



*Command-:*

 nohup nice ~/mosesdecoder/scripts/training/mert-moses.pl 
 \


 ~/corpus/news-test2008.true.fr  
~/corpus/news-test2008.true.en \


~/mosesdecoder/bin/moses train/model/moses.ini --mertdir 
~/mosesdecoder/bin \ &> mert.out &



I have created the *train* folder and *model* folder also but *no 
moses.ini file* has been created.



Please help me out , then i will test my sample translation interface.


   Regards
Deeptiman Pattnaik





___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Reading phrase table taking very long

[rohit dholakia]

Hi

  I have recently managed to install moses from the master branch on
github.  The file format has changed. The training and mert has gone
without incident. But, i have observed that reading the phrase table is
taking very very long compared to the previous moses. From mert.out and
from translating test data log,

*Loading table into memory...done.*
*Start loading text SCFG phrase table. Moses  format : [51.000] seconds*
*Reading /global/scratch/rd/Thesis/TestingMoses/train/model/phrase-table.gz*
*
5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
*


Now, this step, this step is taking really long. Am I the only one who is
observing this ? My phrase table is trained on only 110K sentences, quite
noisy ones at that. Has only 4.8 million rules.

*[rd@seawolf1 TestingMoses]$ zcat train/model/phrase-table.gz | wc -l*
*4838810*

To explain the time taken, its been more than 5 minutes and not even a
single star has come up below that last line (  5 --- 10 ) .

Thanks.
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Reading phrase table taking very long

[rohit dholakia]

Hi,

  Alright, my phrase table is just 100MB compressed. And after taking quite
long to read the phrase table, i got an exception bad_alloc. I have quite a
bit of RAM, so, i am not sure what I am doing wrong .

Start loading text SCFG phrase table. Moses  format : [51.000] seconds
Reading /global/scratch/rd/Thesis/TestingMoses/train/model/phrase-table.gz
5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100

Exception: std::bad_alloc


Are the changes in the master branch mostly related to an updated moses
file format ?  Are there any changes in the algorithms(mert/ibm models)
that warrant a re-run of tests done using the previous version ?




On Mon, Sep 9, 2013 at 1:24 PM, rohit dholakia  wrote:

> Hi
>
>   I have recently managed to install moses from the master branch on
> github.  The file format has changed. The training and mert has gone
> without incident. But, i have observed that reading the phrase table is
> taking very very long compared to the previous moses. From mert.out and
> from translating test data log,
>
> *Loading table into memory...done.*
> *Start loading text SCFG phrase table. Moses  format : [51.000] seconds*
> *Reading
> /global/scratch/rd/Thesis/TestingMoses/train/model/phrase-table.gz*
> *
> 5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
> *
>
>
> Now, this step, this step is taking really long. Am I the only one who is
> observing this ? My phrase table is trained on only 110K sentences, quite
> noisy ones at that. Has only 4.8 million rules.
>
> *[rd@seawolf1 TestingMoses]$ zcat train/model/phrase-table.gz | wc -l*
> *4838810*
>
> To explain the time taken, its been more than 5 minutes and not even a
> single star has come up below that last line (  5 --- 10 ) .
>
> Thanks.
>
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] unformatted biconcor output

[Alex Martinez]

Hi,i have succeeded installing the Web interface to the EMS and I'm 
playing with the config.toy. I still have a small problem... when I 
click over a source word to get the concordance (biconcor) the output I 
get is unformatted likefinally(15)
 enfin , ceux qui s' opposent à une réglementation sur les rémunérations
 seront sûrement ceux qui nous mettront en garde contre les « 
conséquences fortuites » . ||| finally , those opposing pay regulation 
are certain to warn us about " unintended consequences . " ||| 0 0 ||| 0
 0 ||| 0-0 1-1 2-2 19-3 10-4 11-4 7-5 8-5 5-6 5-7 6-8 3-9 4-9 5-9 17-10 
9-11 22-12 24-13 23-14 26-15 enfin , d' autres politiques des pays 
riches pourraient être plus efficaces que l' aide internationale . ||| 
finally , other rich country policies may be much more beneficial than 
aid . ||| 0 0 ||| 0 0 ||| 0-0 1-1 2-2 3-2 5-3 7-3 6-4 4-5 8-6 9-7 11-8 
10-9 11-10 12-11 13-12 14-12 15-12 enfin , dans le cadre de l' 
appréciation de sociétés contrôlées , il importe de ne pas juger l' 
indépendance des administrateurs essentiellement sur leur indépendance 
vis @-@ à-vis du conseil de la société où ils siègent , mais plutôt vis 
@-@ à-vis de son actionnaire majoritaire . ||| finally , when assessing 
controlled companies , the independence of directors should not be 
judged largely by looking at the extent to which they are independent of
 the company on whose board they serve . rather , considerable attention
 should be given to their independence from the controlling shareholder .
 ||| 0 0 ||| 0 0 ||| 0-0 1-1 2-2 7-3 10-4 9-5 11-6 18-7 18-8 19-8 20-9 
21-10 13-11 15-12 16-12 17-13 17-14 17-15 17-16 17-17 30-18 29-19 28-20 
31-21 32-27 33-28 23-29 34-31 35-32 36-33 38-34 39-35 37-36 38-37 42-38 
43-42 24-43 25-44 26-45 27-46 27-47 44-47 45-48 46-48  I have moved biconcor and the web directory to the root of my apache web server, have I missed something?Thanks in advanceAlex.___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] No moses.ini file is created !!!

[Deeptiman Pattnaik]

  Hello ,


 I am in an end of my building my sample language model
of two parallel languages. But i don't know the problem , while trying to
execute following commands, i can't able to create the moses.ini file.


*Command-:*

 nohup nice ~/mosesdecoder/scripts/training/mert-moses.pl \

 ~/corpus/news-test2008.true.fr ~/corpus/news-test2008.true.en \

~/mosesdecoder/bin/moses train/model/moses.ini --mertdir ~/mosesdecoder/bin
\ &> mert.out &


I have created the *train* folder and *model* folder also but *no moses.ini
file* has been created.


Please help me out , then i will test my sample translation interface.


   Regards
Deeptiman Pattnaik
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Fwd:

[Hieu Hoang]

can you try something - don't use the default boost library.

Instead, download a newer (latest) boost from boost.org and compile it.
Here's how I build boost
   ./bootstrap.sh
   ./b2 --prefix=/home/s0565741/workspace/boost/thor/boost_1_54_0/
--libdir=/home/s0565741/workspace/boost/thor/boost_1_54_0/lib64
--layout=tagged link=static threading=multi,single install -j8

recompile moses with this library
  ./bjam -j2 -a
--with-boost=/home/s0565741/workspace/boost/thor/boost_1_54_0




On 7 September 2013 22:44, Shashi Narayan  wrote:

> Hi guys,
>
> Thanks for your quick reply. Unfortunately the problem remains with some
> other file now.
>
> Just to answer Hieu's questions. I am compiling on Ubuntu 10.04 (64 bits).
> I am using the default boost library libboost-all-dev (version 1.40). And
> for moses I have cloned the git repository as suggested on
> http://www.statmt.org/moses/?n=Development.GetStarted ( git clone git://
> github.com/moses-smt/mosesdecoder.git).
>
> I pulled new updates but unfortunately it still fails. Build.log does not
> change but the failure report is attached below.
>
> Thank you very much for your help.
> Shashi
>
> $ ./bjam -j2
> Tip: install tcmalloc for faster threading.  See BUILD-INSTRUCTIONS.txt
> for more information.
> ...patience...
> ...found 3443 targets...
> ...updating 3 targets...
> gcc.link
> moses/LM/bin/BackwardTest.test/gcc-4.1.3/release/debug-symbols-on/link-static/threading-multi/BackwardTest
> moses/LM/bin/BackwardTest.test/gcc-4.1.3/release/debug-symbols-on/link-static/threading-multi/Backward.o:
> In function `BackwardLanguageModel':
> ~/Softwares/moses/moses/LM/Backward.cpp:42: undefined reference to
> `Moses::LanguageModelKen::LanguageModelKen(std::basic_string std::char_traits, std::allocator > const&,
> std::basic_string, std::allocator >
> const&, unsigned long, bool)'
> ~/Softwares/moses/moses/LM/Backward.cpp:42: undefined reference to
> `Moses::LanguageModelKen::LanguageModelKen(std::basic_string std::char_traits, std::allocator > const&,
> std::basic_string, std::allocator >
> const&, unsigned long, bool)'
> ~/Softwares/moses/moses/LM/Backward.cpp:42: undefined reference to
> `Moses::LanguageModelKen::LanguageModelKen(std::basic_string std::char_traits, std::allocator > const&,
> std::basic_string, std::allocator >
> const&, unsigned long, bool)'
> ~/Softwares/moses/moses/LM/Backward.cpp:42: undefined reference to
> `Moses::LanguageModelKen::LanguageModelKen(std::basic_string std::char_traits, std::allocator > const&,
> std::basic_string, std::allocator >
> const&, unsigned long, bool)'
> ~/Softwares/moses/moses/LM/Backward.cpp:42: undefined reference to
> `Moses::LanguageModelKen::LanguageModelKen(std::basic_string std::char_traits, std::allocator > const&,
> std::basic_string, std::allocator >
> const&, unsigned long, bool)'
> ~/Softwares/moses/moses/LM/Backward.cpp:42: undefined reference to
> `Moses::LanguageModelKen::LanguageModelKen(std::basic_string std::char_traits, std::allocator > const&,
> std::basic_string, std::allocator >
> const&, unsigned long, bool)'
> ~/Softwares/moses/moses/LM/Backward.cpp:42: undefined reference to
> `Moses::LanguageModelKen::LanguageModelKen(std::basic_string std::char_traits, std::allocator > const&,
> std::basic_string, std::allocator >
> const&, unsigned long, bool)'
> collect2: ld returned 1 exit status
>
> "g++"-o
> "moses/LM/bin/BackwardTest.test/gcc-4.1.3/release/debug-symbols-on/link-static/threading-multi/BackwardTest"
> -Wl,--start-group
> "moses/LM/bin/BackwardTest.test/gcc-4.1.3/release/debug-symbols-on/link-static/threading-multi/BackwardTest.o"
> "moses/LM/bin/BackwardTest.test/gcc-4.1.3/release/debug-symbols-on/link-static/threading-multi/Backward.o"
> "moses/LM/bin/BackwardTest.test/gcc-4.1.3/release/debug-symbols-on/link-static/threading-multi/BackwardLMState.o"
> "moses/LM/bin/BackwardTest.test/gcc-4.1.3/release/debug-symbols-on/link-static/threading-multi/Base.o"
> "moses/LM/bin/BackwardTest.test/gcc-4.1.3/release/debug-symbols-on/link-static/threading-multi/Implementation.o"
> "moses/LM/bin/BackwardTest.test/gcc-4.1.3/release/debug-symbols-on/link-static/threading-multi/Joint.o"
> "moses/LM/bin/BackwardTest.test/gcc-4.1.3/release/debug-symbols-on/link-static/threading-multi/Ken.o"
> "moses/LM/bin/BackwardTest.test/gcc-4.1.3/release/debug-symbols-on/link-static/threading-multi/MultiFactor.o"
> "moses/LM/bin/BackwardTest.test/gcc-4.1.3/release/debug-symbols-on/link-static/threading-multi/Remote.o"
> "moses/LM/bin/BackwardTest.test/gcc-4.1.3/release/debug-symbols-on/link-static/threading-multi/SingleFactor.o"
> "moses/LM/bin/gcc-4.1.3/release/debug-symbols-on/link-static/threading-multi/ORLM.o"
> "util/bin/gcc-4.1.3/release/debug-symbols-on/link-static/threading-multi/read_compressed.o"
> "util/double-conversion/bin/gcc-4.1.3/release/debug-symbols-on/link-static/threading-multi/diy-fp.o"
> "util/double-conversion/bin/gcc-4.1.3/release/debug-symbols-on/link-static/threading-multi/strtod.o"

Re: [Moses-support] Getting Error in Command followed from Moses Manual !!!

[Hieu Hoang]

seems fine to me. Using your data:
   ./scripts/recaser/train-truecaser.perl --model model --corpus
~/Downloads/english.tok.en
i get:
# head model
's (1/1)
the (12/13) THE (1)
that (1/1)
, (4/4)
on (1/1)
earth (1/1)
Let (1/1)
day (1/1)
darkness (1/1)
earth. (2/2)


On 7 September 2013 19:04, Deeptiman Pattnaik
wrote:

> Hello,
>
> I am very new Moses, so i was applying the some of the commands from moses
> mannual but got following error.
>
> I have started doing the Corpus preparation in Moses. But getting some
> error.
>
>you can see in the Moses manual Page No-: 27 , Corpus Preparation
>
> Moses Manual -: http://www.statmt.org/moses/manual/manual.pdf
>
>
> *Tokenisation*
>
> i have successfully completed Tokenisation  of English Corpora, i got
> two output file
>
> english.tok.en
>
>
> *Truecaser
> *
>
> When i proceed for the truecase of the English corpora , we typed
> following commands.
>
>
> ~/mosesdecoder/scripts/recaser/train-truecaser.perl \
>
> --model ~/mosesdecoder/corpus/truecase-model.en --corpus \
>
> ~/mosesdecoder/corpus/english.tok.en
>
> but we got the error as follows-:
>
> train-truecaser.perl --model truecaser --corpus cased
> [--possiblyUseFirstToken] at
> /home/deeptiman/mosesdecoder/scripts/recaser/train-truecaser.perl line 16.
>
>
> i have searched in net , i have found that the first character every
> sentence should be Capitalized and our corpora also Capitalized but we
> can't able to execute this command.
>
>
>  Please go through this problem and help us out.
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


-- 
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support