On Monday 09 August 2010 20:43:53 Chris Dyer wrote: > I'm sorry I haven't had an opportunity to look into this yet > (hopefully later this evening). But, one thing that you need to do is > make sure that your config file has an entry setting the maximum > phrase size to a very large number, which prevents some bad pruning > from taking place that can lead to crashes. > > [max-phrase-length] > 9999 > > Chris >
Hi, thanks for your answer! I run moses with the option: -max-phrase-length 100000 this should do, right? regards, Sylvain > On Mon, Aug 9, 2010 at 12:28 PM, Sylvain Raybaud > > <sylvain.rayb...@loria.fr> wrote: > > Some more information before I leave on a two weeks break. > > > > I've run it on a different "real world" graph (test-07), it also crashes, > > apparently at the same location but with "free(): invalid next size > > (normal): 0x000000001db930d0" (see back trace attached). > > > > moses output (with -v 2) shows so called "shortest paths" which I would > > expect to be computed only if the lattice is completely loaded > > > > if someone feels like running some tests he'll find attached: > > test-07.slf.gz generated by sphinx > > test-07.moses-verbose2.stderr.gz moses output > > backtrace.gz > > latest version of script > > > > when I come back I'll try to add more debug output to moses so that I can > > pinpoint where the error occurs (I think this is gonna be painful since > > I've never had a look in moses source code so far) > > > > see you > > > > Sylvain > > > > On Monday 09 August 2010 15:56:31 Sylvain Raybaud wrote: > >> Hello > >> > >> Here is the latest version of the script, which replace noise > >> information with epsilon transitions. Moses successfully translates > >> very small "toy" lattice generated with this script but on a real world > >> example > >> (http://perso.crans.org/raybaud/realworld.slf.gz) Moses fails with glibc > >> complaining about a corrupted double-linked list: > >> sylv...@markov /global/markov/raybauds/XP/S2T $ nice > >> ~/softs/moses/moses- cmd/src/moses -max-phrase-length 100000 -inputtype > >> 2 -weight-i 1 -f working- dir/filtered-OlliRehn-1/moses.ini < > >> realworld.plf > > >> realworld.plf.translated Defined parameters (per moses.ini or switch): > >> config: working-dir/filtered-OlliRehn-1/moses.ini > >> distortion-file: 0-0 msd-bidirectional-fe 6 > >> /global/markov/raybauds/XP/S2T/working-dir/filtered-OlliRehn-1/reorderin > >> g-t able distortion-limit: 6 > >> input-factors: 0 > >> inputtype: 2 > >> lmodel-file: 0 0 5 > >> /home/sylvain/GMR/SYSTEMS/WMT09/lm/europarl.lm mapping: 0 T 0 > >> max-phrase-length: 100000 > >> ttable-file: 0 0 5 > >> /global/markov/raybauds/XP/S2T/working-dir/filtered- > >> OlliRehn-1/phrase-table > >> ttable-limit: 20 > >> weight-d: 0.054012 0.196064 -0.003181 0.081071 0.086735 0.062332 > >> 0.090617 > >> weight-i: 1 > >> weight-l: 0.064254 > >> weight-t: 0.013876 0.081935 0.059305 0.013773 0.145934 > >> weight-w: 0.046909 > >> Loading lexical distortion models...have 1 models > >> [...] > >> Finished loading phrase tables : [13.000] seconds > >> IO from STDOUT/STDIN > >> Created input-output object : [13.000] seconds > >> WARN: probability > 1: 1.492 > >> *** glibc detected *** /home/sylvain/softs/moses/moses-cmd/src/moses: > >> corrupted double-linked list: 0x000000001e7d8610 *** > >> > >> I used the very latest version, fresh trunk from this morning. Of course > >> I can't say if this is because my lattices are ill formed or because of > >> moses. So I recompiled it without optimisation (CFLAGS = -g -ggdb -O0, > >> CXXFLAGS also) to get a full backtrace: > >> > >> #0 0x00007ffff70f81b5 in raise () from /lib/libc.so.6 > >> #1 0x00007ffff70f95e0 in abort () from /lib/libc.so.6 > >> #2 0x00007ffff71333d7 in __libc_message () from /lib/libc.so.6 > >> #3 0x00007ffff7138966 in malloc_printerr () from /lib/libc.so.6 > >> #4 0x00007ffff713a398 in _int_free () from /lib/libc.so.6 > >> #5 0x00007ffff713d71c in free () from /lib/libc.so.6 > >> #6 0x000000000040d820 in __gnu_cxx::new_allocator<unsigned > >> long>::deallocate (this=0x1d8d7738, __p=0x1da04c60) at > >> /usr/lib/gcc/x86_64-pc-linux- > >> gnu/4.4.4/include/g++-v4/ext/new_allocator.h:95 > >> #7 0x000000000040c45e in std::_Bvector_base<std::allocator<bool> > >> > >> >::_M_deallocate (this=0x1d8d7738) at /usr/lib/gcc/x86_64-pc-linux- > >> > >> gnu/4.4.4/include/g++-v4/bits/stl_bvector.h:444 > >> #8 0x000000000040b6a1 in ~_Bvector_base (this=0x1d8d7738, > >> __in_chrg=<value optimized out>) at > >> /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4/include/g++- > >> v4/bits/stl_bvector.h:430 > >> #9 0x000000000040a6d6 in ~vector (this=0x1d8d7738, __in_chrg=<value > >> optimized out>) at /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4/include/g++- > >> v4/bits/stl_bvector.h:547 > >> #10 0x00000000004b87df in std::_Destroy<std::vector<bool, > >> std::allocator<bool> > >> > >> > > (__pointer=0x1d8d7738) at /usr/lib/gcc/x86_64-pc-linux- > >> > >> gnu/4.4.4/include/g++-v4/bits/stl_construct.h:83 > >> #11 0x00000000004b82e1 in > >> std::_Destroy_aux<false>::__destroy<std::vector<bool, > >> std::allocator<bool> > >> > >> >*> (__first=0x1d8d7738, __last=0x1d8e56a8) > >> > >> at /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4/include/g++- > >> v4/bits/stl_construct.h:93 > >> #12 0x00000000004b77ee in std::_Destroy<std::vector<bool, > >> std::allocator<bool> > >> > >> >*> (__first=0x1d8c0470, __last=0x1d8e56a8) at > >> >/usr/lib/gcc/x86_64-pc-linux- > >> > >> gnu/4.4.4/include/g++-v4/bits/stl_construct.h:116 > >> #13 0x00000000004b68f9 in std::_Destroy<std::vector<bool, > >> std::allocator<bool> > >> > >> >*, std::vector<bool, std::allocator<bool> > > (__first=0x1d8c0470, > >> > >> __last=0x1d8e56a8) > >> at /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4/include/g++- > >> v4/bits/stl_construct.h:142 > >> #14 0x00000000004b6030 in ~vector (this=0x7fffffffcc30, __in_chrg=<value > >> optimized out>) at /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4/include/g++- > >> v4/bits/stl_vector.h:313 > >> #15 0x00000000004b5141 in Moses::WordLattice::Read (this=0x1caf2d70, > >> in=..., factorOrder=...) at WordLattice.cpp:110 > >> #16 0x0000000000416d2b in IOWrapper::GetInput (this=0x1caf0760, > >> inputType=0x1caf2d70) at IOWrapper.cpp:171 > >> #17 0x0000000000418c1c in ReadInput (ioWrapper=..., > >> inputType=Moses::WordLatticeInput, sour...@0x7fffffffcf48) at > >> IOWrapper.cpp:511 #18 0x00000000004086c4 in main (argc=11, > >> argv=0x7fffffffd0f8) at Main.cpp:398 > >> > >> As I understand it, it crashed while it reads the lattice. So there must > >> be something wrong in my script. It can't be probability above one, can > >> it? > >> > >> Any hint is more than welcome... > >> > >> regards, > >> > >> Sylvain > >> > >> On Friday 06 August 2010 11:44:25 Sylvain Raybaud wrote: > >> > Hello > >> > > >> > A few days ago I came to you asking if someone knew of a script to > >> > turn > >> > > >> > word lattice from SLF (HTK format) into PLF (Moses input format). It > >> > seems that doesn't exist yet, so here is my proposal (python file > >> > attached). > >> > > >> > I tried to include all necessary information in help message and in > >> > file header. > >> > > >> > basic use : > >> > slf2plf input > output > >> > slf2plf < input > output > >> > slf2plf -h for help > >> > > >> > it works on small "toy" lattices, I checked the results. I've run it > >> > on real world lattices but I have not checked the results yet. > >> > > >> > I tried to keep as much flexibility of SLF format as possible, but > >> > sublattices won't work yet (they will be parsed but in the resulting > >> > graph they will appear as a single node. I haven't checked this > >> > though). > >> > > >> > Also, for the moment the software requires that words are specified on > >> > links (not nodes) and that acoustic and language model scores are > >> > present. The resulting score in output graph is: > >> > acoustic+language*lmscale > >> > > >> > all testing, remarks and requests welcome (of course). I can make the > >> > script accessible on a mercurial repository if you want. > >> > > >> > Hope this helps > >> > > >> > regards, > > > > -- > > Sylvain > > > > _______________________________________________ > > Moses-support mailing list > > Moses-support@mit.edu > > http://mailman.mit.edu/mailman/listinfo/moses-support -- Sylvain _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
