I get the same problem also.
The issue seems to be with obtuse unix sort command.
In some versions of sort, it may be sorting by a hash index, rather than
alphanumberic sort. Therefore, you need to force it to do an alphanumberic
sort
sort -t"|" -k1,1
This fixed it for me. It's not the perfect solution, but it'll do for now.
Unix - guaranteed to give you a headache
Hieu Hoang
www.hoang.co.uk/hieu
--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
On Behalf Of Megan Elmore ([EMAIL PROTECTED])
Sent: 14 July 2008 16:17
To: moses-support@mit.edu
Subject: [Bulk] Re: [Moses-support] phrase table memory issue
Hello again,
Yes, I was using the command as described on the Moses web site at
http://www.statmt.org/moses/?n=Moses.AdvancedFeatures. I have also tried
piping the results from sort through uniq before piping it into
processPhraseTable and encountered the same error. Perhaps I am unaware of
some option to provide to sort or uniq to alleviate this problem. At what
step in the code for processPhraseTable would this error be generated?
-Megan
----- Original Message -----
From: Philipp Koehn <[EMAIL PROTECTED]>
Date: Monday, July 14, 2008 1:46 am
Subject: Re: [Moses-support] phrase table memory issue
To: "Megan Elmore ([EMAIL PROTECTED])" <[EMAIL PROTECTED]>
Cc: moses-support@mit.edu
> Hi,
>
> are you sorting the phrase table?
> Check the command as described on the Moses web site.
>
> -phi
>
> On Wed, Jul 9, 2008 at 8:21 PM, Megan Elmore ([EMAIL PROTECTED])
> <[EMAIL PROTECTED]> wrote:
> > Hello,
> >
> > Thanks very much for your quick reply. I am currently trying to
> generate a binary phrase table but am getting an error:
> >
> > ERROR: xsource phrase already inserted (B)!
> > line(17): '000 - ||| 000 รข ||| (0) (1) ||| (0) (1) ||| 0.5
> 0.540651 0.25 0.178456 2.718'
> > f: 2 0 2
> >
> > Does this indicate a problem with my phrase table or with the
> processPhraseTable process? In the event that I need to run the
> training process differently - what error or warning messages, if any,
> that are generated during the training process would let me know of
> any errors in my phrase table?
> >
> > Currently, the phrase table generated during the training process
> was left in a gzip'ped format as phrase-table.0-0.gz - I am not sure
> if this is relevant, but maybe the odd naming (as opposed to just
> "phrase-table" listed in the online documentation) sheds light on a
> step of the training process that did not complete normally for me?
> >
> > -Megan
> >
> > ----- Original Message -----
> > From: Philipp Koehn <[EMAIL PROTECTED]>
> > Date: Wednesday, July 9, 2008 2:25 pm
> > Subject: Re: [Moses-support] phrase table memory issue
> > To: "Megan Elmore ([EMAIL PROTECTED])" <[EMAIL PROTECTED]>
> > Cc: moses-support@mit.edu
> >
> >> Hi,
> >>
> >> this is a sign that the phrase table is too big to load into
> memory,>> there are three options:
> >> (a) use the binary phrase table
> >> (b) filter the phrase table for the test set you are using
> >> (c) both
> >>
> >> See the Moses web page for details.
> >>
> >> -phi
> >>
> >> On Wed, Jul 9, 2008 at 7:17 PM, Megan Elmore
> ([EMAIL PROTECTED])>> <[EMAIL PROTECTED]> wrote:
> >> > Hello,
> >> >
> >> > I have installed Moses and run the training process using the
> >> europarl corpus but am now having problems with the decoder loading
> >> the phrase table. Like a previous message on this list, I am
> >> getting the error
> >> >
> >> > terminate called after throwing an instance of 'std::bad_alloc'
> >> > what(): St9bad_alloc
> >> > Aborted
> >> >
> >> > while the decoder is trying to load the phrase table, regardless
> >> of the machine I run the decoder on (I've tried four now). Is there
> >> a way I can optimize how much space the phrase table uses? Or is
> >> there something that could be going wrong in the training or
> >> decoding processes? I am not sure where to look for the error but
> >> with a little direction I could keep trying to debug it.
> >> >
> >> > Thanks,
> >> > -Megan E.
> >> > _______________________________________________
> >> > Moses-support mailing list
> >> > Moses-support@mit.edu
> >> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >> >
> >> >
> >>
> >
> >
>
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
