Re: [Moses-support] accessing the compact format of the phrase-table
Hello Tom, Yes, I guess I will have to rebuild. I have the corpus so it will be ok... Thanks a lot. Cheers, Dimitar. Dimitar Shterionov | dimit...@kantanmt.com | Machine Translation Researcher www.KantanMT.com <http://www.kantanmt.com/> | Easy Translation - No Software. No Hardware. No Hassle MT. <https://www.facebook.com/KantanMT> <https://plus.google.com/+Kantanmt_cloudmachinetranslation> <https://twitter.com/KantanMT> <https://www.linkedin.com/company/kantanmt> <http://www.slideshare.net/kantanmt> <https://www.youtube.com/user/KantanMT> <http://kantanmtblog.com/> <https://kantanmt.com/rssfeeds.php> On 23 September 2016 at 12:10, Tom Hoar wrote: > Sorry Dimitar, > > I understand. I can't think of a way. If you have any of the output stages > of train-model.perl (assuming that's what you're using), you could resume > at a downstream step, for example `--do-steps 4-9`. Since you're saving > disk space, I wouldn't expect you saved things like the GIZA word alignment > files, etc. Rerunning train-model.perl from the beginning might be your > only way, assuming you saved the original corpus. > Best regards, > > Tom Hoar > Chief Executive Officer > *Precision Translation Tools Pte Ltd* > Singapore/Thailand > Web: www.precisiontranslationtools.com > Thailand Mobile: +66 87 345-1875 > Skype call: tahoar > Skype chat: tahoar > > > On 9/23/2016 4:45 PM, moses-support-requ...@mit.edu wrote: > > Date: Fri, 23 Sep 2016 10:44:57 +0100 > From: Dimitar Shterionov > Subject: Re: [Moses-support] accessing the compact format of the > phrase-table > To: Tom Hoar > Cc: moses-support@mit.edu > > Hello Tom, > > When I store the phrase-table for later use I store only the compact > version - it's simply smaller. > > Is there another way to get the original text phrase-table without > rebuilding? > > Cheers, > Dimitar. > > Dimitar Shterionov | dimit...@kantanmt.com | Machine Translation Researcher > www.KantanMT.com <http://www.kantanmt.com/> <http://www.kantanmt.com/> | Easy > Translation - No > Software. No Hardware. No Hassle MT. > <https://www.facebook.com/KantanMT> > <https://www.facebook.com/KantanMT><https://plus.google.com/+Kantanmt_cloudmachinetranslation> > > <https://plus.google.com/+Kantanmt_cloudmachinetranslation><https://twitter.com/KantanMT> > <https://twitter.com/KantanMT> <https://www.linkedin.com/company/kantanmt> > <https://www.linkedin.com/company/kantanmt><http://www.slideshare.net/kantanmt> > <http://www.slideshare.net/kantanmt> <https://www.youtube.com/user/KantanMT> > <https://www.youtube.com/user/KantanMT><http://kantanmtblog.com/> > <http://kantanmtblog.com/> <https://kantanmt.com/rssfeeds.php> > <https://kantanmt.com/rssfeeds.php> > > > > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] accessing the compact format of the phrase-table
Hello Tom, When I store the phrase-table for later use I store only the compact version - it's simply smaller. Is there another way to get the original text phrase-table without rebuilding? Cheers, Dimitar. Dimitar Shterionov | dimit...@kantanmt.com | Machine Translation Researcher www.KantanMT.com <http://www.kantanmt.com/> | Easy Translation - No Software. No Hardware. No Hassle MT. <https://www.facebook.com/KantanMT> <https://plus.google.com/+Kantanmt_cloudmachinetranslation> <https://twitter.com/KantanMT> <https://www.linkedin.com/company/kantanmt> <http://www.slideshare.net/kantanmt> <https://www.youtube.com/user/KantanMT> <http://kantanmtblog.com/> <https://kantanmt.com/rssfeeds.php> On 23 September 2016 at 10:37, Tom Hoar wrote: > Dimitar, is there any reason you can't go back to the original text > phrase-table that was used to create the compact format? > > > On 9/23/2016 4:26 PM, moses-support-requ...@mit.edu wrote: > > Date: Fri, 23 Sep 2016 10:24:51 +0100 > From: Dimitar Shterionov > Subject: Re: [Moses-support] accessing the compact format of the > phrase-table > To: Marcin Junczys-Dowmunt > Cc: moses-support@mit.edu > > Hi Marcin and thanks for the quick reply. > > With the queryPhraseTableMin do I always need to provide exact string to > match an entry or can I actually provide partial string? or regex? > Basically I want to extract all 1-grams. Any suggestions? > > Thanks once again. > Dimitar. > > Dimitar Shterionov | dimit...@kantanmt.com | Machine Translation Researcher > www.KantanMT.com <http://www.kantanmt.com/> <http://www.kantanmt.com/> | Easy > Translation - No > Software. No Hardware. No Hassle MT. > <https://www.facebook.com/KantanMT> > <https://www.facebook.com/KantanMT><https://plus.google.com/+Kantanmt_cloudmachinetranslation> > > <https://plus.google.com/+Kantanmt_cloudmachinetranslation><https://twitter.com/KantanMT> > <https://twitter.com/KantanMT> <https://www.linkedin.com/company/kantanmt> > <https://www.linkedin.com/company/kantanmt><http://www.slideshare.net/kantanmt> > <http://www.slideshare.net/kantanmt> <https://www.youtube.com/user/KantanMT> > <https://www.youtube.com/user/KantanMT><http://kantanmtblog.com/> > <http://kantanmtblog.com/> <https://kantanmt.com/rssfeeds.php> > <https://kantanmt.com/rssfeeds.php> > > > > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] accessing the compact format of the phrase-table
Thanks a lot Marcin. I was fearing that would be the case :). Cheers, Dimitar. Dimitar Shterionov | dimit...@kantanmt.com | Machine Translation Researcher www.KantanMT.com <http://www.kantanmt.com/> | Easy Translation - No Software. No Hardware. No Hassle MT. <https://www.facebook.com/KantanMT> <https://plus.google.com/+Kantanmt_cloudmachinetranslation> <https://twitter.com/KantanMT> <https://www.linkedin.com/company/kantanmt> <http://www.slideshare.net/kantanmt> <https://www.youtube.com/user/KantanMT> <http://kantanmtblog.com/> <https://kantanmt.com/rssfeeds.php> On 23 September 2016 at 10:26, Marcin Junczys-Dowmunt wrote: > Hi Dimitar, > full string unfortunately. This is a hash table, so no partial methods > possible. The source string itself isn't stored anywhere. > Best, > Marcin > > W dniu 23/09/16 o 10:24, Dimitar Shterionov pisze: > > Hi Marcin and thanks for the quick reply. > > With the queryPhraseTableMin do I always need to provide exact string to > match an entry or can I actually provide partial string? or regex? > Basically I want to extract all 1-grams. Any suggestions? > > Thanks once again. > Dimitar. > > Dimitar Shterionov | dimit...@kantanmt.com | Machine Translation > Researcher > > www.KantanMT.com <http://www.kantanmt.com/> | Easy Translation - No > Software. No Hardware. No Hassle MT. > > <https://www.facebook.com/KantanMT> > <https://plus.google.com/+Kantanmt_cloudmachinetranslation> > <https://twitter.com/KantanMT> <https://www.linkedin.com/company/kantanmt> > <http://www.slideshare.net/kantanmt> > <https://www.youtube.com/user/KantanMT> <http://kantanmtblog.com/> > <https://kantanmt.com/rssfeeds.php> > > > On 23 September 2016 at 10:19, Marcin Junczys-Dowmunt > wrote: > >> Hi, >> If you want a complete dump of the phrase table as text, this is not >> possible. The compact phrase table is not reversible. You can use >> queryPhraseTableMin to ask for the translations of specific phrases. >> Best, >> Marcin >> >> W dniu 23/09/16 o 10:16, Dimitar Shterionov pisze: >> >> Dear all, >> >> I want to extract some specific data from the table after the model has >> been built. Before it is compacted I have it in raw format and it is easy. >> But I don't know how to process and read the compacted format of the >> phrase-table. Is there a way to read the phrase-table.minphr format as >> text? Any suggestions? >> >> Thank you very much. >> Kind regards, >> Dimitar. >> >> >> ___ >> Moses-support mailing >> listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support >> >> ___ Moses-support mailing >> list Moses-support@mit.edu http://mailman.mit.edu/mailman >> /listinfo/moses-support > > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] accessing the compact format of the phrase-table
Hi Marcin and thanks for the quick reply. With the queryPhraseTableMin do I always need to provide exact string to match an entry or can I actually provide partial string? or regex? Basically I want to extract all 1-grams. Any suggestions? Thanks once again. Dimitar. Dimitar Shterionov | dimit...@kantanmt.com | Machine Translation Researcher www.KantanMT.com <http://www.kantanmt.com/> | Easy Translation - No Software. No Hardware. No Hassle MT. <https://www.facebook.com/KantanMT> <https://plus.google.com/+Kantanmt_cloudmachinetranslation> <https://twitter.com/KantanMT> <https://www.linkedin.com/company/kantanmt> <http://www.slideshare.net/kantanmt> <https://www.youtube.com/user/KantanMT> <http://kantanmtblog.com/> <https://kantanmt.com/rssfeeds.php> On 23 September 2016 at 10:19, Marcin Junczys-Dowmunt wrote: > Hi, > If you want a complete dump of the phrase table as text, this is not > possible. The compact phrase table is not reversible. You can use > queryPhraseTableMin to ask for the translations of specific phrases. > Best, > Marcin > > W dniu 23/09/16 o 10:16, Dimitar Shterionov pisze: > > Dear all, > > I want to extract some specific data from the table after the model has > been built. Before it is compacted I have it in raw format and it is easy. > But I don't know how to process and read the compacted format of the > phrase-table. Is there a way to read the phrase-table.minphr format as > text? Any suggestions? > > Thank you very much. > Kind regards, > Dimitar. > > > ___ > Moses-support mailing > listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support > > > > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] accessing the compact format of the phrase-table
Dear all, I want to extract some specific data from the table after the model has been built. Before it is compacted I have it in raw format and it is easy. But I don't know how to process and read the compacted format of the phrase-table. Is there a way to read the phrase-table.minphr format as text? Any suggestions? Thank you very much. Kind regards, Dimitar. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] after tuning UnknownWordPenalty parameter is missing
Hello, I am trying to tune an engine using mert. I run the mert-moses.pl script without filtering (i.e., --no-filter-phrase-table). At the end I use substitute-weights.perl to have a new ini file. I noticed that in the new ini file I don't have the UnknownWordPenalty paramter. The original file looks like this (some lines omitted): ... # feature functions [feature] UnknownWordPenalty WordPenalty PhrasePenalty ... # dense weights for feature functions [weight] UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5 And the new ini file looks like this: ... # feature functions [feature] UnknownWordPenalty WordPenalty PhrasePenalty ... ... [weight] LexicalReordering0= 0.0762169 0.00608827 0.0268392 0.123948 0.0511544 0.127807 Distortion0= 0.0304069 LM0= 0.0635183 WordPenalty0= -0.268702 PhrasePenalty0= 0.0536959 TranslationModel0= 0.0525019 0.0459397 0.000797115 0.0723839 Does anyone have any idea why is the UnknownWordPenalty missing? Thank you for your support. Kind regards, Dimitar. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] mosesserver vs moses
Dear all, I have noticed that moses and mosesserver return different translations sometimes. I am wondering what are the differences between moses and mosesserver with respect to decoding? Are there different parameters set up during compilation? Can it be ensured that moses and mosesserver will return the same results? Thank you, Kind regards, Dimitar ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] moses 2.1 and mosesserver different result
Hello, I have a question regarding the differences between Moses and mosesserver. I have noticed that sometimes Moses server translates segments differently from Moses. Is that normal? or is there some good explanation? I use the same parameters when invoking both. Thank you, Kind regards, Dimitar. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
