[Moses-support] large language model building

[Li Xianhua]

 

 

Hello everyone,

 

 Recently we’re working on building a huge language model. The
corpus size is about 3G, and our computer memory is 40G.

 We failed to build a 5-gram language model with SRILM because of
insufficient memory.

 We divided the corpus into 2 parts and trained language model on
both of them separately, however ,this still failed.

 I would like to know, how huge the corpus could approximately be
when training a 5-gram language model from SRILM？

 As my colleague reported, it cost 9G memory when the corpus size is
150M to train a 5-gram language model. Is this normal? 

 We are now trying to use IRSTLM. Is there any suggestions ?

 

 



Best wishes!

Xianhua Li 

 

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] counting the score of features

[Neda NoorMohammadi]

Thank you very much in answer.

On Tue, Aug 16, 2011 at 7:49 PM, Barry Haddow  wrote:

> On Tuesday 16 Aug 2011 16:00:37 Neda NoorMohammadi wrote:
> > thanks in your answer.
> >
> > I have completely investigated the the part of the code which loads the
> > models.
> > For loading lexical reordering model LexicalReorderingTable class is
> > activated , but in that code the probabilities are directly loaded from
> the
> > file with no changes and then will be stored on the memory!
> >
>
> Hi Neda
>
> The following line takes logs:
>
> std::transform(p.begin(),p.end(),p.begin(),TransformScore);
>
> cheers - Barry
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] can not find the GIZA++

[Thanh-Le Ha]

James,

Sorry for not reading your question carefully. I am not sure that the
featured version of GIZA++ fully supports incremental training.

On Wed, Aug 17, 2011 at 7:27 AM, Thanh-Le Ha  wrote:

> Dear James,
>
> You could find GIZA++ in this correct link:
> http://code.google.com/p/giza-pp/downloads/list
> Using former versions of GIZA++ might lead to the incompatibility problem
> with gcc.
>
> Regards,
> Thanh-Le.
>
>
> On Wed, Aug 17, 2011 at 5:26 AM, 蒋乾  wrote:
>
>> Hi all,
>> Now I am trying doing incremental training. As is described in
>>
>>  http://www.statmt.org/moses/?n=Moses.AdvancedFeatures
>> "Initial Training This section describes how to initially train and use a
>> model which support incremental training. Setup the EMS as normal, but
>> use the version of GIZA++ found at http://code.google.com/p/inc-giza-pp/";
>>
>> Unfortunately, I can not find the required GIZA++ in the metioned website.
>> 1. Where could I download from alternatively?
>> 2. Is there any problem if I use the former version of GIZA++?
>>
>>
>> Thank you.
>>
>> Best regards,
>>
>> James
>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] can not find the GIZA++

[Thanh-Le Ha]

Dear James,

You could find GIZA++ in this correct link:
http://code.google.com/p/giza-pp/downloads/list
Using former versions of GIZA++ might lead to the incompatibility problem
with gcc.

Regards,
Thanh-Le.


On Wed, Aug 17, 2011 at 5:26 AM, 蒋乾  wrote:

> Hi all,
> Now I am trying doing incremental training. As is described in
>
>  http://www.statmt.org/moses/?n=Moses.AdvancedFeatures
> "Initial Training This section describes how to initially train and use a
> model which support incremental training. Setup the EMS as normal, but use
> the version of GIZA++ found at http://code.google.com/p/inc-giza-pp/ "
>
> Unfortunately, I can not find the required GIZA++ in the metioned website.
> 1. Where could I download from alternatively?
> 2. Is there any problem if I use the former version of GIZA++?
>
>
> Thank you.
>
> Best regards,
>
> James
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] can not find the GIZA++

[蒋乾]

Hi all,
Now I am trying doing incremental training. As is described in

 http://www.statmt.org/moses/?n=Moses.AdvancedFeatures
"Initial Training This section describes how to initially train and use a
model which support incremental training. Setup the EMS as normal, but use
the version of GIZA++ found at http://code.google.com/p/inc-giza-pp/ "

Unfortunately, I can not find the required GIZA++ in the metioned website.
1. Where could I download from alternatively?
2. Is there any problem if I use the former version of GIZA++?


Thank you.

Best regards,

James
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] DEADLINE EXTENSION CfP: Third Joint EM+/CNGL Workshop (JEC2011) 'Bringing MT to the User: Research Meets Translators'

[Ventsislav Zhechev]

FINAL CALL FOR PAPERS
DEADLINE EXTENSION

“Bringing MT to the User: Research Meets Translators”
Third Joint EM+/CNGL Workshop (JEC 2011)
http://web.me.com/emcnglworkshop/JEC2011

The EuroMatrixPlus Project (http://www.euromatrixplus.eu), the Centre for Next 
Generation Localisation (CNGL) (http://cngl.ie), the Directorate-General for 
Translation (DGT, European Commission) (http://ec.europa.eu/dgs/translation) 
and Autodesk (http://www.autodesk.ch) are co-organising the Third Joint 
EM+/CNGL Workshop (JEC 2011), entitled “Bringing MT to the User: Research Meets 
Translators”. 
The JEC 2011 workshop will be hosted by the Directorate General for Translation 
(DGT) (http://ec.europa.eu/dgs/translation) in Luxembourg on October 14th, 
2011. In keeping with previous JECs, the format of the workshop is highly 
interactive with research paper presentations, invited talks and a panel 
discussion.


Premise:
Recent years have seen a revolution in MT triggered by the emergence of 
statistical approaches and improvements in translation quality. MT (rule-based, 
statistical and hybrid) is now available for many languages for free (on the 
Web) or for a fee and MT technologies are making strong inroads into the 
corporate localisation and translation industries as well as large public and 
administrative organisations dealing with multi-lingual content. Open-source MT 
solutions are competing with proprietary products. Increasing numbers of 
(professional) translators are post-editing TM/MT output. MT is a reality for 
internet users accessing and gisting content which is not available in their 
native language.
At the same time, there has been a degree of disconnect between mainstream 
academic research and conferences on MT, often (and rightly so) focusing on 
algorithms to improve translation quality, and many of the important practical 
issues that need to be addressed to make MT maximally useful in real 
translation and localisation workflows, with human translators and users in 
general.

Objectives:
JEC 2011 brings together translators, users, academic and industrial MT 
researchers and developers to discuss issues that are most important in real 
world industrial settings and applications involving MT, but currently 
under-represented in research circles.


Call for Research Papers:
We solicit full research papers with industry, academic and/or user background 
to highlight real-world issues that need to be tackled by new research and 
recent advancements that improve translation quality, as well as novel and 
successful methods for the integration of machine translation with translation 
memories, localisation workflows, human translators and users. Papers should 
present clearly identifiable problem statements, research methodologies, 
measurable outcomes and evaluation. Papers are reviewed anonymously. 
Papers should follow the submission guidelines listed on the workshop website 
(http://web.me.com/emcnglworkshop/JEC2011/Call_for_Papers.html), with the 
maximum length being 10 pages in A4 format, including references. Please, do 
not include your name in the paper text and avoid overt self-references to 
facilitate the blind review process.
If a paper is accepted, at least one author will have to register for JEC 2011 
and travel to Luxembourg to present the paper.
Workshop proceedings will be made available in hard-copy by DGT, and will be 
available for download on the JEC 2011 website. Publication of selected revised 
and expanded papers from JEC 2010 and JEC 2011 in journal or book form is 
currently under negotiation. 

Topics include but are not limited to:
• Human Factors and MT
• Introducing MT into large organisations
• MT and language technologies for SMEs
• MT/TM in Localisation/Translation and Content Management Workflows
• MT/TM Combinations
• Post-Editing Support for MT
• MT and Monolingual Post-Editing
• Smart Learning from Post-Edits
• Interactive MT
• MT Confidence Scores and Post-Editing Effort
• Training Data for MT: Size, Domain and Quality
• Data Cleanup and Preparation for MT
• Meta-Data Mark-Up/Annotation and MT
• Terminology and MT
• Interoperability and Localisation/Translation Workflows
• Standards and Localisation/Translation Workflows
• MT Evaluation
• Costing/Pricing MT
• MT for Free/for a Fee
• Rule-Based, Statistical and Hybrid MT
• Linguistic resources for MT
• Computing Resources for MT
• MT in the Cloud
• MT and the Crowd
• MT, Games, Video and TV Localisation
• (Machine) Translation in Context
• Social Aspects of (Machine) Translation: Access to Information as a Human 
Right

Deadlines (all 23:59 GMT -11):
21th August:  Submission deadline for papers (EXTENDED DEADLINE)
12th September: Announcement for submitted papers
30th September: Camera-ready deadline for accepted papers
14th October: Workshop takes place at DGT in Luxembourg

Submission URL: http:/

Re: [Moses-support] counting the score of features

[Barry Haddow]

On Tuesday 16 Aug 2011 16:00:37 Neda NoorMohammadi wrote:
> thanks in your answer.
> 
> I have completely investigated the the part of the code which loads the
> models.
> For loading lexical reordering model LexicalReorderingTable class is
> activated , but in that code the probabilities are directly loaded from the
> file with no changes and then will be stored on the memory!
> 

Hi Neda

The following line takes logs:

std::transform(p.begin(),p.end(),p.begin(),TransformScore);

cheers - Barry
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] counting the score of features

[Neda NoorMohammadi]

thanks in your answer.

I have completely investigated the the part of the code which loads the
models.
For loading lexical reordering model LexicalReorderingTable class is
activated , but in that code the probabilities are directly loaded from the
file with no changes and then will be stored on the memory!

Does occur the loading model in another part of code?


On Tue, Aug 16, 2011 at 6:58 PM, Philipp Koehn  wrote:

> Hi,
>
> in all text model files (the reordering model, the translation model),
> probabilities are used. These are converted into log-probabilities
> upon loading, so internally log-probabilities are used.
>
> -phi
>
> On Tue, Aug 16, 2011 at 1:42 PM, Neda NoorMohammadi
>  wrote:
> > Hello,
> >
> > In the Moses manual is mentioned that the log linear model is used for
> > counting the sentences scores .
> >
> > For scoring the features in the ScoreComponentCollection class the score
> of
> > each feature is added to its previous value, that shows it is using the
> log
> > value of the probabilities.
> > but in the lexical reordering table, the saved numbers, does not seem
> log.
> > (they are between 0 and 1)
> >
> > What is wrong?
> >
> > sincerely,
> >
> > Neda Noormohammadi
> >
> >
> > ___
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> >
>



-- 
نورمحمدی
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] counting the score of features

[Philipp Koehn]

Hi,

in all text model files (the reordering model, the translation model),
probabilities are used. These are converted into log-probabilities
upon loading, so internally log-probabilities are used.

-phi

On Tue, Aug 16, 2011 at 1:42 PM, Neda NoorMohammadi
 wrote:
> Hello,
>
> In the Moses manual is mentioned that the log linear model is used for
> counting the sentences scores .
>
> For scoring the features in the ScoreComponentCollection class the score of
> each feature is added to its previous value, that shows it is using the log
> value of the probabilities.
> but in the lexical reordering table, the saved numbers, does not seem log.
> (they are between 0 and 1)
>
> What is wrong?
>
> sincerely,
>
> Neda Noormohammadi
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] KenLM build-binary trie problems (SRILM ARPA file)

[Kenneth Heafield]

Revision 4145 fixes this.  All probing binary files will continue to
work.  All trie files built from an ARPA containing  will continue
to work.  Trie binary files built from an ARPA without  must be
rebuilt using build_binary; after this is done it will no longer
segfault and will return correct values.  There isn't a nice way to
check for these broken binary files and I'm hesitant to increment the
version number because that would require everybody to rebuild. 

The issue was that trie writes corrected (including ) counts to the
header but the vocabulary lookup table was sized based on the counts
given in the ARPA file.  When  is missing from the ARPA file, I now
pad the vocabulary to the size it expects for the corrected count. 

Sorry it took so long! 

Kenneth

On 08/15/11 22:12, Kenneth Heafield wrote:
> Ok I have reproduced the problem.  It only happens when the ARPA file
> is missing  and is probably an off-by-one on vocabulary size. 
> I'll have a fix soon. 
>
> Kenneth
>
> On 08/15/11 19:20, Kenneth Heafield wrote:
>> Hi,
>>
>> Back from vacation and sorry but I'm having trouble reproducing
>> this locally. 
>>
>> - Latest Moses (revision 4143); I haven't made any changes that
>> should impact language modeling since 4096.
>> - svn status says the relevant source code is unmodified.
>> - Tried an SRI model, including rebuilding with build_binary that
>> ships with Moses. 
>> - Ran threaded and not threaded. 
>>
>> Can you send me your very small SRILM model?  Does it have ? 
>>
>> Kenneth
>>
>> On 08/04/11 11:42, Kenneth Heafield wrote:
>>> Sorry I am slow to respond. This is my first thing to look at, but I
>>> am traveling a lot through the 14th.
>>>
>>> Alex Fraser  wrote:
>>>
>>> Hi Kenneth --
>>>
>>> Latest revision, 4096. Single threaded also crashes.
>>>
>>> Cheers, Alex
>>>
>>>
>>> On Fri, Jul 29, 2011 at 6:00 PM, Kenneth Heafield  
>>> wrote:
>>> > Hi,
>>> >
>>> >There was a problem with this; thought it was fixed but maybe 
>>> it came
>>> > back.  Which revision are you running?  Does it still happen if you 
>>> run
>>> > single-threaded?
>>> >
>>> > Kenneth
>>> >
>>> > On 07/29/11 09:39, Alex Fraser wrote:
>>> >> Hi Folks,
>>> >>
>>> >> Tom Hoar previously mentioned that he had a problem with KenLMs built
>>> >> from SRILM crashing Moses.
>>> >>
>>> >> Fabienne Cap and I also have had a problem with this. It seems to be
>>> >> restricted to using the trie option with build-binary.
>>> >>
>>> >> Ken, if you have any problems repr!
>>>  oducing
>>> this, please let me know. I
>>> >> can send you a very small SRILM trained language model that crashes
>>> >> moses when converted to binary with the trie option, but works fine 
>>> as
>>> >> a probing binary and using the original ARPA. (BTW, this is running
>>> >> the decoder multi-threaded and the crash comes at some point during
>>> >> decoding the first sentence, not during loading files)
>>> >>
>>> >> Cheers, Alex
>>> >>
>>> 
>>>
>>> >> Moses-support mailing list
>>> >> Moses-support@mit.edu
>>> >> http://mailman.mit.edu/mailman/listinfo/moses-support
>>> >
>>> >
>>> 
>>>
>>> > Moses-support mailing list
>>> > Moses-support@mit.edu
>>> > http://mailman.mit.edu/mailman/listinfo/moses-support
>>> >
>>>
>>>
>>> ___
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] JOB OPPORTUNITY IN LUXEMBOURG

[Riad, Ali]

Dear All,

I allow myself to use this network of experts in the field of machine 
translation as part of a large project that will start soon in Luxembourg. 
Indeed, one of my clients, specialized in IT services is currently looking for 
several "Computational Linguists" focused on "Statistical Machine Translation" 
for a challenging project in a European institution in Luxembourg.


Indeed, given the large number of people from across the European Union who are 
working in this institution, people sometimes have problems with communication 
in writing. In order to solve this problem this particular institution seeks to 
develop a machine translation tool that will be developed from scratch


Please find below a brief description of the position:

Écouter

Lire phonétiquement



Dictionnaire

Bottom of Form

We are looking for experts in statistical machine translation (SMT).
Required experience and qualifications:
§  PhD or MSc/Diploma in computational linguistics or computer science with 
excellent knowledge of machine translation (MT) and/or machine learning for 
natural language processing.
§  Familiarity with recent developments in SMT and the motivation to apply 
novel approaches emanating from ongoing research to practical translation tasks 
in many languages and to systematically evaluate their suitability.
§  Ability to write scientific papers and technical documents in English.
§  Open-mindedness and the will to collaborate closely within an international, 
multidisciplinary team, motivation to work with a multilingual user community.
§  Experience in designing and running large-scale experiments, including the 
necessary skills to manage large data sets and long-running experiments in a 
distributed computational environment would be highly desirable, interest in 
developing such skills is mandatory.
§  Experience with both statistical and grammar-based approaches to MT would be 
highly desirable.
§  Programming skills in languages like Perl, Python, C, or C++ would be a 
strong plus.
§  Good oral and written command of English is required; familiarity with 
additional EU languages (preferably Eastern/Northern European) would be a 
strong plus.
§  Job location: Luxembourg
If you are interested by this position, do not hesitate to provide me with your 
resume and /or your availabilities for a first phone conversation.

If you are not interested, it would be very helpful to forward this opportunity 
to persons that might be interested.

I look forward to hearing from you soon.


Best Regards,


Ali RIAD
Sales & Recruitment Consultant

ELAN - A Manpower Professional Company

Rue de l'Industrie, 20
L-8399 Windhof

Tel:   + 352 27 39 51 03 21
Fax:  + 352 27 39 51 03 24
Mobile   + 352 661 209 107
Email:   ali.r...@elanit.lu
Web: www.elanit.lu

Elan awarded Top Employer and Top IT Employer status.
Best examples of HR management.

Recognised by CRF - IDENTIFYINGTOPPERFORMERS.

P Please consider the environment before printing this mail.

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] counting the score of features

[Neda NoorMohammadi]

Hello,

In the Moses manual is mentioned that the log linear model is used for
counting the sentences scores .

For scoring the features in the ScoreComponentCollection class the score of
each feature is added to its previous value, that shows it is using the log
value of the probabilities.
but in the lexical reordering table, the saved numbers, does not seem log.
(they are between 0 and 1)

What is wrong?

sincerely,

Neda Noormohammadi
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] memory leak in PhraseDictionaryTree.cpp

[Barry Haddow]

Hi Marc

Whilst this is technically a leak because the memory is allocated but never 
freed,  it's not really a leak as it does not cause the process to grow in 
size.

There are only ever two WordVoc objects (1 source, 1 target) allocated for 
each binarised phrase table, and these objects are allocated when the phrase 
table is loaded and used until decoding finishes. 

I think it keeps the code slightly simpler to 'leak' in this way, and since 
the memory is just abandoned on the heap at shutdown, it allows the decoder to 
exit quickly,

cheers - Barry

On Tuesday 16 Aug 2011 10:33:19 Marc LEGENDRE wrote:
> Hello,
> 
> at trunk/moses/src/PhraseDictionaryTree.cpp:131, in
> static WordVoc* ReadVoc(const std::string& filename),
> there's a 'new' for some memory (WordVoc* voc = new WordVoc();),
> the pointer of which is stored in a static map
> (static std::map vocs), and which is never freed.
> This, I noticed, may lead to a 10~20 MB leak at least.
> 
> Was it willfully forgotten because no easy solution was found and
> because it was considered harmless ? (compared with the huge size
> of LMs; and because it is needed as long as Moses is running ?)
> 
> I used boost::shared_ptr in my branch to deal with this,
> but I understood that you don't want to use boost in the trunk.
> I didn't find another solution either, I'm just asking.
> 
> Regards,
> Marc Legendre
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
> 
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] memory leak in PhraseDictionaryTree.cpp

[Marc LEGENDRE]

Hello,

at trunk/moses/src/PhraseDictionaryTree.cpp:131, in
static WordVoc* ReadVoc(const std::string& filename),
there's a 'new' for some memory (WordVoc* voc = new WordVoc();),
the pointer of which is stored in a static map
(static std::map vocs), and which is never freed.
This, I noticed, may lead to a 10~20 MB leak at least.

Was it willfully forgotten because no easy solution was found and 
because it was considered harmless ? (compared with the huge size
of LMs; and because it is needed as long as Moses is running ?)

I used boost::shared_ptr in my branch to deal with this,
but I understood that you don't want to use boost in the trunk.
I didn't find another solution either, I'm just asking.

Regards,
Marc Legendre
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] KenLM build-binary trie problems (SRILM ARPA file)

[Tom Hoar]

Thanks Miles. I'll update my SRILM scripts accordingly. 

I've
picked-apart IRSTLM's build-lm.sh script, and there's no switch there.
I'm confident Ken can update his code, but is there a way to add this 
token to the IRSTLM converted ARPA file for use with KenLM? 

If there's
no definitive answer, I'll build one SRILM with -unk and one IRSTLM/ARPA
from a small corpus and look for the differences. 

Tom 

On Tue, 16 Aug
2011 07:44:07 +0100, Miles Osborne  wrote:  

for the SRILM, you use the
-unk flag; RandLM does this by default if I recall 

Miles

On 16 August
2011 06:28, Tom Hoar  wrote:

Ken, 

Does the online moses documentation
refer to how to ensure the language model has in the vocabulary? I've
never seen it. 

What's the best way to ensure a LM has the token in the
vocabulary? Is it as simple as appending one line consisting of one
token to the language model corpus? Or, is there command line switch for
ngram-count, build-lm.sh, buildlm? Or, should we just edit the raw text
language model and add it to the vocabulary manually? 

Thanks,
Tom 

On
Mon, 15 Aug 2011 22:12:36 +0100, Kenneth Heafield  wrote: Ok I have
reproduced the problem. It only happens when the ARPA file is missing
and is probably an off-by-one on vocabulary size. I'll have a fix soon.


 Kenneth

 On 08/15/11 19:20, Kenneth Heafield wrote: Hi,

 Back from
vacation and sorry but I'm having trouble reproducing this locally. 

 -
Latest Moses (revision 4143); I haven't made any changes that should
impact language modeling since 4096.
 - svn status says the relevant
source code is unmodified.
 - Tried an SRI model, including rebuilding
with build_binary that ships with Moses. 
 - Ran threaded and not
threaded. 

 Can you send me your very small SRILM model? Does it have ?


 Kenneth

 On 08/04/11 11:42, Kenneth Heafield wrote: Sorry I am slow
to respond. This is my first thing to look at, but I am traveling a lot
through the 14th. 

Alex Fraser  [3] wrote:  

Hi Kenneth --

Latest
revision, 4096. Single threaded also crashes.

Cheers, Alex

On Fri, Jul
29, 2011 at 6:00 PM, Kenneth Heafield  [4] wrote:

> Hi,
>
> There was a
problem with this; thought it was fixed but maybe it came
> back. Which
revision are you running? Does it still happen if you run
>
single-threaded?
>
> Kenneth

>
> On 07/29/11 09:39, Alex Fraser
wrote:
>> Hi Folks,
>>
>> Tom Hoar previously mentioned that he had a
problem with KenLMs built
>> from SRILM crashing Moses.
>>

>> Fabienne
Cap and I also have had a problem with this. It seems to be
>>
restricted to using the trie option with build-binary.
>>
>> Ken, if you
have any problems repr!
 oducing

this, please let me know. I

>> can
send you a very small SRILM trained language model that crashes
>> moses
when converted to binary with the trie option, but works fine as
>> a
probing binary and using the original ARPA. (BTW, this is running

>>
the decoder multi-threaded and the crash comes at some point during
>>
decoding the first sentence, not during loading files)
>>
>> Cheers,
Alex
>>
-

>> Moses-support mailing list

>>
Moses-support@mit.edu [5]
>>
http://mailman.mit.edu/mailman/listinfo/moses-support
[6]

>
>
-

> Moses-support mailing list
>
Moses-support@mit.edu [7]
>
http://mailman.mit.edu/mailman/listinfo/moses-support [8]

>

   
___ Moses-support mailing
list Moses-support@mit.edu [9]
http://mailman.mit.edu/mailman/listinfo/moses-support [10] 

___ Moses-support mailing
list Moses-support@mit.edu [11]
http://mailman.mit.edu/mailman/listinfo/moses-support [12]


___
 Moses-support mailing
list
Moses-support@mit.edu
[13]
http://mailman.mit.edu/mailman/listinfo/moses-support [14]

 --

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336. 

Links:
--
[1]
mailto:tah...@precisiontranslationtools.com
[2]
mailto:mo...@kheafield.com
[3] mailto:alexfra...@gmail.com
[4]
mailto:mo...@kheafield.com
[5] mailto:Moses-support@mit.edu
[6]
http://mailman.mit.edu/mailman/listinfo/moses-support
[7]
mailto:Moses-support@mit.edu
[8]
http://mailman.mit.edu/mailman/listinfo/moses-support
[9]
mailto:Moses-support@mit.edu
[10]
http://mailman.mit.edu/mailman/listinfo/moses-support
[11]
mailto:Moses-support@mit.edu
[12]
http://mailman.mit.edu/mailman/listinfo/moses-support
[13]
mailto:Moses-support@mit.edu
[14]
http://mailman.mit.edu/mailman/listinfo/moses-support
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support