Re: [Moses-support] Character ngrams using KenLM

[Nat Gillin]

Dear Kenneth and Moses community,

@Kenneth, Thank you for the tip!

Regards,
Nat

On Wed, Nov 9, 2016 at 4:46 PM, Kenneth Heafield 
wrote:

> No. Tokenizer and LM are separate tools. You can of course replace space
> with a token like  or something.
>
> On November 9, 2016 6:04:07 AM GMT+00:00, Nat Gillin 
> wrote:
>
>> Dear Moses community,
>>
>> Other than manually replacing space with an unused character and adding
>> spaces to each character before training a language model with KenLM. Is it
>> possible for KenLM to generate character ngrams and output in arpa format
>> without altering the input file?
>>
>> Regards,
>> Nat
>>
>> --
>>
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Character ngrams using KenLM

[Kenneth Heafield]

No.  Tokenizer and LM are separate tools.  You can of course replace space with 
a token like  or something.  

On November 9, 2016 6:04:07 AM GMT+00:00, Nat Gillin  
wrote:
>Dear Moses community,
>
>Other than manually replacing space with an unused character and adding
>spaces to each character before training a language model with KenLM.
>Is it
>possible for KenLM to generate character ngrams and output in arpa
>format
>without altering the input file?
>
>Regards,
>Nat
>
>
>
>
>___
>Moses-support mailing list
>Moses-support@mit.edu
>http://mailman.mit.edu/mailman/listinfo/moses-support
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support