[Moses-support] error while running mosesserver

[Santanu Pal]

I ran the following comment

mosesserver -f ./resource/MT/EN-DE/Baseline/moses.ini
--server-port 9081 --server-log ./moses-log.txt

error:


FeatureFunction: UnknownWordPenalty0 start: 0 end: 0

libc++abi.dylib: terminating with uncaught exception of type
util::Exception: ./moses/ScoreComponentCollection.h:108 in static IndexPair
Moses::ScoreComponentCollection::GetIndexes(const Moses::FeatureFunction *)
threw util::Exception'.

ERROR: FeatureFunction: (?_?


please find the attached moses.ini

This ini file running well in my mac,

moses -f ./resource/MT/EN-DE/Baseline/moses.ini


best regards,

Santanu


moses.ini
Description: Binary data
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] changing the jamfile for integrating LM

[koormoosh]

Hi Hieu,

I've created MYLM.cpp, MYLM.h and defined them in FF/Factory.cpp. It turns
out that I also need to change the jamfile but it is not clear how to do
it. I looked at others and tried to add something similar but the "lib"
part keeps fails the build process. Here is what I added to the jamfile:

#MYLM
local with-mylm = [ option.get "with-mylm" ] ;
if $(with-mylm) {
  lib mylm-lib : : $(with-mylm)/external/toolkit/lib ;
  obj MYLM.o : MYLM.cpp ..//headers : $(with-mylm)/include ;
  alias mylm : MYLM.o mylm-lib : : : LM_MY ;
  dependencies += mylm ;
  lmmacros += LM_MY ;
}

*After calling:*

./bjam --with-boost=/LM/mosesdecoder/boost_1_55_0 -j8
--with-mylm=/MYLANGMOD/ toolset=gcc cxxflags="-std=c++11"

*I get this error:*
gcc.link moses-cmd/bin/gcc-4.9.2/release/link-static/threading-multi/moses
/usr/bin/ld: cannot find -lmylm-lib
collect2: error: ld returned 1 exit status

I cannot interpret the error as any solutions I tried have failed. Any
comments?
Thanks!
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Moses-support Digest, Vol 109, Issue 19

[Barry Haddow]

Hi Tomasz

The moseserver is just the decoder, so it doesn't do any of the pre- and 
post-processing steps that you also need. In particular it does not do 
tokenisation. You need to send it tokenised text, and then de-tokenise 
the output,

cheers - Barry

On 12/11/15 13:40, Tomasz Gawryl wrote:
> Hi Ulrich,
>
> I have a question about Moses server too. I'm testing it as a wrapper for
> Across server to check pre-translation possibilities. It generally works but
> there is one problem. Input segments are translated without tokenization, so
> every word close to special character (for example `this is small house.`)
> remains untranslated ('to jest mały house.'). I was searching list archive
> and I found similar question here:
> http://comments.gmane.org/gmane.comp.nlp.moses.user/14020  but for me it's
> not yet answered. I would appreciate any information on this subject.
>
> Best regards,
> Tomek
>
> -Original Message-
> From: moses-support-boun...@mit.edu [mailto:moses-support-boun...@mit.edu]
> On Behalf Of moses-support-requ...@mit.edu
> Sent: Wednesday, November 11, 2015 10:31 AM
> To: moses-support@mit.edu
> Subject: Moses-support Digest, Vol 109, Issue 19
>
> Send Moses-support mailing list submissions to
>   moses-support@mit.edu
>
> To subscribe or unsubscribe via the World Wide Web, visit
>   http://mailman.mit.edu/mailman/listinfo/moses-support
> or, via email, send a message with subject or body 'help' to
>   moses-support-requ...@mit.edu
>
> You can reach the person managing the list at
>   moses-support-ow...@mit.edu
>
> When replying, please edit your Subject line so it is more specific than
> "Re: Contents of Moses-support digest..."
>
>
> Today's Topics:
>
> 1. Re: use placeholder with mosesserver (Ulrich Germann)
> 2. Re: use placeholder with mosesserver (Evgeny Matusov)
>
>
> --
>
> Message: 1
> Date: Wed, 11 Nov 2015 01:58:40 +
> From: Ulrich Germann 
> Subject: Re: [Moses-support] use placeholder with mosesserver
> To: Evgeny Matusov 
> Cc: "moses-support@mit.edu" 
> Message-ID:
>   
> Content-Type: text/plain; charset="utf-8"
>
> Hi all,
>
> I've just pushed what I believe might address a few of the issues in this
> thread:
>
> - the more fine-grained configuration options for request handling and
> queuing, server timeouts etc. (added in August due to threading issue) have
> been transferred to the main moses executable.
>
> - the server now pays attention to the xml-input option specified via json;
> the range of accepted values is the same as when specified on the command
> line. I have not written the xml-input handling and do not actively use it,
> so it may or may not work. I don't think there are any regression tests that
> test this right now. Reports from the trenches are welcome.
>
> - mosesserver.cpp is deprecated. It is now merely a shell around the regular
> moses call with --server. I did not remove it from the code base entirely,
> as I assume that there's a plethora of setups out there that rely on the
> existence of mosesserver. What the wrapper does is add --server to the
> options and then call run regular moses.
>
> - anyone adding stuff to mosesserver.cpp from now on owes me a lifetime
> supply of the finest Laphroaig. Just send me a quarter cask every year for
> Burns Nicht for the rest of my life if you do. If I haven't pushed anything
> for two years, you may assume I'm dead.
>
>
> - Uli
>
> On Tue, Nov 10, 2015 at 2:58 PM, Ulrich Germann 
> wrote:
>
>> Hi all,
>>
>> mosesserver is deprecated and should not be used any more. I'll
>> transfer the threading-related changes to the server implementation in
>> the regular moses executable and let you know once I'm done so that
>> other things (like
>> passthrough) can be added. By the looks of it, the changes are fairly
>> straightforward, so it shouldn't take long. However, I can't guarantee
>> that the new server will do everything the old server did, (or do it
>> the same way).
>>
>> It would be fantastic if a few people could design and contribute test
>> cases so that we can do some regression testing for the server.
>> Ideally a test case should provide:
>>
>> - tiny models to work with (or we may be able to recycle some that
>> already
>> exist)
>> - sample input (json)
>> - expected output (json)
>>
>> Cheers - Uli
>>
>> On Tue, Nov 10, 2015 at 11:37 AM, Evgeny Matusov 
>> wrote:
>>
>>> Hi,
>>>
>>> can any of the more active recent developers advise what is the
>>> latest stable mosesserver implementation?
>>>
>>> It seems to be the one in moses/server, but the  one in in
>>> contrib/server/mosesserver.cpp has been updated in August of this
>>> year with an important fix related to multiple threads:
>>>
>>>
>>> 

Re: [Moses-support] Moses-support Digest, Vol 109, Issue 19

[Tomasz Gawryl]

Hi Ulrich,

I have a question about Moses server too. I'm testing it as a wrapper for
Across server to check pre-translation possibilities. It generally works but
there is one problem. Input segments are translated without tokenization, so
every word close to special character (for example `this is small house.`)
remains untranslated ('to jest mały house.'). I was searching list archive
and I found similar question here:
http://comments.gmane.org/gmane.comp.nlp.moses.user/14020  but for me it's
not yet answered. I would appreciate any information on this subject.

Best regards,
Tomek

-Original Message-
From: moses-support-boun...@mit.edu [mailto:moses-support-boun...@mit.edu]
On Behalf Of moses-support-requ...@mit.edu
Sent: Wednesday, November 11, 2015 10:31 AM
To: moses-support@mit.edu
Subject: Moses-support Digest, Vol 109, Issue 19

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-requ...@mit.edu

You can reach the person managing the list at
moses-support-ow...@mit.edu

When replying, please edit your Subject line so it is more specific than
"Re: Contents of Moses-support digest..."


Today's Topics:

   1. Re: use placeholder with mosesserver (Ulrich Germann)
   2. Re: use placeholder with mosesserver (Evgeny Matusov)


--

Message: 1
Date: Wed, 11 Nov 2015 01:58:40 +
From: Ulrich Germann 
Subject: Re: [Moses-support] use placeholder with mosesserver
To: Evgeny Matusov 
Cc: "moses-support@mit.edu" 
Message-ID:

Content-Type: text/plain; charset="utf-8"

Hi all,

I've just pushed what I believe might address a few of the issues in this
thread:

- the more fine-grained configuration options for request handling and
queuing, server timeouts etc. (added in August due to threading issue) have
been transferred to the main moses executable.

- the server now pays attention to the xml-input option specified via json;
the range of accepted values is the same as when specified on the command
line. I have not written the xml-input handling and do not actively use it,
so it may or may not work. I don't think there are any regression tests that
test this right now. Reports from the trenches are welcome.

- mosesserver.cpp is deprecated. It is now merely a shell around the regular
moses call with --server. I did not remove it from the code base entirely,
as I assume that there's a plethora of setups out there that rely on the
existence of mosesserver. What the wrapper does is add --server to the
options and then call run regular moses.

- anyone adding stuff to mosesserver.cpp from now on owes me a lifetime
supply of the finest Laphroaig. Just send me a quarter cask every year for
Burns Nicht for the rest of my life if you do. If I haven't pushed anything
for two years, you may assume I'm dead.


- Uli

On Tue, Nov 10, 2015 at 2:58 PM, Ulrich Germann 
wrote:

> Hi all,
>
> mosesserver is deprecated and should not be used any more. I'll 
> transfer the threading-related changes to the server implementation in 
> the regular moses executable and let you know once I'm done so that 
> other things (like
> passthrough) can be added. By the looks of it, the changes are fairly 
> straightforward, so it shouldn't take long. However, I can't guarantee 
> that the new server will do everything the old server did, (or do it 
> the same way).
>
> It would be fantastic if a few people could design and contribute test 
> cases so that we can do some regression testing for the server. 
> Ideally a test case should provide:
>
> - tiny models to work with (or we may be able to recycle some that 
> already
> exist)
> - sample input (json)
> - expected output (json)
>
> Cheers - Uli
>
> On Tue, Nov 10, 2015 at 11:37 AM, Evgeny Matusov 
> wrote:
>
>>
>> Hi,
>>
>> can any of the more active recent developers advise what is the 
>> latest stable mosesserver implementation?
>>
>> It seems to be the one in moses/server, but the  one in in 
>> contrib/server/mosesserver.cpp has been updated in August of this 
>> year with an important fix related to multiple threads:
>>
>>
>> https://github.com/moses-smt/mosesdecoder/commit/3c682fa8b05af6bff1a0
>> 9f420141795875cf9685 
>> https://www.mail-archive.com/moses-support%40mit.edu/msg12875.html
>>
>> As Gregor mentioned, we would like to share our fix so that 
>> Mosesserver correctly supports placeholders. I want to make sure that 
>> this is a fix for something that many people use without problems.
>>
>> Thanks,
>> Evgeny.
>>
>>
>> 
>> From: 

Re: [Moses-support] Moses-support Digest, Vol 109, Issue 19

[Panos Kanavos]

Hi Barry,

Have there ever been any thoughts about implementing 
tokenization/detokenization directly in Moses? I suppose this is some 
work as Moses should become language-aware, but I can only see 
advantanges from this. Besides, Moses is a language tool so these 
concepts shouldn't be so disconnected from its core logic.

Best,

Panos


On 12/11/2015 4:02 μμ, Barry Haddow wrote:
> Hi Tomasz
>
> The moseserver is just the decoder, so it doesn't do any of the pre- and
> post-processing steps that you also need. In particular it does not do
> tokenisation. You need to send it tokenised text, and then de-tokenise
> the output,
>
> cheers - Barry
>
> On 12/11/15 13:40, Tomasz Gawryl wrote:
>> Hi Ulrich,
>>
>> I have a question about Moses server too. I'm testing it as a wrapper for
>> Across server to check pre-translation possibilities. It generally works but
>> there is one problem. Input segments are translated without tokenization, so
>> every word close to special character (for example `this is small house.`)
>> remains untranslated ('to jest mały house.'). I was searching list archive
>> and I found similar question here:
>> http://comments.gmane.org/gmane.comp.nlp.moses.user/14020  but for me it's
>> not yet answered. I would appreciate any information on this subject.
>>
>> Best regards,
>> Tomek
>>
>> -Original Message-
>> From: moses-support-boun...@mit.edu [mailto:moses-support-boun...@mit.edu]
>> On Behalf Of moses-support-requ...@mit.edu
>> Sent: Wednesday, November 11, 2015 10:31 AM
>> To: moses-support@mit.edu
>> Subject: Moses-support Digest, Vol 109, Issue 19
>>
>> Send Moses-support mailing list submissions to
>>  moses-support@mit.edu
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>  http://mailman.mit.edu/mailman/listinfo/moses-support
>> or, via email, send a message with subject or body 'help' to
>>  moses-support-requ...@mit.edu
>>
>> You can reach the person managing the list at
>>  moses-support-ow...@mit.edu
>>
>> When replying, please edit your Subject line so it is more specific than
>> "Re: Contents of Moses-support digest..."
>>
>>
>> Today's Topics:
>>
>>  1. Re: use placeholder with mosesserver (Ulrich Germann)
>>  2. Re: use placeholder with mosesserver (Evgeny Matusov)
>>
>>
>> --
>>
>> Message: 1
>> Date: Wed, 11 Nov 2015 01:58:40 +
>> From: Ulrich Germann 
>> Subject: Re: [Moses-support] use placeholder with mosesserver
>> To: Evgeny Matusov 
>> Cc: "moses-support@mit.edu" 
>> Message-ID:
>>  
>> Content-Type: text/plain; charset="utf-8"
>>
>> Hi all,
>>
>> I've just pushed what I believe might address a few of the issues in this
>> thread:
>>
>> - the more fine-grained configuration options for request handling and
>> queuing, server timeouts etc. (added in August due to threading issue) have
>> been transferred to the main moses executable.
>>
>> - the server now pays attention to the xml-input option specified via json;
>> the range of accepted values is the same as when specified on the command
>> line. I have not written the xml-input handling and do not actively use it,
>> so it may or may not work. I don't think there are any regression tests that
>> test this right now. Reports from the trenches are welcome.
>>
>> - mosesserver.cpp is deprecated. It is now merely a shell around the regular
>> moses call with --server. I did not remove it from the code base entirely,
>> as I assume that there's a plethora of setups out there that rely on the
>> existence of mosesserver. What the wrapper does is add --server to the
>> options and then call run regular moses.
>>
>> - anyone adding stuff to mosesserver.cpp from now on owes me a lifetime
>> supply of the finest Laphroaig. Just send me a quarter cask every year for
>> Burns Nicht for the rest of my life if you do. If I haven't pushed anything
>> for two years, you may assume I'm dead.
>>
>>
>> - Uli
>>
>> On Tue, Nov 10, 2015 at 2:58 PM, Ulrich Germann 
>> wrote:
>>
>>> Hi all,
>>>
>>> mosesserver is deprecated and should not be used any more. I'll
>>> transfer the threading-related changes to the server implementation in
>>> the regular moses executable and let you know once I'm done so that
>>> other things (like
>>> passthrough) can be added. By the looks of it, the changes are fairly
>>> straightforward, so it shouldn't take long. However, I can't guarantee
>>> that the new server will do everything the old server did, (or do it
>>> the same way).
>>>
>>> It would be fantastic if a few people could design and contribute test
>>> cases so that we can do some regression testing for the server.
>>> Ideally a test case should provide:
>>>
>>> - tiny models to work with (or we may be able to recycle some that
>>> already
>>> exist)
>>> 

Re: [Moses-support] Moses-support Digest, Vol 109, Issue 19

[Hieu Hoang]

there has been thoughts. There is a c++ tokenizer in
   contrib/c++tokenizer
it compiles into a library file, ready for integration.

The last time i checked, it gave a slightly worse BLEU. Not much, but
consistent.

If anyone wants to carry on with it, they're welcome to

Hieu Hoang
http://www.hoang.co.uk/hieu

On 12 November 2015 at 14:37, Panos Kanavos  wrote:

> Hi Barry,
>
> Have there ever been any thoughts about implementing
> tokenization/detokenization directly in Moses? I suppose this is some
> work as Moses should become language-aware, but I can only see
> advantanges from this. Besides, Moses is a language tool so these
> concepts shouldn't be so disconnected from its core logic.
>
> Best,
>
> Panos
>
>
> On 12/11/2015 4:02 μμ, Barry Haddow wrote:
> > Hi Tomasz
> >
> > The moseserver is just the decoder, so it doesn't do any of the pre- and
> > post-processing steps that you also need. In particular it does not do
> > tokenisation. You need to send it tokenised text, and then de-tokenise
> > the output,
> >
> > cheers - Barry
> >
> > On 12/11/15 13:40, Tomasz Gawryl wrote:
> >> Hi Ulrich,
> >>
> >> I have a question about Moses server too. I'm testing it as a wrapper
> for
> >> Across server to check pre-translation possibilities. It generally
> works but
> >> there is one problem. Input segments are translated without
> tokenization, so
> >> every word close to special character (for example `this is small
> house.`)
> >> remains untranslated ('to jest mały house.'). I was searching list
> archive
> >> and I found similar question here:
> >> http://comments.gmane.org/gmane.comp.nlp.moses.user/14020  but for me
> it's
> >> not yet answered. I would appreciate any information on this subject.
> >>
> >> Best regards,
> >> Tomek
> >>
> >> -Original Message-
> >> From: moses-support-boun...@mit.edu [mailto:
> moses-support-boun...@mit.edu]
> >> On Behalf Of moses-support-requ...@mit.edu
> >> Sent: Wednesday, November 11, 2015 10:31 AM
> >> To: moses-support@mit.edu
> >> Subject: Moses-support Digest, Vol 109, Issue 19
> >>
> >> Send Moses-support mailing list submissions to
> >>  moses-support@mit.edu
> >>
> >> To subscribe or unsubscribe via the World Wide Web, visit
> >>  http://mailman.mit.edu/mailman/listinfo/moses-support
> >> or, via email, send a message with subject or body 'help' to
> >>  moses-support-requ...@mit.edu
> >>
> >> You can reach the person managing the list at
> >>  moses-support-ow...@mit.edu
> >>
> >> When replying, please edit your Subject line so it is more specific than
> >> "Re: Contents of Moses-support digest..."
> >>
> >>
> >> Today's Topics:
> >>
> >>  1. Re: use placeholder with mosesserver (Ulrich Germann)
> >>  2. Re: use placeholder with mosesserver (Evgeny Matusov)
> >>
> >>
> >> --
> >>
> >> Message: 1
> >> Date: Wed, 11 Nov 2015 01:58:40 +
> >> From: Ulrich Germann 
> >> Subject: Re: [Moses-support] use placeholder with mosesserver
> >> To: Evgeny Matusov 
> >> Cc: "moses-support@mit.edu" 
> >> Message-ID:
> >>  <
> cahqsruojzbwqie_+1-si7i7e1cmyt3tq4gcfl-2ge2bggd6...@mail.gmail.com>
> >> Content-Type: text/plain; charset="utf-8"
> >>
> >> Hi all,
> >>
> >> I've just pushed what I believe might address a few of the issues in
> this
> >> thread:
> >>
> >> - the more fine-grained configuration options for request handling and
> >> queuing, server timeouts etc. (added in August due to threading issue)
> have
> >> been transferred to the main moses executable.
> >>
> >> - the server now pays attention to the xml-input option specified via
> json;
> >> the range of accepted values is the same as when specified on the
> command
> >> line. I have not written the xml-input handling and do not actively use
> it,
> >> so it may or may not work. I don't think there are any regression tests
> that
> >> test this right now. Reports from the trenches are welcome.
> >>
> >> - mosesserver.cpp is deprecated. It is now merely a shell around the
> regular
> >> moses call with --server. I did not remove it from the code base
> entirely,
> >> as I assume that there's a plethora of setups out there that rely on the
> >> existence of mosesserver. What the wrapper does is add --server to the
> >> options and then call run regular moses.
> >>
> >> - anyone adding stuff to mosesserver.cpp from now on owes me a lifetime
> >> supply of the finest Laphroaig. Just send me a quarter cask every year
> for
> >> Burns Nicht for the rest of my life if you do. If I haven't pushed
> anything
> >> for two years, you may assume I'm dead.
> >>
> >>
> >> - Uli
> >>
> >> On Tue, Nov 10, 2015 at 2:58 PM, Ulrich Germann <
> ulrich.germ...@gmail.com>
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> mosesserver is deprecated and should not be used any more. I'll
> >>> transfer the threading-related changes to 

Re: [Moses-support] Moses-support Digest, Vol 109, Issue 19

[Panos Kanavos]

Thanks for the info Hieu, didn't know that:) I'll try it sometime.

Best,

Panos

On 12/11/2015 4:41 μμ, Hieu Hoang wrote:

there has been thoughts. There is a c++ tokenizer in
   contrib/c++tokenizer
it compiles into a library file, ready for integration.

The last time i checked, it gave a slightly worse BLEU. Not much, but 
consistent.


If anyone wants to carry on with it, they're welcome to

Hieu Hoang
http://www.hoang.co.uk/hieu

On 12 November 2015 at 14:37, Panos Kanavos > wrote:


Hi Barry,

Have there ever been any thoughts about implementing
tokenization/detokenization directly in Moses? I suppose this is some
work as Moses should become language-aware, but I can only see
advantanges from this. Besides, Moses is a language tool so these
concepts shouldn't be so disconnected from its core logic.

Best,

Panos


On 12/11/2015 4:02 μμ, Barry Haddow wrote:
> Hi Tomasz
>
> The moseserver is just the decoder, so it doesn't do any of the
pre- and
> post-processing steps that you also need. In particular it does
not do
> tokenisation. You need to send it tokenised text, and then
de-tokenise
> the output,
>
> cheers - Barry
>
> On 12/11/15 13:40, Tomasz Gawryl wrote:
>> Hi Ulrich,
>>
>> I have a question about Moses server too. I'm testing it as a
wrapper for
>> Across server to check pre-translation possibilities. It
generally works but
>> there is one problem. Input segments are translated without
tokenization, so
>> every word close to special character (for example `this is
small house.`)
>> remains untranslated ('to jest mały house.'). I was searching
list archive
>> and I found similar question here:
>> http://comments.gmane.org/gmane.comp.nlp.moses.user/14020 but
for me it's
>> not yet answered. I would appreciate any information on this
subject.
>>
>> Best regards,
>> Tomek
>>
>> -Original Message-
>> From: moses-support-boun...@mit.edu

[mailto:moses-support-boun...@mit.edu
]
>> On Behalf Of moses-support-requ...@mit.edu

>> Sent: Wednesday, November 11, 2015 10:31 AM
>> To: moses-support@mit.edu 
>> Subject: Moses-support Digest, Vol 109, Issue 19
>>
>> Send Moses-support mailing list submissions to
>> moses-support@mit.edu 
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>> or, via email, send a message with subject or body 'help' to
>> moses-support-requ...@mit.edu

>>
>> You can reach the person managing the list at
>> moses-support-ow...@mit.edu 
>>
>> When replying, please edit your Subject line so it is more
specific than
>> "Re: Contents of Moses-support digest..."
>>
>>
>> Today's Topics:
>>
>>  1. Re: use placeholder with mosesserver (Ulrich Germann)
>>  2. Re: use placeholder with mosesserver (Evgeny Matusov)
>>
>>
>>
--
>>
>> Message: 1
>> Date: Wed, 11 Nov 2015 01:58:40 +
>> From: Ulrich Germann >
>> Subject: Re: [Moses-support] use placeholder with mosesserver
>> To: Evgeny Matusov >
>> Cc: "moses-support@mit.edu "
>
>> Message-ID:
>> 
>

>> Content-Type: text/plain; charset="utf-8"
>>
>> Hi all,
>>
>> I've just pushed what I believe might address a few of the
issues in this
>> thread:
>>
>> - the more fine-grained configuration options for request
handling and
>> queuing, server timeouts etc. (added in August due to threading
issue) have
>> been transferred to the main moses executable.
>>
>> - the server now pays attention to the xml-input option
specified via json;
>> the range of accepted values is the same as when specified on
the command
>> line. I have not written the xml-input handling and do not
actively use it,
>> so it may or may not work. I don't think there are any
regression tests that
>> test this right now. Reports from the trenches are welcome.
>>
>> - mosesserver.cpp is deprecated. It is now merely a shell

Re: [Moses-support] Moses-support Digest, Vol 109, Issue 19

[Panos Kanavos]

Hi Dingyuan,

I was actually thinking about implementing the logic and rules from the 
perl scripts, which seem to do the job, into a separate library (that 
thankfully exists as Heiu informed:)).
Asking an end-user to add himself a programming layer in a seemingly 
straightforward process is rather ambitious. What I'm saying is that, 
despite the effort needed, I think it's worth it since it will help 
enormously Moses' deployment by mere mortals and will attract more 
end-users :)

Best,

Panos

On 12/11/2015 4:42 μμ, Dingyuan Wang wrote:
> Hi,
>
> Tokenization and detokenization is heavily dependent on the source and
> target language and the application case. Some languages need complex
> algorithms and/or large dictionaries just to tokenize. Therefore, leave
> the task for the users is better, I think.
>
> Regards,
> Dingyuan
>
> 在 2015年11月12日 22:37, Panos Kanavos 写道:
>> Hi Barry,
>>
>> Have there ever been any thoughts about implementing
>> tokenization/detokenization directly in Moses? I suppose this is some
>> work as Moses should become language-aware, but I can only see
>> advantanges from this. Besides, Moses is a language tool so these
>> concepts shouldn't be so disconnected from its core logic.
>>
>> Best,
>>
>> Panos
>>
>>
>> On 12/11/2015 4:02 μμ, Barry Haddow wrote:
>>> Hi Tomasz
>>>
>>> The moseserver is just the decoder, so it doesn't do any of the pre- and
>>> post-processing steps that you also need. In particular it does not do
>>> tokenisation. You need to send it tokenised text, and then de-tokenise
>>> the output,
>>>
>>> cheers - Barry
>>>
>>> On 12/11/15 13:40, Tomasz Gawryl wrote:
 Hi Ulrich,

 I have a question about Moses server too. I'm testing it as a wrapper for
 Across server to check pre-translation possibilities. It generally works 
 but
 there is one problem. Input segments are translated without tokenization, 
 so
 every word close to special character (for example `this is small house.`)
 remains untranslated ('to jest mały house.'). I was searching list archive
 and I found similar question here:
 http://comments.gmane.org/gmane.comp.nlp.moses.user/14020  but for me it's
 not yet answered. I would appreciate any information on this subject.

 Best regards,
 Tomek

 -Original Message-
 From: moses-support-boun...@mit.edu [mailto:moses-support-boun...@mit.edu]
 On Behalf Of moses-support-requ...@mit.edu
 Sent: Wednesday, November 11, 2015 10:31 AM
 To: moses-support@mit.edu
 Subject: Moses-support Digest, Vol 109, Issue 19

 Send Moses-support mailing list submissions to
moses-support@mit.edu

 To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
 or, via email, send a message with subject or body 'help' to
moses-support-requ...@mit.edu

 You can reach the person managing the list at
moses-support-ow...@mit.edu

 When replying, please edit your Subject line so it is more specific than
 "Re: Contents of Moses-support digest..."


 Today's Topics:

   1. Re: use placeholder with mosesserver (Ulrich Germann)
   2. Re: use placeholder with mosesserver (Evgeny Matusov)


 --

 Message: 1
 Date: Wed, 11 Nov 2015 01:58:40 +
 From: Ulrich Germann 
 Subject: Re: [Moses-support] use placeholder with mosesserver
 To: Evgeny Matusov 
 Cc: "moses-support@mit.edu" 
 Message-ID:

 Content-Type: text/plain; charset="utf-8"

 Hi all,

 I've just pushed what I believe might address a few of the issues in this
 thread:

 - the more fine-grained configuration options for request handling and
 queuing, server timeouts etc. (added in August due to threading issue) have
 been transferred to the main moses executable.

 - the server now pays attention to the xml-input option specified via json;
 the range of accepted values is the same as when specified on the command
 line. I have not written the xml-input handling and do not actively use it,
 so it may or may not work. I don't think there are any regression tests 
 that
 test this right now. Reports from the trenches are welcome.

 - mosesserver.cpp is deprecated. It is now merely a shell around the 
 regular
 moses call with --server. I did not remove it from the code base entirely,
 as I assume that there's a plethora of setups out there that rely on the
 existence of mosesserver. What the wrapper does is add --server to the
 options and then call run regular moses.

 - anyone adding stuff to mosesserver.cpp from now on owes 

Re: [Moses-support] Moses-support Digest, Vol 109, Issue 19

[Philipp Koehn]

Hi,

there are a lot of different pre and post processing
steps that you may want to apply for any given
language pair, so it makes sense to keep them
out of the decoder.

If you are interested in a server implementation
that integrates tokenization, truecasing, etc., check
out Christian Buck's implementation of the Google
Translate API for Moses:

https://github.com/casmacat/moses-mt-server/tree/master/python_server

-phi

On Thu, Nov 12, 2015 at 9:52 AM, Panos Kanavos 
wrote:

> Thanks for the info Hieu, didn't know that:) I'll try it sometime.
>
> Best,
>
> Panos
>
>
> On 12/11/2015 4:41 μμ, Hieu Hoang wrote:
>
> there has been thoughts. There is a c++ tokenizer in
>contrib/c++tokenizer
> it compiles into a library file, ready for integration.
>
> The last time i checked, it gave a slightly worse BLEU. Not much, but
> consistent.
>
> If anyone wants to carry on with it, they're welcome to
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 12 November 2015 at 14:37, Panos Kanavos 
> wrote:
>
>> Hi Barry,
>>
>> Have there ever been any thoughts about implementing
>> tokenization/detokenization directly in Moses? I suppose this is some
>> work as Moses should become language-aware, but I can only see
>> advantanges from this. Besides, Moses is a language tool so these
>> concepts shouldn't be so disconnected from its core logic.
>>
>> Best,
>>
>> Panos
>>
>>
>> On 12/11/2015 4:02 μμ, Barry Haddow wrote:
>> > Hi Tomasz
>> >
>> > The moseserver is just the decoder, so it doesn't do any of the pre- and
>> > post-processing steps that you also need. In particular it does not do
>> > tokenisation. You need to send it tokenised text, and then de-tokenise
>> > the output,
>> >
>> > cheers - Barry
>> >
>> > On 12/11/15 13:40, Tomasz Gawryl wrote:
>> >> Hi Ulrich,
>> >>
>> >> I have a question about Moses server too. I'm testing it as a wrapper
>> for
>> >> Across server to check pre-translation possibilities. It generally
>> works but
>> >> there is one problem. Input segments are translated without
>> tokenization, so
>> >> every word close to special character (for example `this is small
>> house.`)
>> >> remains untranslated ('to jest mały house.'). I was searching list
>> archive
>> >> and I found similar question here:
>> >> http://comments.gmane.org/gmane.comp.nlp.moses.user/14020  but for me
>> it's
>> >> not yet answered. I would appreciate any information on this subject.
>> >>
>> >> Best regards,
>> >> Tomek
>> >>
>> >> -Original Message-
>> >> From: moses-support-boun...@mit.edu [mailto:
>> moses-support-boun...@mit.edu]
>> >> On Behalf Of moses-support-requ...@mit.edu
>> >> Sent: Wednesday, November 11, 2015 10:31 AM
>> >> To: moses-support@mit.edu
>> >> Subject: Moses-support Digest, Vol 109, Issue 19
>> >>
>> >> Send Moses-support mailing list submissions to
>> >>  moses-support@mit.edu
>> >>
>> >> To subscribe or unsubscribe via the World Wide Web, visit
>> >>  http://mailman.mit.edu/mailman/listinfo/moses-support
>> >> or, via email, send a message with subject or body 'help' to
>> >>  moses-support-requ...@mit.edu
>> >>
>> >> You can reach the person managing the list at
>> >>  moses-support-ow...@mit.edu
>> >>
>> >> When replying, please edit your Subject line so it is more specific
>> than
>> >> "Re: Contents of Moses-support digest..."
>> >>
>> >>
>> >> Today's Topics:
>> >>
>> >>  1. Re: use placeholder with mosesserver (Ulrich Germann)
>> >>  2. Re: use placeholder with mosesserver (Evgeny Matusov)
>> >>
>> >>
>> >> --
>> >>
>> >> Message: 1
>> >> Date: Wed, 11 Nov 2015 01:58:40 +
>> >> From: Ulrich Germann < 
>> ulrich.germ...@gmail.com>
>> >> Subject: Re: [Moses-support] use placeholder with mosesserver
>> >> To: Evgeny Matusov < ematu...@apptek.com>
>> >> Cc: "moses-support@mit.edu" 
>> >> Message-ID:
>> >>  <
>> cahqsruojzbwqie_+1-si7i7e1cmyt3tq4gcfl-2ge2bggd6...@mail.gmail.com>
>> >> Content-Type: text/plain; charset="utf-8"
>> >>
>> >> Hi all,
>> >>
>> >> I've just pushed what I believe might address a few of the issues in
>> this
>> >> thread:
>> >>
>> >> - the more fine-grained configuration options for request handling and
>> >> queuing, server timeouts etc. (added in August due to threading issue)
>> have
>> >> been transferred to the main moses executable.
>> >>
>> >> - the server now pays attention to the xml-input option specified via
>> json;
>> >> the range of accepted values is the same as when specified on the
>> command
>> >> line. I have not written the xml-input handling and do not actively
>> use it,
>> >> so it may or may not work. I don't think there are any regression
>> tests that
>> >> test this right now. Reports from the trenches are welcome.
>> >>
>> >> - mosesserver.cpp is deprecated. It is now merely a shell around the
>> regular
>> 

Re: [Moses-support] changing the jamfile for integrating LM

[Philipp Koehn]

Hi,

do you really have a library associated with your language model?

I would have thought that adding your language model would as easy as
adding it to the line

alias LM : Backward.cpp BackwardLMState.cpp Base.cpp BilingualLM.cpp
Implementation.cpp Ken.cpp MultiFactor.cpp Remote.cpp SingleFactor.cpp Skel
etonLM.cpp

in moses/LM/Jamfile.


-phi



On Thu, Nov 12, 2015 at 5:40 AM, koormoosh  wrote:

> Hi Hieu,
>
> I've created MYLM.cpp, MYLM.h and defined them in FF/Factory.cpp. It turns
> out that I also need to change the jamfile but it is not clear how to do
> it. I looked at others and tried to add something similar but the "lib"
> part keeps fails the build process. Here is what I added to the jamfile:
>
> #MYLM
> local with-mylm = [ option.get "with-mylm" ] ;
> if $(with-mylm) {
>   lib mylm-lib : : $(with-mylm)/external/toolkit/lib ;
>   obj MYLM.o : MYLM.cpp ..//headers : $(with-mylm)/include ;
>   alias mylm : MYLM.o mylm-lib : : : LM_MY ;
>   dependencies += mylm ;
>   lmmacros += LM_MY ;
> }
>
> *After calling:*
>
> ./bjam --with-boost=/LM/mosesdecoder/boost_1_55_0 -j8
> --with-mylm=/MYLANGMOD/ toolset=gcc cxxflags="-std=c++11"
>
> *I get this error:*
> gcc.link moses-cmd/bin/gcc-4.9.2/release/link-static/threading-multi/moses
> /usr/bin/ld: cannot find -lmylm-lib
> collect2: error: ld returned 1 exit status
>
> I cannot interpret the error as any solutions I tried have failed. Any
> comments?
> Thanks!
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support