Re: [Moses-support] Europarl monolingual corpus
ah thanks, just what i was looking for. Hieu Hoang Researcher New York University, Abu Dhabi http://www.hoang.co.uk/hieu On 6 May 2015 at 11:18, joerg tiede...@gmail.com wrote: Go to http://opus.lingfil.uu.se Select the language pair your interested in and click on the language ID's in the column mono (or raw next to it) to download the data you like to use. Europarl is version 7 in the list. You can also take them from here: http://opus.lingfil.uu.se/Europarl/mono/ Best, Jörg ** Jörg Tiedemann http://stp.lingfil.uu.se/~joerg/ On May 6, 2015, at 7:25 AM, Hieu Hoang wrote: ah thx. The tgz file only has data for a subset of the languages. It would be useful to be able to download them all, or at least know how to extract them from the raw data. Hieu Hoang Researcher New York University, Abu Dhabi http://www.hoang.co.uk/hieu On 6 May 2015 at 03:04, Ulrich Germann ulrich.germ...@gmail.com wrote: Extract it from commoncrawl, of course! ;-) ... or get it here: http://www.statmt.org/wmt13/training-monolingual-europarl-v7.tgz - Uli On Mon, May 4, 2015 at 5:46 AM, Hieu Hoang hieuho...@gmail.com wrote: What's the easiest way get the single-language data from the Europarl corpus as described in the 1st table in: http://statmt.org/europarl/ I tried downloading the xml source http://statmt.org/europarl/v7/europarl.tgz stripping the xml and running split-sentence.perl, but this takes an unfathomably long time Hieu Hoang Researcher New York University, Abu Dhabi http://www.hoang.co.uk/hieu ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- Ulrich Germann Senior Researcher School of Informatics University of Edinburgh ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Fwd: Fwd: Server development
Hi Barry, Thanks for explanation – I was referring to “[Moses-support] mosesserver parallelization issuehttps://www.mail-archive.com/search?l=moses-support@mit.eduq=subject:%22%5C%5BMoses%5C-support%5C%5D+mosesserver+parallelization+issue%22o=newest” thread in April 2014. Do you thing the thread pool would be also the solution for this issue reported on 2.1.1 branch.? Thanks, Tomas From: Barry Haddow [mailto:bhad...@inf.ed.ac.uk] Sent: Tuesday, May 5, 2015 9:27 PM To: Hieu Hoang; moses-support Subject: Re: [Moses-support] Fwd: Fwd: Server development HI Tomas There were some issues in v2 with the way that caching was done in the binarised phrase table. It used a cache per thread, and mosesserver used a thread per request, so caching was effectively broken in the server. Since last Autumn, mosesserver uses a thread pool ... and the binarised phrase table is gone now anyway, cheers - Barry On 05/05/15 18:27, Hieu Hoang wrote: What limitations are you referring to? -- Forwarded message -- From: Ulrich Germann ulrich.germ...@gmail.commailto:ulrich.germ...@gmail.com Date: 5 May 2015 19:49 Subject: [Moses-support] Fwd: Server development To: moses-support@mit.edumailto:moses-support@mit.edu moses-support@mit.edumailto:moses-support@mit.edu Cc: This response was meant to go to moses-support as well Tomas. -- Forwarded message -- From: Tomas Fulajtar toma...@moravia.commailto:toma...@moravia.com Date: Fri, Apr 3, 2015 at 5:03 PM Subject: RE: [Moses-support] Server development To: ugerm...@inf.ed.ac.ukmailto:ugerm...@inf.ed.ac.uk ugerm...@inf.ed.ac.ukmailto:ugerm...@inf.ed.ac.uk Hi Ulrich, Thanks for the thorough explanation - the idea of merging the server code back to moses is great. Apart from this (and I know is is a huge workload), were there any changes in the thread support? I know this part had some limitations – as discussed on the forum. Kind regards, Tomas From: Ulrich Germann [mailto:ulrich.germ...@gmail.commailto:ulrich.germ...@gmail.com] Sent: Thursday, April 2, 2015 12:57 AM To: Tomas Fulajtar Subject: Re: [Moses-support] Server development Hi Tomas, the plan is to fold server capabilities into the main moses executable. In fact, that has already happened (in the sense that you can run the main moses executable in server mode), but functional equivalence with the old code has not been tested. There are currently no server tests included in the regression tests, so I left the old code mostly intact (adjusting only for changes in the API of functions called) for legacy reasons, but adding new functionality to mosesserver is extremely strongly DIScouraged. Supplying regression tests for server functionality, on the other hand, is equally strongly ENcouraged. In a nutshell, what you get back from calling mosesserver and moses --server should be identical. The long-term plan is to offer through RPC calls (almost) everything that moses offers in batch mode (i.e., send search and output parameters through json/RCP calls and have them noticed and respected). Notice the long-term there. So mosesserver is on its way out, and moses --server-port=port --server will replace the old call to mosesserver. Best regards - Uli On Wed, Apr 1, 2015 at 9:48 AM, Tomas Fulajtar toma...@moravia.commailto:toma...@moravia.com wrote: Dear all, I have spotted there were numerous commits in the server side development - could the developers share the news/goals with the forum? I think it might be interesting for more users – especially those out of core team. Thank you, Tomáš Fulajtár | Researcher T: +420-545-552-340tel:%2B420-545-552-340 toma...@moravia.commailto:toma...@moravia.com | moravia.comhttp://www.moravia.com/ | Skype: tomasfulajtar ___ Moses-support mailing list Moses-support@mit.edumailto:Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- Ulrich Germann Senior Researcher School of Informatics University of Edinburgh -- Ulrich Germann Senior Researcher School of Informatics University of Edinburgh ___ Moses-support mailing list Moses-support@mit.edumailto:Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edumailto:Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ---BeginMessage--- the cache is in moses/TranslationModel/PhraseDictionary.h line 160 It's 1 cache per thread On 16 June 2014 17:48, Mike Ladwig mdlad...@gmail.com wrote: Barry Haddow writes: The -threads switch is intended for command-line Moses, not for Moses server, but I didn't realise that it had this negative effect on the Moses server. Moses server uses the threading model provided by xmlrpc-c which, afaik, creates a new thread for each new tcp/ip connection. The
Re: [Moses-support] Europarl monolingual corpus
Go to http://opus.lingfil.uu.se Select the language pair your interested in and click on the language ID's in the column mono (or raw next to it) to download the data you like to use. Europarl is version 7 in the list. You can also take them from here: http://opus.lingfil.uu.se/Europarl/mono/ Best, Jörg ** Jörg Tiedemann http://stp.lingfil.uu.se/~joerg/ On May 6, 2015, at 7:25 AM, Hieu Hoang wrote: ah thx. The tgz file only has data for a subset of the languages. It would be useful to be able to download them all, or at least know how to extract them from the raw data. Hieu Hoang Researcher New York University, Abu Dhabi http://www.hoang.co.uk/hieu On 6 May 2015 at 03:04, Ulrich Germann ulrich.germ...@gmail.com wrote: Extract it from commoncrawl, of course! ;-) ... or get it here: http://www.statmt.org/wmt13/training-monolingual-europarl-v7.tgz - Uli On Mon, May 4, 2015 at 5:46 AM, Hieu Hoang hieuho...@gmail.com wrote: What's the easiest way get the single-language data from the Europarl corpus as described in the 1st table in: http://statmt.org/europarl/ I tried downloading the xml source http://statmt.org/europarl/v7/europarl.tgz stripping the xml and running split-sentence.perl, but this takes an unfathomably long time Hieu Hoang Researcher New York University, Abu Dhabi http://www.hoang.co.uk/hieu ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- Ulrich Germann Senior Researcher School of Informatics University of Edinburgh ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Fwd: Fwd: Server development
i'm not sure about v 2.1.1 but v 3.0 should have no problems using threads. (Apart from when using it with IRSTLM, but that's another issue) Hieu Hoang Researcher New York University, Abu Dhabi http://www.hoang.co.uk/hieu On 6 May 2015 at 11:38, Tomas Fulajtar toma...@moravia.com wrote: Hi Barry, Thanks for explanation – I was referring to “[Moses-support] mosesserver parallelization issue https://www.mail-archive.com/search?l=moses-support@mit.eduq=subject:%22%5C%5BMoses%5C-support%5C%5D+mosesserver+parallelization+issue%22o=newest” thread in April 2014. Do you thing the thread pool would be also the solution for this issue reported on 2.1.1 branch.? Thanks, Tomas *From:* Barry Haddow [mailto:bhad...@inf.ed.ac.uk] *Sent:* Tuesday, May 5, 2015 9:27 PM *To:* Hieu Hoang; moses-support *Subject:* Re: [Moses-support] Fwd: Fwd: Server development HI Tomas There were some issues in v2 with the way that caching was done in the binarised phrase table. It used a cache per thread, and mosesserver used a thread per request, so caching was effectively broken in the server. Since last Autumn, mosesserver uses a thread pool ... and the binarised phrase table is gone now anyway, cheers - Barry On 05/05/15 18:27, Hieu Hoang wrote: What limitations are you referring to? -- Forwarded message -- From: Ulrich Germann ulrich.germ...@gmail.com Date: 5 May 2015 19:49 Subject: [Moses-support] Fwd: Server development To: moses-support@mit.edu moses-support@mit.edu Cc: This response was meant to go to moses-support as well Tomas. -- Forwarded message -- From: *Tomas Fulajtar* toma...@moravia.com Date: Fri, Apr 3, 2015 at 5:03 PM Subject: RE: [Moses-support] Server development To: ugerm...@inf.ed.ac.uk ugerm...@inf.ed.ac.uk Hi Ulrich, Thanks for the thorough explanation - the idea of merging the server code back to moses is great. Apart from this (and I know is is a huge workload), were there any changes in the thread support? I know this part had some limitations – as discussed on the forum. Kind regards, Tomas *From:* Ulrich Germann [mailto:ulrich.germ...@gmail.com] *Sent:* Thursday, April 2, 2015 12:57 AM *To:* Tomas Fulajtar *Subject:* Re: [Moses-support] Server development Hi Tomas, the plan is to fold server capabilities into the main moses executable. In fact, that has already happened (in the sense that you can run the main moses executable in server mode), but functional equivalence with the old code has not been tested. There are currently no server tests included in the regression tests, so I left the old code mostly intact (adjusting only for changes in the API of functions called) for legacy reasons, but adding new functionality to mosesserver is extremely strongly DIScouraged. Supplying regression tests for server functionality, on the other hand, is equally strongly ENcouraged. In a nutshell, what you get back from calling mosesserver and moses --server should be identical. The long-term plan is to offer through RPC calls (almost) everything that moses offers in batch mode (i.e., send search and output parameters through json/RCP calls and have them noticed and respected). Notice the long-term there. So mosesserver is on its way out, and moses --server-port=port --server will replace the old call to mosesserver. Best regards - Uli On Wed, Apr 1, 2015 at 9:48 AM, Tomas Fulajtar toma...@moravia.com wrote: Dear all, I have spotted there were numerous commits in the server side development - could the developers share the news/goals with the forum? I think it might be interesting for more users – especially those out of core team. Thank you, *Tomáš Fulajtár* | Researcher *T:* +420-545-552-340 toma...@moravia.com | moravia.com http://www.moravia.com/ | *Skype:* tomasfulajtar ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- Ulrich Germann Senior Researcher School of Informatics University of Edinburgh -- Ulrich Germann Senior Researcher School of Informatics University of Edinburgh ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Fwd: Fwd: Server development
Moses has supported IRSTLM in training and decoding since the beginning and we've always tried to maintain the link. There's occasional issues with incompatible versions 'cos they're an independent project with their own timeline. But these are sorted out pdq. The threading issue is a IRSTLM-specific problem 'cos I think they have a non-threadsafe cache Hieu Hoang Researcher New York University, Abu Dhabi http://www.hoang.co.uk/hieu On 6 May 2015 at 11:58, Tomas Fulajtar toma...@moravia.com wrote: Dear Hieu, That sounds great – then it’s finally a time for upgrade J As Barry mentioned – could you briefly share with us where moses stands with IRTSLM? Thanks, Tomas *From:* Hieu Hoang [mailto:hieuho...@gmail.com] *Sent:* Wednesday, May 6, 2015 9:51 AM *To:* Tomas Fulajtar *Cc:* Barry Haddow; moses-support *Subject:* Re: [Moses-support] Fwd: Fwd: Server development i'm not sure about v 2.1.1 but v 3.0 should have no problems using threads. (Apart from when using it with IRSTLM, but that's another issue) Hieu Hoang Researcher New York University, Abu Dhabi http://www.hoang.co.uk/hieu On 6 May 2015 at 11:38, Tomas Fulajtar toma...@moravia.com wrote: Hi Barry, Thanks for explanation – I was referring to “[Moses-support] mosesserver parallelization issue https://www.mail-archive.com/search?l=moses-support@mit.eduq=subject:%22%5C%5BMoses%5C-support%5C%5D+mosesserver+parallelization+issue%22o=newest” thread in April 2014. Do you thing the thread pool would be also the solution for this issue reported on 2.1.1 branch.? Thanks, Tomas *From:* Barry Haddow [mailto:bhad...@inf.ed.ac.uk] *Sent:* Tuesday, May 5, 2015 9:27 PM *To:* Hieu Hoang; moses-support *Subject:* Re: [Moses-support] Fwd: Fwd: Server development HI Tomas There were some issues in v2 with the way that caching was done in the binarised phrase table. It used a cache per thread, and mosesserver used a thread per request, so caching was effectively broken in the server. Since last Autumn, mosesserver uses a thread pool ... and the binarised phrase table is gone now anyway, cheers - Barry On 05/05/15 18:27, Hieu Hoang wrote: What limitations are you referring to? -- Forwarded message -- From: Ulrich Germann ulrich.germ...@gmail.com Date: 5 May 2015 19:49 Subject: [Moses-support] Fwd: Server development To: moses-support@mit.edu moses-support@mit.edu Cc: This response was meant to go to moses-support as well Tomas. -- Forwarded message -- From: *Tomas Fulajtar* toma...@moravia.com Date: Fri, Apr 3, 2015 at 5:03 PM Subject: RE: [Moses-support] Server development To: ugerm...@inf.ed.ac.uk ugerm...@inf.ed.ac.uk Hi Ulrich, Thanks for the thorough explanation - the idea of merging the server code back to moses is great. Apart from this (and I know is is a huge workload), were there any changes in the thread support? I know this part had some limitations – as discussed on the forum. Kind regards, Tomas *From:* Ulrich Germann [mailto:ulrich.germ...@gmail.com] *Sent:* Thursday, April 2, 2015 12:57 AM *To:* Tomas Fulajtar *Subject:* Re: [Moses-support] Server development Hi Tomas, the plan is to fold server capabilities into the main moses executable. In fact, that has already happened (in the sense that you can run the main moses executable in server mode), but functional equivalence with the old code has not been tested. There are currently no server tests included in the regression tests, so I left the old code mostly intact (adjusting only for changes in the API of functions called) for legacy reasons, but adding new functionality to mosesserver is extremely strongly DIScouraged. Supplying regression tests for server functionality, on the other hand, is equally strongly ENcouraged. In a nutshell, what you get back from calling mosesserver and moses --server should be identical. The long-term plan is to offer through RPC calls (almost) everything that moses offers in batch mode (i.e., send search and output parameters through json/RCP calls and have them noticed and respected). Notice the long-term there. So mosesserver is on its way out, and moses --server-port=port --server will replace the old call to mosesserver. Best regards - Uli On Wed, Apr 1, 2015 at 9:48 AM, Tomas Fulajtar toma...@moravia.com wrote: Dear all, I have spotted there were numerous commits in the server side development - could the developers share the news/goals with the forum? I think it might be interesting for more users – especially those out of core team. Thank you, *Tomáš Fulajtár* | Researcher *T:* +420-545-552-340 toma...@moravia.com | moravia.com http://www.moravia.com/ | *Skype:* tomasfulajtar ___ Moses-support mailing list Moses-support@mit.edu
Re: [Moses-support] Fwd: Fwd: Server development
Dear Hieu, That sounds great – then it’s finally a time for upgrade ☺ As Barry mentioned – could you briefly share with us where moses stands with IRTSLM? Thanks, Tomas From: Hieu Hoang [mailto:hieuho...@gmail.com] Sent: Wednesday, May 6, 2015 9:51 AM To: Tomas Fulajtar Cc: Barry Haddow; moses-support Subject: Re: [Moses-support] Fwd: Fwd: Server development i'm not sure about v 2.1.1 but v 3.0 should have no problems using threads. (Apart from when using it with IRSTLM, but that's another issue) Hieu Hoang Researcher New York University, Abu Dhabi http://www.hoang.co.uk/hieu On 6 May 2015 at 11:38, Tomas Fulajtar toma...@moravia.commailto:toma...@moravia.com wrote: Hi Barry, Thanks for explanation – I was referring to “[Moses-support] mosesserver parallelization issuehttps://www.mail-archive.com/search?l=moses-support@mit.eduq=subject:%22%5C%5BMoses%5C-support%5C%5D+mosesserver+parallelization+issue%22o=newest” thread in April 2014. Do you thing the thread pool would be also the solution for this issue reported on 2.1.1 branch.? Thanks, Tomas From: Barry Haddow [mailto:bhad...@inf.ed.ac.ukmailto:bhad...@inf.ed.ac.uk] Sent: Tuesday, May 5, 2015 9:27 PM To: Hieu Hoang; moses-support Subject: Re: [Moses-support] Fwd: Fwd: Server development HI Tomas There were some issues in v2 with the way that caching was done in the binarised phrase table. It used a cache per thread, and mosesserver used a thread per request, so caching was effectively broken in the server. Since last Autumn, mosesserver uses a thread pool ... and the binarised phrase table is gone now anyway, cheers - Barry On 05/05/15 18:27, Hieu Hoang wrote: What limitations are you referring to? -- Forwarded message -- From: Ulrich Germann ulrich.germ...@gmail.commailto:ulrich.germ...@gmail.com Date: 5 May 2015 19:49 Subject: [Moses-support] Fwd: Server development To: moses-support@mit.edumailto:moses-support@mit.edu moses-support@mit.edumailto:moses-support@mit.edu Cc: This response was meant to go to moses-support as well Tomas. -- Forwarded message -- From: Tomas Fulajtar toma...@moravia.commailto:toma...@moravia.com Date: Fri, Apr 3, 2015 at 5:03 PM Subject: RE: [Moses-support] Server development To: ugerm...@inf.ed.ac.ukmailto:ugerm...@inf.ed.ac.uk ugerm...@inf.ed.ac.ukmailto:ugerm...@inf.ed.ac.uk Hi Ulrich, Thanks for the thorough explanation - the idea of merging the server code back to moses is great. Apart from this (and I know is is a huge workload), were there any changes in the thread support? I know this part had some limitations – as discussed on the forum. Kind regards, Tomas From: Ulrich Germann [mailto:ulrich.germ...@gmail.commailto:ulrich.germ...@gmail.com] Sent: Thursday, April 2, 2015 12:57 AM To: Tomas Fulajtar Subject: Re: [Moses-support] Server development Hi Tomas, the plan is to fold server capabilities into the main moses executable. In fact, that has already happened (in the sense that you can run the main moses executable in server mode), but functional equivalence with the old code has not been tested. There are currently no server tests included in the regression tests, so I left the old code mostly intact (adjusting only for changes in the API of functions called) for legacy reasons, but adding new functionality to mosesserver is extremely strongly DIScouraged. Supplying regression tests for server functionality, on the other hand, is equally strongly ENcouraged. In a nutshell, what you get back from calling mosesserver and moses --server should be identical. The long-term plan is to offer through RPC calls (almost) everything that moses offers in batch mode (i.e., send search and output parameters through json/RCP calls and have them noticed and respected). Notice the long-term there. So mosesserver is on its way out, and moses --server-port=port --server will replace the old call to mosesserver. Best regards - Uli On Wed, Apr 1, 2015 at 9:48 AM, Tomas Fulajtar toma...@moravia.commailto:toma...@moravia.com wrote: Dear all, I have spotted there were numerous commits in the server side development - could the developers share the news/goals with the forum? I think it might be interesting for more users – especially those out of core team. Thank you, Tomáš Fulajtár | Researcher T: +420-545-552-340tel:%2B420-545-552-340 toma...@moravia.commailto:toma...@moravia.com | moravia.comhttp://www.moravia.com/ | Skype: tomasfulajtar ___ Moses-support mailing list Moses-support@mit.edumailto:Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- Ulrich Germann Senior Researcher School of Informatics University of Edinburgh -- Ulrich Germann Senior Researcher School of Informatics University of Edinburgh ___ Moses-support mailing list Moses-support@mit.edumailto:Moses-support@mit.edu
Re: [Moses-support] Fwd: Fwd: Server development
Hi Tomas The thread pool fixes the issue you mention, but it was never backported to v2.1.1. If you can move to v3, that would be the best way to go, cheers - Barry On 06/05/15 08:38, Tomas Fulajtar wrote: Hi Barry, Thanks for explanation – I was referring to “[Moses-support] mosesserver parallelization issue https://www.mail-archive.com/search?l=moses-support@mit.eduq=subject:%22%5C%5BMoses%5C-support%5C%5D+mosesserver+parallelization+issue%22o=newest” thread in April 2014. Do you thing the thread pool would be also the solution for this issue reported on 2.1.1 branch.? Thanks, Tomas *From:*Barry Haddow [mailto:bhad...@inf.ed.ac.uk] *Sent:* Tuesday, May 5, 2015 9:27 PM *To:* Hieu Hoang; moses-support *Subject:* Re: [Moses-support] Fwd: Fwd: Server development HI Tomas There were some issues in v2 with the way that caching was done in the binarised phrase table. It used a cache per thread, and mosesserver used a thread per request, so caching was effectively broken in the server. Since last Autumn, mosesserver uses a thread pool ... and the binarised phrase table is gone now anyway, cheers - Barry On 05/05/15 18:27, Hieu Hoang wrote: What limitations are you referring to? -- Forwarded message -- From: Ulrich Germann ulrich.germ...@gmail.com mailto:ulrich.germ...@gmail.com Date: 5 May 2015 19:49 Subject: [Moses-support] Fwd: Server development To: moses-support@mit.edu mailto:moses-support@mit.edu moses-support@mit.edu mailto:moses-support@mit.edu Cc: This response was meant to go to moses-support as well Tomas. -- Forwarded message -- From: *Tomas Fulajtar* toma...@moravia.com mailto:toma...@moravia.com Date: Fri, Apr 3, 2015 at 5:03 PM Subject: RE: [Moses-support] Server development To: ugerm...@inf.ed.ac.uk mailto:ugerm...@inf.ed.ac.uk ugerm...@inf.ed.ac.uk mailto:ugerm...@inf.ed.ac.uk Hi Ulrich, Thanks for the thorough explanation - the idea of merging the server code back to moses is great. Apart from this (and I know is is a huge workload), were there any changes in the thread support? I know this part had some limitations – as discussed on the forum. Kind regards, Tomas *From:*Ulrich Germann [mailto:ulrich.germ...@gmail.com mailto:ulrich.germ...@gmail.com] *Sent:* Thursday, April 2, 2015 12:57 AM *To:* Tomas Fulajtar *Subject:* Re: [Moses-support] Server development Hi Tomas, the plan is to fold server capabilities into the main moses executable. In fact, that has already happened (in the sense that you can run the main moses executable in server mode), but functional equivalence with the old code has not been tested. There are currently no server tests included in the regression tests, so I left the old code mostly intact (adjusting only for changes in the API of functions called) for legacy reasons, but adding new functionality to mosesserver is extremely strongly DIScouraged. Supplying regression tests for server functionality, on the other hand, is equally strongly ENcouraged. In a nutshell, what you get back from calling mosesserver and moses --server should be identical. The long-term plan is to offer through RPC calls (almost) everything that moses offers in batch mode (i.e., send search and output parameters through json/RCP calls and have them noticed and respected). Notice the long-term there. So mosesserver is on its way out, and moses --server-port=port --server will replace the old call to mosesserver. Best regards - Uli On Wed, Apr 1, 2015 at 9:48 AM, Tomas Fulajtar toma...@moravia.com mailto:toma...@moravia.com wrote: Dear all, I have spotted there were numerous commits in the server side development - could the developers share the news/goals with the forum? I think it might be interesting for more users – especially those out of core team. Thank you, *Tomáš Fulajtár*| Researcher *T:* +420-545-552-340 tel:%2B420-545-552-340 toma...@moravia.com mailto:toma...@moravia.com | moravia.com http://www.moravia.com/ | *Skype:* tomasfulajtar ___ Moses-support mailing list Moses-support@mit.edu mailto:Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- Ulrich Germann Senior Researcher School of Informatics University of Edinburgh -- Ulrich Germann Senior Researcher School of Informatics University of Edinburgh ___ Moses-support mailing list Moses-support@mit.edu mailto:Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Fwd: Transliteration model is using processPhraseTable, which is not found in Moses version 3.0
Dear Nadir, Thank you very much for explaining transliteration. I have yes for both transliteration-module and post-decoding-transliteration in the EMS configuration file used for en-ru. Best Regards, Ergun Ergun Biçici, CNGL, School of Computing, DCU, www.cngl.ie http://www.computing.dcu.ie/~ebicici/ -- Forwarded message -- From: Nadir Durrani nadir.durr...@nu.edu.pk Date: Wed, May 6, 2015 at 11:17 AM Subject: Re: Transliteration model is using processPhraseTable, which is not found in Moses version 3.0 To: Ergun Bicici ergun.bic...@computing.dcu.ie Hi Ergun, If you are only going to do transliteration-module = yes Moses will train the transliteration system but not going to do anything with it. You have to select whether you want to use post-deocoding or in-decoding transliteration. In post-decoding method, transliteration is done in the post-decoding step i.e. the decoder has translated all the sentences and now you just need to replace OOV words with their best transliteration given the context. This is Method 2 as described in the following paper http://aclweb.org/anthology//E/E14/E14-4029.pdf you can enable it by using post-decoding-transliteration = yes Using in-decoding method (Method 3 in the paper), you do transliteration inside the decoder on the fly. The advantage of this over Method 2 in theory is that you can also reorder the OOV word and make use of other features. But it does not give any clear-cut gains. More details here: http://www.statmt.org/moses/?n=Advanced.OOVs Nadir On Tue, May 5, 2015 at 5:33 PM, Ergun Bicici ergun.bic...@computing.dcu.ie wrote: Hi Nadir, I am using Moses 3.0 and for transliteration to work, I copied scripts/Transliteration/ from latest onto Moses 3.0 path, re-ran, and obtained translation results. Best Regards, Ergun Ergun Biçici, CNGL, School of Computing, DCU, www.cngl.ie http://www.computing.dcu.ie/~ebicici/ On Mon, May 4, 2015 at 7:32 AM, Nadir Durrani nadir.durr...@nu.edu.pk wrote: Hi Ergun, processPhraseTable is no longer supported by Moses. But I see that Phil Williams has already fixed this problem in transliteration module, by changing `$MOSES_SRC/scripts/training/filter-model-given-input.pl $TRANSLIT_MODEL/evaluation/$eval_file.filtered $TRANSLIT_MODEL/evaluation/$eval_file.moses.table.ini $TRANSLIT_MODEL/evaluation/$eval_file -Binarizer $MOSES_SRC/bin/processPhraseTable`; to `$MOSES_SRC/scripts/training/filter-model-given-input.pl $TRANSLIT_MODEL/evaluation/$eval_file.filtered $TRANSLIT_MODEL/evaluation/$eval_file.moses.table.ini $TRANSLIT_MODEL/evaluation/$eval_file -Binarizer $MOSES_SRC/bin/CreateOnDiskPt 1 1 4 100 2`; in path-to-moses/scripts/Transliteration/in-decoding-transliteration.pl Here's the commit https://github.com/moses-smt/mosesdecoder/commit/7e54e23fe234ac48f44b0e473d09a5b4d5f6 May be you pulled and in between version where the processPhraseTable was removed but transliteration scripts were not fixed. Cheers, Nadir On Mon, May 4, 2015 at 7:46 AM, moses-support-requ...@mit.edu wrote: Send Moses-support mailing list submissions to moses-support@mit.edu To subscribe or unsubscribe via the World Wide Web, visit http://mailman.mit.edu/mailman/listinfo/moses-support or, via email, send a message with subject or body 'help' to moses-support-requ...@mit.edu You can reach the person managing the list at moses-support-ow...@mit.edu When replying, please edit your Subject line so it is more specific than Re: Contents of Moses-support digest... Today's Topics: 1. Re: 12-gram language model ARPA file for 16GB (liling tan) 2. Transliteration model is using processPhraseTable, which is not found in Moses version 3.0 (Ergun Bicici) 3. Re: Transliteration model is using processPhraseTable, which is not found in Moses version 3.0 (Hieu Hoang) 4. Europarl monolingual corpus (Hieu Hoang) -- Message: 1 Date: Sun, 3 May 2015 19:44:12 +0200 From: liling tan alvati...@gmail.com Subject: Re: [Moses-support] 12-gram language model ARPA file for 16GB To: moses-support moses-support@mit.edu Message-ID: CAKzPaJJ7fY=9C89POact542vu32d+H3=0i_Dnaj=yfizbfa...@mail.gmail.com Content-Type: text/plain; charset=utf-8 Dear Moses devs/users, For now, I only know that it takes more than 250GB. I've 250GB of free space and KenLM got poisoned by insufficient space... Does anyone have an idea how big would a 12-gram language model ARPA file trained on 16GB of text become? STDERR: === 1/5 Counting and sorting n-grams === Reading /media/2tb/wmt15/corpus.truecase/train-lm.en
Re: [Moses-support] Problem with qsub when running experiments
Hi again, this is the command I use: ~/mosesdecoder/scripts/ems/experiment.perl -config config.toy -exec I have also tried a modification of the configuration file config.placeables. This is when the problem started. To make sure that it was not the config file, I tried with an older config.file from another experiment which finished without problems (just changed the path for the experiment). I tried to run an older experiment which worked and also failed this time with the same error. I attach both files, so that you can see them. I don't know if it might help, but I also tried to run the experiment step-by-step and at decoding I also got an error: /home/hermesta/mosesdecoder/scripts/training/mert-moses.pl /home/hermesta/Exps/KES_newDev_placeholders/data/dev/KES10.dev.preproc.tok.true.en /home/hermesta/Exps/KES_newDev_placeholders/data/dev/KES10.dev.preproc.tok.true.es /home/hermesta/mosesdecoder/bin/moses /home/hermesta/Exps/KES_newDev_placeholders/working/train/model/moses.ini --mertdir /home/hermesta/mosesdecoder/bin/ Using SCRIPTS_ROOTDIR: /home/hermesta/mosesdecoder/scripts filtering the phrase tables... mié may 6 15:58:15 CEST 2015 exec: /home/hermesta/mosesdecoder/scripts/training/filter-model-given-input.pl ./filtered /home/hermesta/Exps/KES_newDev_placeholders/working/train/model/moses.ini /home/hermesta/Exps/KES_newDev_placeholders/data/dev/KES10.dev.preproc.tok.true.en Executing: /home/hermesta/mosesdecoder/scripts/training/filter-model-given-input.pl ./filtered /home/hermesta/Exps/KES_newDev_placeholders/working/train/model/moses.ini /home/hermesta/Exps/KES_newDev_placeholders/data/dev/KES10.dev.preproc.tok.true.en filterphrases.out 2 filterphrases.err Exit code: 1 ERROR: Failed to run '/home/hermesta/mosesdecoder/scripts/training/filter-model-given-input.pl ./filtered /home/hermesta/Exps/KES_newDev_placeholders/working/train/model/moses.ini /home/hermesta/Exps/KES_newDev_placeholders/data/dev/KES10.dev.preproc.tok.true.en'. at /home/hermesta/mosesdecoder/scripts/training/mert-moses.pl line 1719. When checking line 1719 of mert-moses.pl, I realized it is also related to qsub. Thank you so much! Carla El 06.05.2015 15:27, Barry Haddow escribió: Hi Carla Not sure what's going on, and no reason why things should change when you installed asiya. Something else must have changed. Could you post your EMS config file, and the exact command you use to run EMS? cheers - Barry On 06/05/15 13:32, carla.pa...@hermestrans.com wrote: Hi Barry, thanks for your prompt reply. If I am not wrong the name of the server is hermesta-Z10PE-D8-WS (I have taken it from the machine information, I attach a screenshot). If I should look somewhere else please let me know. Thanks, Carla El 06.05.2015 14:10, Barry Haddow escribió: Hi Carla What's your server called? There's a hard-coded list of Edinburgh machines in ems, so I'm wondering if it collides with one of them, cheers - Barry On 06/05/15 12:56, carla.pa...@hermestrans.com wrote: Hi everyone, First of all, thanks for reading and hopefully giving me some useful pointer. I am running several SMT experiments on an Ubuntu machine. It is a multicore machine, but I have commented out the options for running experiments on multicore machines in the config.file. Up to last week, I was able to run experiments without problems. However, since yesterday I get the error: Can't exec qsub: No existe el archivo o el directorio at /home/hermesta/mosesdecoder/scripts/ems/experiment.perl line 1291. Does anyone know what could be going on? The only thing I did was installing asiya to assess MT output. My guess is that somehow MOSES detects that it is a multicore machine and tries to parallelize jobs. However, I don't understand why this was not happening last week, for instance. I have also tried to update MOSES by running git pull and I also reinstalled MOSES hoping this would fix the problem. My background is linguistics, and thus I am a bit lost now. Thank you very much, Carla Parra Escartín Marie Curie ER - EXPERT ITN Hermes Traducciones ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ### CONFIGURATION FILE FOR AN SMT EXPERIMENT ### [GENERAL] ### directory in which experiment is run # working-dir = /home/hermesta/Exps/KES_newDev_placeholders # specification of the language pair input-extension = en output-extension = es pair-extension = en-es ### directories that contain tools and data # # moses moses-src-dir = /home/hermesta/mosesdecoder # # moses binaries moses-bin-dir = $moses-src-dir/bin # # moses scripts moses-script-dir = $moses-src-dir/scripts # # directory where GIZA++/MGIZA programs resides external-bin-dir = /home/hermesta/my/installation/dir/fast_align # # srilm #srilm-dir =
Re: [Moses-support] Problem with qsub when running experiments
Hi Carla Not sure what's going on, and no reason why things should change when you installed asiya. Something else must have changed. Could you post your EMS config file, and the exact command you use to run EMS? cheers - Barry On 06/05/15 13:32, carla.pa...@hermestrans.com wrote: Hi Barry, thanks for your prompt reply. If I am not wrong the name of the server is hermesta-Z10PE-D8-WS (I have taken it from the machine information, I attach a screenshot). If I should look somewhere else please let me know. Thanks, Carla El 06.05.2015 14:10, Barry Haddow escribió: Hi Carla What's your server called? There's a hard-coded list of Edinburgh machines in ems, so I'm wondering if it collides with one of them, cheers - Barry On 06/05/15 12:56, carla.pa...@hermestrans.com wrote: Hi everyone, First of all, thanks for reading and hopefully giving me some useful pointer. I am running several SMT experiments on an Ubuntu machine. It is a multicore machine, but I have commented out the options for running experiments on multicore machines in the config.file. Up to last week, I was able to run experiments without problems. However, since yesterday I get the error: Can't exec qsub: No existe el archivo o el directorio at /home/hermesta/mosesdecoder/scripts/ems/experiment.perl line 1291. Does anyone know what could be going on? The only thing I did was installing asiya to assess MT output. My guess is that somehow MOSES detects that it is a multicore machine and tries to parallelize jobs. However, I don't understand why this was not happening last week, for instance. I have also tried to update MOSES by running git pull and I also reinstalled MOSES hoping this would fix the problem. My background is linguistics, and thus I am a bit lost now. Thank you very much, Carla Parra Escartín Marie Curie ER - EXPERT ITN Hermes Traducciones ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Problem with qsub when running experiments
Hi Carla What's your server called? There's a hard-coded list of Edinburgh machines in ems, so I'm wondering if it collides with one of them, cheers - Barry On 06/05/15 12:56, carla.pa...@hermestrans.com wrote: Hi everyone, First of all, thanks for reading and hopefully giving me some useful pointer. I am running several SMT experiments on an Ubuntu machine. It is a multicore machine, but I have commented out the options for running experiments on multicore machines in the config.file. Up to last week, I was able to run experiments without problems. However, since yesterday I get the error: Can't exec qsub: No existe el archivo o el directorio at /home/hermesta/mosesdecoder/scripts/ems/experiment.perl line 1291. Does anyone know what could be going on? The only thing I did was installing asiya to assess MT output. My guess is that somehow MOSES detects that it is a multicore machine and tries to parallelize jobs. However, I don't understand why this was not happening last week, for instance. I have also tried to update MOSES by running git pull and I also reinstalled MOSES hoping this would fix the problem. My background is linguistics, and thus I am a bit lost now. Thank you very much, Carla Parra Escartín Marie Curie ER - EXPERT ITN Hermes Traducciones ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] processPhraseTableMin segfault
Hello everyone, I have tried using processPhraseTableMin on my model trained on Europarl v7 that's just too large to load into memory uncompressed, but I get a segfault at each attempt. Any idea why, or any workaround to try? Here is the command used and the output: $ processPhraseTableMin -threads 3 -in train/model/phrase-table.gz -nscores 4 -out binarised-model/phrase-table Used options: Text phrase table will be read from: train/model/phrase-table.gz Output phrase table will be written to: binarised-model/phrase-table.minphr Step size for source landmark phrases: 2^10=1024 Source phrase fingerprint size: 16 bits / P(fp)=1.52588e-05 Selected target phrase encoding: Huffman + PREnc Maxiumum allowed rank for PREnc: 100 Number of score components in phrase table: 4 Single Huffman code set for score components: no Using score quantization: no Explicitly included alignment information: yes Running with 3 threads Pass 1/3: Creating hash function for rank assignment ..[500] ..[1000] ..[1500] ..[2000] ..[2500] ..[3000] ..[3500] ..[4000] ..[4500] ..[5000] ..[5500] ..[6000] ... Pass 2/3: Creating source phrase index + Encoding target phrases *** Segmentation fault Register dump: RAX: 00683fc0 RBX: 0001 RCX: RDX: 0001 RSI: 0008 RDI: 7ffc83939418 RBP: 0008 R8 : 7fc2575af960 R9 : 7fc258a35740 R10: 0008 R11: 7fc257279260 R12: 014b8560 R13: 0003 R14: 7ffc83939350 R15: 7ffc839393f8 RSP: 7ffc83938f30 RIP: 7fc25727927b EFLAGS: 00010202 CS: 0033 FS: GS: Trap: 000e Error: 0004 OldMask: CR2: FPUCW: 037f FPUSW: TAG: RIP: RDP: ST(0) ST(1) ST(2) ST(3) ST(4) ST(5) ST(6) ST(7) mxcsr: 1f80 XMM0: 3cdd67ba XMM1: 3cdd67ba XMM2: 3cdd67ba XMM3: 3cdd67ba XMM4: 3cdd67ba XMM5: 3cdd67ba XMM6: 3cdd67ba XMM7: 3cdd67ba XMM8: 3cdd67ba XMM9: 3cdd67ba XMM10: 3cdd67ba XMM11: 3cdd67ba XMM12: 3cdd67ba XMM13: 3cdd67ba XMM14: 3cdd67ba XMM15: 3cdd67ba Backtrace: /usr/lib/libc.so.6(fwrite+0x1b)[0x7fc25727927b] processPhraseTableMin[0x42c4a4] processPhraseTableMin[0x42c662] processPhraseTableMin[0x417dc3] processPhraseTableMin[0x405f18] /usr/lib/libc.so.6(__libc_start_main+0xf0)[0x7fc257230800] processPhraseTableMin[0x406d29] Memory map: 0040-00483000 r-xp 08:01 8130298 /opt/mosesdecoder/bin/processPhraseTableMin 00682000-00684000 rw-p 00082000 08:01 8130298 /opt/mosesdecoder/bin/processPhraseTableMin 00684000-00685000 rw-p 00:00 0 014b7000-014d8000 rw-p 00:00 0 [heap] 7fc1f2f9-7fc215a76000 rw-p 00:00 0 7fc21c00-7fc21f32f000 rw-p 00:00 0 7fc21f32f000-7fc22000 ---p 00:00 0 7fc22400-7fc2276e6000 rw-p 00:00 0 7fc2276e6000-7fc22800 ---p 00:00 0 7fc22800-7fc22b34a000 rw-p 00:00 0 7fc22b34a000-7fc22c00 ---p 00:00 0 7fc22c00-7fc23000 rw-p 00:00 0 7fc23000-7fc23400 rw-p 00:00 0 7fc23400-7fc237fff000 rw-p 00:00 0 7fc237fff000-7fc23800 ---p 00:00 0 7fc23800-7fc23c00 rw-p 00:00 0 7fc23c00-7fc24000 rw-p 00:00 0 7fc24000-7fc24400 rw-p 00:00 0 7fc24400-7fc2442f6000 rw-p 00:00 0 7fc2442f6000-7fc24800 ---p 00:00 0 7fc24800-7fc2486b5000 rw-p 00:00 0 7fc2486b5000-7fc24c00 ---p 00:00 0 7fc24c00-7fc24cb09000 rw-p 00:00 0 7fc24cb09000-7fc25000 ---p 00:00 0 7fc253a09000-7fc253a0a000 ---p 00:00
Re: [Moses-support] lattice MBR - what happened to my nbest features?
Hi Jeremy It's been a long time since I looked at this, but I think these are the component scores in the linearised corpus bleu used in Lattice MBR (see Tromble et al (2008), section 5). They are used to rescore the nbest list, and must be implemented by replacing the original feature vectors. So it looks like lattice MBR doesn't work with nbest lists, or at least doesn't give you what you need for tuning. Do you really want to tune with lattice MBR? It's going to be very slow, cheers - Barry On 06/05/15 20:49, Jeremy Gwinnup wrote: Hi, I’ve been attempting to experiment with lattice MBR with various settings and I see something weird happen to my nbest lists: 0 ||| the prime ministers of india and japan meet in tokyo ||| Distortion0= … etc etc becomes 0 ||| the prime ministers of india and japan meet in tokyo ||| map: 0 w: 12 11.19 8.06 6.63 5.39 My feature weights get replaced by map and w feature with 5 weights I’m setting the following as moses.ini parameters: lminimum-bayes-risk lmbr-p lmbr-r mbr-scale lmbr-pruning-factor Anything extra that I need to know as far as what map and w are, or how to get my normal features back in the nbest list? Thanks! -Jeremy ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] processPhraseTableMin segfault
You are welcome. You are also welcome to improve robustness. :) W dniu 2015-05-06 18:36, Valmi Dufour-Lussier napisał(a): Your suggestion helped me figure what was wrong. I'm feeling very stupid now: I hadn't created the binarised-model directory, and I guess processPhraseTableMin is slightly lacking in this kind of robustness--no offence meant, you making this tool available is much, much appreciated. This is a good thing, since the phrase-table is almost 3 GB gzipped. Thank you, and sorry, Valmi 2015-05-06 13:11 GMT-03:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Hi, Could it be, you do not have write-access in /tmp ? If not, try -T . to create temporary files in the local dir. Best, Marcin ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] lattice MBR - what happened to my nbest features?
Hi, I’ve been attempting to experiment with lattice MBR with various settings and I see something weird happen to my nbest lists: 0 ||| the prime ministers of india and japan meet in tokyo ||| Distortion0= … etc etc becomes 0 ||| the prime ministers of india and japan meet in tokyo ||| map: 0 w: 12 11.19 8.06 6.63 5.39 My feature weights get replaced by map and w feature with 5 weights I’m setting the following as moses.ini parameters: lminimum-bayes-risk lmbr-p lmbr-r mbr-scale lmbr-pruning-factor Anything extra that I need to know as far as what map and w are, or how to get my normal features back in the nbest list? Thanks! -Jeremy ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Moses as a service under release 3.0
Hi, I am attempting to use the binary release 3 moses as a service. Recent discussion indicates that mosesserver is depreciated and that moses --server should be used instead. However, running 'moses --server --server-port 1234 -f phrase-model/moses.ini' complains about illegal switches. So I looked at mosseserver. It complains about libxmlrpc libraries (*.so.7) not being found. So I installed the libxmlrpc-c++ package (1.33.06) and made soft links from the so.8 libs to so.7 so that mosesserver can load the library (figuring that the functionality/interface was the same). However, moseserver seg faults, when called with 'mosesserver --server-port 1234 -f phrase-model/moses.ini'. Does anyone know where/how to get libxmlrpc libraries required by moses release 3.0 or otherwise make moses as a service w/o having to re-build it? I'm using Ubunut 14.04. If all else fails I can use moses' stdio but I was hoping to get this working. Thank you for your time. - Kwame ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Moses as a service under release 3.0
Hi Kwame The --server switch was introduced post v3, so you would have to use current master if you want to go down that route. However you will still hit the xmlrpc issues. made soft links from the so.8 libs to so.7 so that mosesserver can load the library (figuring that the functionality/interface was the same). Probably there's a reason for the version number change from 7 to 8, and you've just hit the incompatibility. Make sure you build and run against the same version of xmlrpc-c - we're using 1.16.33 which is probably old, but works. cheers - Barry On 06/05/15 19:59, kwame porter robinson wrote: Hi, I am attempting to use the binary release 3 moses as a service. Recent discussion indicates that mosesserver is depreciated and that moses --server should be used instead. However, running 'moses --server --server-port 1234 -f phrase-model/moses.ini' complains about illegal switches. So I looked at mosseserver. It complains about libxmlrpc libraries (*.so.7) not being found. So I installed the libxmlrpc-c++ package (1.33.06) and made soft links from the so.8 libs to so.7 so that mosesserver can load the library (figuring that the functionality/interface was the same). However, moseserver seg faults, when called with 'mosesserver --server-port 1234 -f phrase-model/moses.ini'. Does anyone know where/how to get libxmlrpc libraries required by moses release 3.0 or otherwise make moses as a service w/o having to re-build it? I'm using Ubunut 14.04. If all else fails I can use moses' stdio but I was hoping to get this working. Thank you for your time. - Kwame ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] processPhraseTableMin segfault
Hi, Thanks for your message. Doesn't seem like it. I've tried it anyway and unfortunately still got a segfault at the same point. Best, Valmi 2015-05-06 13:11 GMT-03:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Hi, Could it be, you do not have write-access in /tmp ? If not, try -T . to create temporary files in the local dir. Best, Marcin ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] processPhraseTableMin segfault
Your suggestion helped me figure what was wrong. I'm feeling very stupid now: I hadn't created the binarised-model directory, and I guess processPhraseTableMin is slightly lacking in this kind of robustness--no offence meant, you making this tool available is much, much appreciated. This is a good thing, since the phrase-table is almost 3 GB gzipped. Thank you, and sorry, Valmi 2015-05-06 13:11 GMT-03:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Hi, Could it be, you do not have write-access in /tmp ? If not, try -T . to create temporary files in the local dir. Best, Marcin ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] processPhraseTableMin segfault
Hm. Can you share the phrase-table? W dniu 2015-05-06 18:21, Valmi Dufour-Lussier napisał(a): Hi, Thanks for your message. Doesn't seem like it. I've tried it anyway and unfortunately still got a segfault at the same point. Best, Valmi 2015-05-06 13:11 GMT-03:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Hi, Could it be, you do not have write-access in /tmp ? If not, try -T . to create temporary files in the local dir. Best, Marcin ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] processPhraseTableMin segfault
Hi, Could it be, you do not have write-access in /tmp ? If not, try -T . to create temporary files in the local dir. Best, Marcin W dniu 2015-05-06 18:05, Valmi Dufour-Lussier napisał(a): Hello everyone, I have tried using processPhraseTableMin on my model trained on Europarl v7 that's just too large to load into memory uncompressed, but I get a segfault at each attempt. Any idea why, or any workaround to try? Here is the command used and the output: $ processPhraseTableMin -threads 3 -in train/model/phrase-table.gz -nscores 4 -out binarised-model/phrase-table Used options: Text phrase table will be read from: train/model/phrase-table.gz Output phrase table will be written to: binarised-model/phrase-table.minphr Step size for source landmark phrases: 2^10=1024 Source phrase fingerprint size: 16 bits / P(fp)=1.52588e-05 Selected target phrase encoding: Huffman + PREnc Maxiumum allowed rank for PREnc: 100 Number of score components in phrase table: 4 Single Huffman code set for score components: no Using score quantization: no Explicitly included alignment information: yes Running with 3 threads Pass 1/3: Creating hash function for rank assignment ..[500] ..[1000] ..[1500] ..[2000] ..[2500] ..[3000] ..[3500] ..[4000] ..[4500] ..[5000] ..[5500] ..[6000] ... Pass 2/3: Creating source phrase index + Encoding target phrases *** Segmentation fault Register dump: RAX: 00683fc0 RBX: 0001 RCX: RDX: 0001 RSI: 0008 RDI: 7ffc83939418 RBP: 0008 R8 : 7fc2575af960 R9 : 7fc258a35740 R10: 0008 R11: 7fc257279260 R12: 014b8560 R13: 0003 R14: 7ffc83939350 R15: 7ffc839393f8 RSP: 7ffc83938f30 RIP: 7fc25727927b EFLAGS: 00010202 CS: 0033 FS: GS: Trap: 000e Error: 0004 OldMask: CR2: FPUCW: 037f FPUSW: TAG: RIP: RDP: ST(0) ST(1) ST(2) ST(3) ST(4) ST(5) ST(6) ST(7) mxcsr: 1f80 XMM0: 3cdd67ba XMM1: 3cdd67ba XMM2: 3cdd67ba XMM3: 3cdd67ba XMM4: 3cdd67ba XMM5: 3cdd67ba XMM6: 3cdd67ba XMM7: 3cdd67ba XMM8: 3cdd67ba XMM9: 3cdd67ba XMM10: 3cdd67ba XMM11: 3cdd67ba XMM12: 3cdd67ba XMM13: 3cdd67ba XMM14: 3cdd67ba XMM15: 3cdd67ba Backtrace: /usr/lib/libc.so.6(fwrite+0x1b)[0x7fc25727927b] processPhraseTableMin[0x42c4a4] processPhraseTableMin[0x42c662] processPhraseTableMin[0x417dc3] processPhraseTableMin[0x405f18] /usr/lib/libc.so.6(__libc_start_main+0xf0)[0x7fc257230800] processPhraseTableMin[0x406d29] Memory map: 0040-00483000 r-xp 08:01 8130298 /opt/mosesdecoder/bin/processPhraseTableMin 00682000-00684000 rw-p 00082000 08:01 8130298 /opt/mosesdecoder/bin/processPhraseTableMin 00684000-00685000 rw-p 00:00 0 014b7000-014d8000 rw-p 00:00 0 [heap] 7fc1f2f9-7fc215a76000 rw-p 00:00 0 7fc21c00-7fc21f32f000 rw-p 00:00 0 7fc21f32f000-7fc22000 ---p 00:00 0 7fc22400-7fc2276e6000 rw-p 00:00 0 7fc2276e6000-7fc22800 ---p 00:00 0 7fc22800-7fc22b34a000 rw-p 00:00 0 7fc22b34a000-7fc22c00 ---p 00:00 0 7fc22c00-7fc23000 rw-p 00:00 0 7fc23000-7fc23400 rw-p 00:00 0 7fc23400-7fc237fff000 rw-p 00:00 0 7fc237fff000-7fc23800 ---p 00:00 0 7fc23800-7fc23c00 rw-p 00:00 0 7fc23c00-7fc24000 rw-p 00:00 0 7fc24000-7fc24400 rw-p 00:00 0 7fc24400-7fc2442f6000 rw-p 00:00 0 7fc2442f6000-7fc24800 ---p 00:00 0 7fc24800-7fc2486b5000 rw-p 00:00 0 7fc2486b5000-7fc24c00 ---p
Re: [Moses-support] Extract phrases with no internal alignment points
Hi, the definition of consistent phrase pair that is the basis of the phrase extraction method requires that each phrase pair has at least one alignment point. You should change the function ExtractTask::extract in phrase-extract/extract-main.cpp if you want to base phrase extraction on a different principle. -phi On Tue, May 5, 2015 at 5:18 AM, Miguel Graça miguelrgr...@gmail.com wrote: Hello, I couldn't find any information about this. Is it possible to not only extract phrases which have at least one alignment point, but also extract phrases where there are no alignment points? Thanks in advance, Miguel ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] mert.out error
Hi, what are the contents of extract.out and extract.err? -phi On Fri, May 1, 2015 at 12:54 PM, emna hkiri emna.hk...@gmail.com wrote: dear friends I am stuck with this message in my mert.out after going through line 1539: Scoring the nbestlist. exec: /home/emna/emna/working/mert-work/extractor.sh Executing: /home/emna/emna/working/mert-work/extractor.sh extract.out 2 extract.err Exit code: 1 ERROR: Failed to run '/home/emna/emna/working/mert-work/extractor.sh'. at /home/emna/emna/mosesdecoder/scripts/training/mert-moses.pl line 1539. any help please? ? the file mert-work/extractor.sh is in the correct place thank you in advance ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Multiple Translation Tables and Back-off Models in Moses
Hi, I added a section to the Moses documentation on how to deal with this situation: http://www.statmt.org/moses/?n=Advanced.Models#ntoc5 -phi *Caveat: Multiple Translation Tables and Lexicalized Reordering* You may specify any number of lexicalized reordering models. Each of them will score any translation option, no matter where it comes from. If a lexicalized reordering table does not have an entry for a translation options, it will not assign any score to it. In other words, such a translation option is given the probability 1 no matter how it is reordered. This may not be the way you want to handle it. For instance, if you have an in-domain translation table and an *out-of-domain* translation table, you can also provide an *in-domain* reordering table and an*out-of-domain* reordering table. If a phrase pair occurs in both translation tables, it will be scored by both reordering tables. However, if a phrase pairs occurs only in one of the phrase tables (and hence reordering tables), it will be only score by one of them and get a free ride with the other. this will have the undesirable effect of discouraging phrase pairs that occur in both tables. To avoid this, you can add default scores to the reordering table: LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff [...] default-scores=0.5,0.3,0.2,0.5,0.3,0.2 LexicalReordering name=LexicalReordering1 num-features=6 type=wbe-msd-bidirectional-fe-allff [...] default-scores=0.5,0.3,0.2,0.5,0.3,0.2 On Wed, May 6, 2015 at 3:00 AM, xinzy...@mail.ustc.edu.cn wrote: Hi, I am studying moses to do some research abou SMT. I have read advanced models in moses at the website http://www.statmt.org/moses/?n=Advanced.Models#ntoc5 . I have some questions about the introduction of Multiple Translation Tables and Back-off Models. From the website, I know that moses allows the use of multiple translation tables. But the lexicalized reordering model is not mentioned in the introduction of Multiple Translation Tables and Back-off Models. So, *could I use two translation tables, which I denote translation table1 and translation table2 respectively, and one lexicalized reordering table* according to the introduction of Multiple Translation Tables and Back-off Models? The phrase pairs in reordering table is the same with translation table1, but not the same with translation table2. When I use multiple translation tables and only one reordering table, a phrase pair may exist in only one translation table and don't exist in the reordering model, how to calculate the probability of lexicalized reordering model? I'm sorry for bothering you. Thanks and Best Regards, Zhenxin ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Moses as a service under release 3.0
On May 7, 2015 2:42:53 AM GMT+07:00, Barry Haddow bhad...@inf.ed.ac.uk wrote: Hi Kwame The --server switch was introduced post v3, so you would have to use current master if you want to go down that route. However you will still hit the xmlrpc issues. made soft links from the so.8 libs to so.7 so that mosesserver can load the library (figuring that the functionality/interface was the same). Probably there's a reason for the version number change from 7 to 8, and you've just hit the incompatibility. Make sure you build and run against the same version of xmlrpc-c - we're using 1.16.33 which is probably old, but works. cheers - Barry On 06/05/15 19:59, kwame porter robinson wrote: Hi, I am attempting to use the binary release 3 moses as a service. Recent discussion indicates that mosesserver is depreciated and that moses --server should be used instead. However, running 'moses --server --server-port 1234 -f phrase-model/moses.ini' complains about illegal switches. So I looked at mosseserver. It complains about libxmlrpc libraries (*.so.7) not being found. So I installed the libxmlrpc-c++ package (1.33.06) and made soft links from the so.8 libs to so.7 so that mosesserver can load the library (figuring that the functionality/interface was the same). However, moseserver seg faults, when called with 'mosesserver --server-port 1234 -f phrase-model/moses.ini'. Does anyone know where/how to get libxmlrpc libraries required by moses release 3.0 or otherwise make moses as a service w/o having to re-build it? I'm using Ubunut 14.04. If all else fails I can use moses' stdio but I was hoping to get this working. Thank you for your time. - Kwame ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support Also, C++ ABIs are by nature much more brittle than what you get with C libraries. It only takes a very small change, sometimes even just a compiler upgrade, to break binary compatibility. Jeroen___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
