from:"Andrzej Zydron"

Re: [Moses-support] moses.ini file changes

2014-03-03 Thread Andrzej Zydron

Many thanks Hieu,

Email signature standard

Best Regards,


Andrzej Zydroń

---

CTO

*XTM International Ltd.*

PO Box 2167, Gerrards Cross, SL9 8XF, UK

email: azyd...@xtm-intl.com <mailto:azyd...@xtm-intl.com>

Tel: +44 (0) 1753 480 479

Mob: +44 (0) 7966 477 181

skype: Zydron

www.xtm-intl.com <http://www.xtm-intl.com/>


On 03/03/2014 17:54, Hieu Hoang wrote:
> You can see examples of equivalent old and new ini file format in the 
> regression test, eg.
> https://github.com/moses-smt/moses-regression-tests/blob/master/tests/phrase.basic-surface-only-withirstlm-binlm.oldformat/moses.ini
> https://github.com/moses-smt/moses-regression-tests/blob/master/tests/phrase.basic-surface-only-withirstlm-binlm/moses.ini
>
> and
> https://github.com/moses-smt/moses-regression-tests/blob/master/tests/phrase.show-weights.lex-reorder.oldformat/moses.ini
> https://github.com/moses-smt/moses-regression-tests/blob/master/tests/phrase.show-weights.lex-reorder/moses.ini
> Also, you can see the difference if run an old ini file through
>scripts/training/convert-moses-ini-to-v2.perl
>
>
>
>
> On 3 March 2014 17:03, Andrzej Zydron  <mailto:azyd...@xtm-intl.com>> wrote:
>
> Dear Moses-support,
>
> Can someone point me at any documentation or source module that
> describes the parameters of the moses.ini both 'old' and new '2.1'
> formats, specifically what has become of the ttable-file:
> parameters and
> their new equivalents.
>
> --
> Email signature standard
>
> Thank you in advance,
>
> Andrzej Zydroń
>
> ---
>
> CTO
>
> *XTM International Ltd.*
>
> PO Box 2167, Gerrards Cross, SL9 8XF, UK
>
> email: azyd...@xtm-intl.com <mailto:azyd...@xtm-intl.com>
> <mailto:azyd...@xtm-intl.com <mailto:azyd...@xtm-intl.com>>
>
> Tel: +44 (0) 1753 480 479 
>
> Mob: +44 (0) 7966 477 181 
>
> skype: Zydron
>
> www.xtm-intl.com <http://www.xtm-intl.com> <http://www.xtm-intl.com/>
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> -- 
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] moses.ini file changes

2014-03-03 Thread Andrzej Zydron

Dear Moses-support,

Can someone point me at any documentation or source module that 
describes the parameters of the moses.ini both 'old' and new '2.1' 
formats, specifically what has become of the ttable-file: parameters and 
their new equivalents.

-- 
Email signature standard

Thank you in advance,

Andrzej Zydroń

---

CTO

*XTM International Ltd.*

PO Box 2167, Gerrards Cross, SL9 8XF, UK

email: azyd...@xtm-intl.com 

Tel: +44 (0) 1753 480 479

Mob: +44 (0) 7966 477 181

skype: Zydron

www.xtm-intl.com 


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Moses training performance

2014-02-28 Thread Andrzej Zydron

Hi Barry,

Many thanks for your reply. All the operations on the Xeon are on 
RAMDISK, includind the corpus, training and temp directories. The Xeon 
is totally dedicated to Moses: there is absolutely nothing else running 
on the server. It looks like the i7 3720QM has a better overall 
architecture. It certainly is very fast for a laptop. After a year of 
use of my Mac Book Pro Retina I am still amazed at how good it is.

Tuning though takes twice as long on the Mac: here the extra 
cores/threads on the Xeon come into their own. Admitedly my training set 
is on the small side at 9,300 segments and with much bigger sets the 
small amount of RAM on the Mac Book Pro will certainly slow things down 
a lot, but for development and quick turn around small test data sets it 
certainly is good.

Email signature standard

Best Regards,


Andrzej Zydroń

---

CTO

*XTM International Ltd.*

PO Box 2167, Gerrards Cross, SL9 8XF, UK

email: azyd...@xtm-intl.com <mailto:azyd...@xtm-intl.com>

Tel: +44 (0) 1753 480 479

Mob: +44 (0) 7966 477 181

skype: Zydron

www.xtm-intl.com <http://www.xtm-intl.com/>


On 27/02/2014 20:43, Barry Haddow wrote:
> Hi Andrzej
>
> Whilst mgiza is the time hog in the training, I find it surprising 
> that score takes 10 seconds on the mac and nearly 2 minutes on the 
> xeon. Most of its work is sorting and reading and writing compressed 
> files. I wonder if there is some difference in the sort? Is it using 
> disk on the xeon, and doing everything in ram on the mac? Is it using 
> a temporary directory outside the ram disk - although I think it 
> should put its tmp directory inside the Moses training directory.
>
> cheers - Barry
>
>
> On 27/02/14 20:05, Andrzej Zydron wrote:
>> Hi Hieu, Barry and Marcin,
>>
>> Thank you for your replies and suggestions.
>>
>> The Xeon server is completely dedicated to Moses and is running
>> absolutely nothing else, as opposed to my Mac which is running the usual
>> laptop background stuff like mail etc., as well as having Eclipse doing
>> various Java stuff in the background.
>>
>> I re-ran the tests as Barry advised with only 4 cores and the results 
>> were
>>
>> training: 41:56
>> tuning:   28:16
>> decoding: 01:36
>>
>> Total 01:08:17
>>
>> Therefore 18 minutes slower than the best time on the Xeon with 6 cores
>> (50:01 minutes).
>>
>> Regarding Marcin's suggestion, here are the individual moses-training
>> process' timings:
>>
>> MacBook Pro 4 Threads i7 3720QM 8Gb RAM SSD
>> mkls   17:39:1617:39:4900:00:33
>> snt2cooc.out   17:39:5017:39:5200:00:02
>> mgiza  17:39:5217:45:4700:05:55
>> snt2cooc.out17:45:4717:45:5000:00:03
>> mgiza   17:45:5017:53:2000:07:30
>> giza2bal.pl17:53:2117:53:2300:00:02
>> extract  17:53:2517:53:3100:00:06
>> score 17:53:3117:53:4100:00:10
>> lexical-reordering  17:53:4117:53:4500:00:04
>>
>> Total 00:14:25
>>
>> Using 6 Threads Xeon E5-1650v2 128GB RAM SATA  using 28GB RAMDISK
>> mkls  19:31:0719:35:4300:04:36
>> snt2cooc.out  19:35:4319:36:0500:00:22
>> mgiza 19:36:0519:49:4700:13:42
>> snt2cooc.out  19:49:4719:50:1200:00:25
>> mgiza 19:50:1220:04:0900:13:57
>> giza2bal.pl  20:04:0920:04:3100:00:22
>> extract 20:04:3120:05:1900:00:48
>> score20:05:1920:07:0000:01:41
>> lexical-reordering 20:07:0020:07:0100:00:01
>>
>> Total 00:35:54
>>
>> As you can see the culprit are mkls and mgiza.
>>
>> Email signature standard
>>
>> Best Regards,
>>
>>
>> Andrzej Zydroń
>>
>> ---
>>
>> CTO
>>
>> *XTM International Ltd.*
>>
>> PO Box 2167, Gerrards Cross, SL9 8XF, UK
>>
>> email: azyd...@xtm-intl.com <mailto:azyd...@xtm-intl.com>
>>
>> Tel: +44 (0) 1753 480 479
>>
>> Mob: +44 (0) 7966 477 181
>>
>> skype: Zydron
>>
>> www.xtm-intl.com <http://www.xtm-intl.com/>
>>
>>
>>
>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
>

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Moses training performance

2014-02-27 Thread Andrzej Zydron

Hi Hieu, Barry and Marcin,

Thank you for your replies and suggestions.

The Xeon server is completely dedicated to Moses and is running 
absolutely nothing else, as opposed to my Mac which is running the usual 
laptop background stuff like mail etc., as well as having Eclipse doing 
various Java stuff in the background.

I re-ran the tests as Barry advised with only 4 cores and the results were

training: 41:56
tuning:   28:16
decoding: 01:36

Total 01:08:17

Therefore 18 minutes slower than the best time on the Xeon with 6 cores 
(50:01 minutes).

Regarding Marcin's suggestion, here are the individual moses-training 
process' timings:

MacBook Pro 4 Threads i7 3720QM 8Gb RAM SSD
mkls   17:39:1617:39:4900:00:33
snt2cooc.out   17:39:5017:39:5200:00:02
mgiza  17:39:5217:45:4700:05:55
snt2cooc.out17:45:4717:45:5000:00:03
mgiza   17:45:5017:53:2000:07:30
giza2bal.pl17:53:2117:53:2300:00:02
extract  17:53:2517:53:3100:00:06
score 17:53:3117:53:4100:00:10
lexical-reordering  17:53:4117:53:4500:00:04

Total 00:14:25

Using 6 Threads Xeon E5-1650v2 128GB RAM SATA  using 28GB RAMDISK
mkls  19:31:0719:35:4300:04:36
snt2cooc.out  19:35:4319:36:0500:00:22
mgiza 19:36:0519:49:4700:13:42
snt2cooc.out  19:49:4719:50:1200:00:25
mgiza 19:50:1220:04:0900:13:57
giza2bal.pl  20:04:0920:04:3100:00:22
extract 20:04:3120:05:1900:00:48
score20:05:1920:07:0000:01:41
lexical-reordering 20:07:0020:07:0100:00:01

Total 00:35:54

As you can see the culprit are mkls and mgiza.

Email signature standard

Best Regards,


Andrzej Zydroń

---

CTO

*XTM International Ltd.*

PO Box 2167, Gerrards Cross, SL9 8XF, UK

email: azyd...@xtm-intl.com 

Tel: +44 (0) 1753 480 479

Mob: +44 (0) 7966 477 181

skype: Zydron

www.xtm-intl.com 




___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Moses training performance

2014-02-26 Thread Andrzej Zydron

Title: Email signature standard

  
  
Hi Hieu
  and Barry,
  
  Thank you for your
replies and suggestions.
  
  The Xeon server is completely
dedicated to Moses and is running absolutely nothing else, as
  opposed to
my Mac which
is running the usual laptop background stuff like
mail etc., as well as
having Eclipse doing Java stuff in the
background.

I
  re-ran the tests as Barry advised
with only 4 cores and the results were
  
  training:
41:56
tuning:   28:16
  decoding: 01:36

Total 01:08:17
  
  Therefore 18
  minutes slower than the best time
on the Xeon with 6 cores (50:01
  minutes).


  






  Best Regards,
  
Andrzej Zydroń

  ---
  CTO
  XTM International Ltd.
  PO Box 2167, Gerrards Cross, SL9 8XF, UK
  email: azyd...@xtm-intl.com  


  Tel: +44 (0) 1753 480 479
  Mob: +44 (0) 7966 477 181
  skype: Zydron
  www.xtm-intl.com
   
  
 



   
   
   

  
  On 26/02/2014 17:36, Hieu Hoang wrote:


  Is there any other tasks running on either
machines, especially the Xeon? If there is, the comparison is
invalid. 


Other processes may not only hog cpu, but disk io and
  filesystem cache. 


To get good comparison, you need to run it twice, 1st to
  make sure everything is read from disk & in cache, 2nd to
  do the timing


  
  


  On 25 February 2014 20:37, Andrzej Zydron <azyd...@xtm-intl.com> wrote:
  

  Many thanks
Hieu,

I did specify "-mgiza-cpus 4" for the Mac and
"-mgiza-cpus 12" for the Xeon server. Interestingly
"-mgiza-cpus 10" gave slightly better performance (5
mins). Looking at the io stats mgiza did not appear to
be io bound.


  

  Best
  Regards,
  
Andrzej
  Zydroń

  ---
  CTO

XTM International Ltd.

  PO Box 2167, Gerrards Cross, SL9
  8XF, UK
  email: azyd...@xtm-intl.com  




Tel: +44 (0)
  1753 480 479

  Mob: +44
(0) 7966 477 181
  skype: Zydron
  www.xtm-intl.com
   

 
 
  
  
 
 
 
  


   On 25/02/2014 18:19, Hieu Hoang
wrote:
  

  
  

  
Strange and interesting.
  
  
  I can think of 2 issues:
    1. The number of cores isn't relevant
unless you explicitly ask mgiza & the
various extraction steps to use multiple cores.
    2. It looks like mgiza is the issue
    3. I'm not sure how io-bound mgiza is.
However, in my test with virtual machines,
io-bound processes are slow
       http://www.hanselman.com/blog/VMPerformanceCheck

Re: [Moses-support] Moses training performance

2014-02-25 Thread Andrzej Zydron

Title: Email signature standard

  
  
Many thanks Hieu,
  
  I did specify "-mgiza-cpus 4" for the Mac and "-mgiza-cpus 12" for
  the Xeon server. Interestingly "-mgiza-cpus 10" gave slightly
  better performance (5 mins). Looking at the io stats mgiza did not
  appear to be io bound.
  
  






  Best Regards,
  
Andrzej Zydroń

  ---
  CTO
  XTM International Ltd.
  PO Box 2167, Gerrards Cross, SL9 8XF, UK
  email: azyd...@xtm-intl.com  


  Tel: +44 (0) 1753 480 479
  Mob: +44 (0) 7966 477 181
  skype: Zydron
  www.xtm-intl.com
   
  
 



   
   
   

  
  On 25/02/2014 18:19, Hieu Hoang wrote:


  Strange and interesting.


I can think of 2 issues:
  1. The number of cores isn't relevant unless you
  explicitly ask mgiza & the various extraction steps to use
  multiple cores.
  2. It looks like mgiza is the issue
  3. I'm not sure how io-bound mgiza is. However, in my
  test with virtual machines, io-bound processes are slow
     http://www.hanselman.com/blog/VMPerformanceChecklistBeforeYouComplainThatYourVirtualMachineIsSlow.aspx


This may be the case with ram-disk
  
  
    
On 25 February 2014 18:01, Andrzej
  Zydron <azyd...@xtm-intl.com>
  wrote:
  Dear
Support,

I realize that there may not be a simple answer, but I would
like to
understand why running training on a 9300 segment corpus
takes nearly
three times as long on a 12 core Xeon E5-1650v2 128GB RAM
Running CentOS
6.5, than on my MacBook Pro 4 core i7 3720QM 8GB RAM running
Mavericks.
I am at a loss to explain. On the Xeon server I used a 28GB
RAMDISK to
simulate an SSD to make things more equal. I have used mgiza
throughout.
I have used the same data nad identical settings throughout
on both
machines and I have used the official Moses 2.1 Git
distribution and
compiled and linked on the machine.

These are the timings in minutes for the MacBook Pro 4 core
i7 3720QM
8Gb RAM SSD:

                 Start        End       Time taken
mkls           10:18:50    10:19:23    00:00:33
snt2cooc       10:19:23    10:19:25    00:00:02
mgiza          10:19:25    10:31:58    00:12:33
extract        10:31:58    10:32:04    00:00:06
score          10:32:04    10:32:14    00:00:10
reordering     10:32:14    10:32:17    00:00:03

Total                                  00:13:27

and these for the 12 core Xeon E5-1650v2 128GB RAM using
28GB RAMDISKfor
all the data:

                 Start           End      Time taken
mkls            09:44:24     09:49:00    00:04:36
snt2cooc        09:49:00     09:49:23    00:00:23
mgiza           09:49:23     10:23:32    00:34:09
extract         10:23:32     10:24:20    00:00:48
score           10:24:20     10:26:08    00:01:48
reordering      10:26:08     10:26:20    00:00:12

Total                                    00:41:56

I know that the Mac is a superb machine (the best I have
ever put my
hands on), but I find it difficult to understand why it
should be so
much faster than a state of the art Xeon server for Moses
training.

Email signature standard

Best Regards,


Andrzej Zydroń

---

CTO

*XTM International Ltd.*

PO Box 2167, Gerrards Cross, SL9 8XF, UK

email: azyd...@xtm-intl.com
azyd...@xtm-intl.com>

Tel: +44 (0) 1753 480 479

Mob: +44 (0) 7966 477 181

skype: Zydron

www.xtm-intl.com <http://www.xtm-intl.com/>


___
Moses-support mailing list
Moses-support@mit.ed

[Moses-support] Moses training performance

2014-02-25 Thread Andrzej Zydron

Dear Support,

I realize that there may not be a simple answer, but I would like to 
understand why running training on a 9300 segment corpus takes nearly 
three times as long on a 12 core Xeon E5-1650v2 128GB RAM Running CentOS 
6.5, than on my MacBook Pro 4 core i7 3720QM 8GB RAM running Mavericks. 
I am at a loss to explain. On the Xeon server I used a 28GB RAMDISK to 
simulate an SSD to make things more equal. I have used mgiza throughout. 
I have used the same data nad identical settings throughout on both 
machines and I have used the official Moses 2.1 Git distribution and 
compiled and linked on the machine.

These are the timings in minutes for the MacBook Pro 4 core i7 3720QM 
8Gb RAM SSD:

 StartEnd   Time taken
mkls   10:18:5010:19:2300:00:33
snt2cooc   10:19:2310:19:2500:00:02
mgiza  10:19:2510:31:5800:12:33
extract10:31:5810:32:0400:00:06
score  10:32:0410:32:1400:00:10
reordering 10:32:1410:32:1700:00:03

Total  00:13:27

and these for the 12 core Xeon E5-1650v2 128GB RAM using 28GB RAMDISKfor 
all the data:

 Start   End  Time taken
mkls09:44:24 09:49:0000:04:36
snt2cooc09:49:00 09:49:2300:00:23
mgiza   09:49:23 10:23:3200:34:09
extract 10:23:32 10:24:2000:00:48
score   10:24:20 10:26:0800:01:48
reordering  10:26:08 10:26:2000:00:12

Total00:41:56

I know that the Mac is a superb machine (the best I have ever put my 
hands on), but I find it difficult to understand why it should be so 
much faster than a state of the art Xeon server for Moses training.

Email signature standard

Best Regards,


Andrzej Zydroń

---

CTO

*XTM International Ltd.*

PO Box 2167, Gerrards Cross, SL9 8XF, UK

email: azyd...@xtm-intl.com 

Tel: +44 (0) 1753 480 479

Mob: +44 (0) 7966 477 181

skype: Zydron

www.xtm-intl.com 


___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] Problems with linking mosesdecoder under Mac OS Mavericks

2014-02-06 Thread Andrzej Zydron

Many thanks Hieu!

Email signature standard

Best Regards,


Andrzej Zydroń

---

CTO

*XTM International Ltd.*

PO Box 2167, Gerrards Cross, SL9 8XF, UK

email: azyd...@xtm-intl.com <mailto:azyd...@xtm-intl.com>

Tel: +44 (0) 1753 480 479

Mob: +44 (0) 7966 477 181

skype: Zydron

www.xtm-intl.com <http://www.xtm-intl.com/>


On 06/02/2014 11:30, Hieu Hoang wrote:
> If the executable
>bin/moses
>bin/moses_chart
> has been successfully created, I would ignore the test errors for now.
>
> There's some strangeness on different versions of compilers and OSes. 
> We try to work through them all but they do crop up from time to time
>
>
> On 6 February 2014 10:23, Andrzej Zydron  <mailto:azyd...@xtm-intl.com>> wrote:
>
> Apologies,
>
> The build.log.zip on my post today was wrong. Please find attached
> a gzip'ed versionshowing the 3 test failures.
>
> -- 
> Email signature standard
>
> Best Regards,
>
>
> Andrzej Zydroń
>
> ---
>
> CTO
>
> *XTM International Ltd.*
>
> PO Box 2167, Gerrards Cross, SL9 8XF, UK
>
> email: azyd...@xtm-intl.com <mailto:azyd...@xtm-intl.com>
> <mailto:azyd...@xtm-intl.com <mailto:azyd...@xtm-intl.com>>
>
> Tel: +44 (0) 1753 480 479 
>
> Mob: +44 (0) 7966 477 181 
>
> skype: Zydron
>
> www.xtm-intl.com <http://www.xtm-intl.com> <http://www.xtm-intl.com/>
>
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> -- 
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] moses.ini file changes

[Moses-support] moses.ini file changes

Re: [Moses-support] Moses training performance

Re: [Moses-support] Moses training performance

Re: [Moses-support] Moses training performance

Re: [Moses-support] Moses training performance

[Moses-support] Moses training performance

Re: [Moses-support] Problems with linking mosesdecoder under Mac OS Mavericks

8 matches

Site Navigation

Mail list logo

Footer information