Re: [Moses-support] Performance issue using Moses Server with Moses 3 (probably same as Oren's)
Hi Barry and Martin, Has this issue been fixed in the source code? Should I take thr current master branch and compile it myself to avoid this issue? Thanks. On Friday, July 24, 2015, Barry Haddow bhad...@staffmail.ed.ac.uk wrote: Hi Martin So it looks like it was the abyss connection limit that was causing the problem? I'm not sure why this should be, either it should queue the jobs up or discard them. Probably Moses server should allow users to configure the number of abyss connections directly rather than tying it to the number of Moses threads. cheers - Barry On 24/07/15 14:17, Martin Baumgärtner wrote: Hi Barry, thanks for your quick reply! We're currently testing on SHA e53ad4085942872f1c4ce75cb99afe66137e1e17 (master, from 2015-07-23). This version includes the fix for mosesserver recently mentioned by Hieu in the performance thread. Following my first intuition, I ran the critical experiments after having modified mosesserver.cpp just by simply doubling the given --threads value, but only for abyss server: .maxConn((unsigned int)numThreads*2): 2.) server: --threads: 8 (i.e. abyss: 16) client: shoots 10 threads = about 11 seconds, server shows busy CPU workload = OK 5.) server: --threads: 16 (i.e. abyss: 32) client: shoots 20 threads = about 11 seconds, server shows busy CPU workload = OK Helps. :-) Best wishes, Martin Am 24.07.2015 um 13:26 schrieb Barry Haddow: Hi Martin Thanks for the detailed information. It's a bit strange since command-line Moses uses the same threadpool, and we always overload the threadpool since the entire test set is read in and queued. The server was refactored somewhat recently - which git revision are you using? In the case where Moses takes a long time, and cpu activity is low, it could be either waiting on IO, or waiting on locks. If the former, I don't know why it works fine for command-line Moses, and if the latter then it's odd how it eventually frees itself. Is it possible to run scenario 2, then attach a debugger whilst Moses is in the low-CPU phase to see what it is doing? (You can do this in gdb with info threads) cheers - Barry On 24/07/15 12:07, Martin Baumgärtner wrote: Hi, followed your discussion about mosesserver performance issue with much interest so far. We're having similar behaviour in our perfomance tests with a current github master clone. Both, mosesserver and complete engine run from same local machine, i.e. no NFS. Machine is virtualized CentOS 7 using Hyper-V: lscpu Architecture: x86_64 CPU op-mode(s):32-bit, 64-bit Byte Order:Little Endian CPU(s):8 On-line CPU(s) list: 0-7 Thread(s) per core:1 Core(s) per socket:8 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family:6 Model: 30 Model name:Intel(R) Core(TM) i7 CPU 860 @ 2.80GHz Stepping: 5 CPU MHz: 2667.859 BogoMIPS: 5335.71 Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 8192K Following experiments using an engine with 75000 segments for TM/LM (--minphr-memory, --minlexr-memory): 1.) server: --threads: 8 client: shoots 8 threads = about 12 seconds, server shows full CPU workload = OK 2.) server: --threads: 8 client: shoots 10 threads = about 85 seconds, server shows mostly low activity, full CPU workload only near end of process = NOT OK 3.) server: --threads: 16 client: shoots 10 threads = about 12 seconds, server shows busy CPU workload = OK 4.) server: --threads: 16 client: shoots 16 threads = about 11 seconds, server shows busy CPU workload = OK 5.) server: --threads: 16 client: shoots 20 threads = about 40-60 seconds (depending), server shows mostly low activity, full CPU workload only near end of process = NOT OK We've seen a breakdown in performance always when the client threads exceed the number of threads given by the --threads param. Kind regards, Martin -- *STAR Group* http://www.star-group.net http://www.star-group.net/ *Martin Baumgärtner* STAR Language Technology Solutions GmbH Umberto-Nobile-Straße 19 | 71063 Sindelfingen | Germany Tel. +49 70 31-4 10 92-0 martin.baumgaert...@star-group.net javascript:_e(%7B%7D,'cvml','martin.baumgaert...@star-group.net'); Fax +49 70 31-4 10 92-70 www.star-group.net Geschäftsführer: Oliver Rau, Bernd Barth Handelsregister Stuttgart HRB 245654 | St.-Nr. 56098/11677 ___ Moses-support mailing listmoses-supp...@mit.edu javascript:_e(%7B%7D,'cvml','Moses-support@mit.edu');http://mailman.mit.edu/mailman/listinfo/moses-support The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
Re: [Moses-support] Performance issues using Moses Server with Moses 3
I am using the in-memory mode, using about 50GB of RAM. (No swap issues as far as I can tell.) Could that cause issues? I looked at the commit you linked to, but it doesn't seem to be something configurable beyond the -threads switch. Am I missing something? On Tuesday, July 21, 2015, Barry Haddow bhad...@staffmail.ed.ac.uk wrote: Hi Oren Does your host have 18 threads available? It could also be that xmlrpc-c is limiting the number of connections - this can now be configured: https://github.com/moses-smt/mosesdecoder/commit/b3baade7f022edbcea2969679a40616683f63523 Slowdowns in Moses are often caused by disk access bottlenecks. You can use --minphr-memory and --minlexr-memory to make sure that the phrase and reordering tables are loaded in to memory, rather than being access on-demand. Make sure your host has enough RAM and is not swapping. As I mentioned before there are various ways to make your models smaller ( http://www.statmt.org/moses/?n=Advanced.RuleTables), which can make a big difference to speed depending on your setup. cheers - Barry On 21/07/15 09:30, Oren wrote: Hi Barry, Thanks for the quick response. I added the switch -threads 18 to the command to raise moses server. The slowness issue persists but in a different form. Most requests return right away, even under heavy load, but some requests (about 5%) take far longer - about 15-20 seconds. Perhaps there are other relevant switches? Thanks again. On Monday, July 20, 2015, Barry Haddow bhad...@staffmail.ed.ac.uk javascript:_e(%7B%7D,'cvml','bhad...@staffmail.ed.ac.uk'); wrote: Hi Oren The threading model is different. In v1, the server created a new thread for every request, v3 uses a thread pool. Try increasing the number of threads. Also, make sure you use the compact phrase table and KenLM as they are normally faster, and pre-pruning your phrase table can help, cheers - Barry On 20/07/15 12:01, Oren wrote: Hi all, We are in the process of migrating from Moses 1 to Moses 3. We have noticed a significant slowdown when sending many requests at once to Moses Server. The first request will actually finish about 25% faster that a single request using Moses 1, but as more requests accumulate there is a marked slowdown, until requests take 5 times longer or more. Is this a known issue? Is it specific to Moses Server? What can we do about it? Thanks! Oren. ___ Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing listmoses-supp...@mit.edu javascript:_e(%7B%7D,'cvml','Moses-support@mit.edu');http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Performance issues using Moses Server with Moses 3
Hi Barry, Thanks for the quick response. I added the switch -threads 18 to the command to raise moses server. The slowness issue persists but in a different form. Most requests return right away, even under heavy load, but some requests (about 5%) take far longer - about 15-20 seconds. Perhaps there are other relevant switches? Thanks again. On Monday, July 20, 2015, Barry Haddow bhad...@staffmail.ed.ac.uk wrote: Hi Oren The threading model is different. In v1, the server created a new thread for every request, v3 uses a thread pool. Try increasing the number of threads. Also, make sure you use the compact phrase table and KenLM as they are normally faster, and pre-pruning your phrase table can help, cheers - Barry On 20/07/15 12:01, Oren wrote: Hi all, We are in the process of migrating from Moses 1 to Moses 3. We have noticed a significant slowdown when sending many requests at once to Moses Server. The first request will actually finish about 25% faster that a single request using Moses 1, but as more requests accumulate there is a marked slowdown, until requests take 5 times longer or more. Is this a known issue? Is it specific to Moses Server? What can we do about it? Thanks! Oren. ___ Moses-support mailing listmoses-supp...@mit.edu javascript:_e(%7B%7D,'cvml','Moses-support@mit.edu');http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Performance issues using Moses Server with Moses 3
Hi all, We are in the process of migrating from Moses 1 to Moses 3. We have noticed a significant slowdown when sending many requests at once to Moses Server. The first request will actually finish about 25% faster that a single request using Moses 1, but as more requests accumulate there is a marked slowdown, until requests take 5 times longer or more. Is this a known issue? Is it specific to Moses Server? What can we do about it? Thanks! Oren. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
