Hi Oren

You can fit a lot of model in 50G RAM. It's worth looking at the compact phrase and reordering models, pruning options, and kenlm quantisation, and you might fit the models on one machine.

As to the slowdown, a delay of 20s or so suggests that it's waiting on I/O, or perhaps xmlrpc-c is queueing up requests or connections.

cheers - Barry

On 23/07/15 15:06, Oren wrote:
More details...

Our NFS is mounted at /mnt/storage .

The command to run moses server is:

/mnt/storage/Common/mosesdecoder3/bin/mosesserver -f /mnt/storage/Common/models/translation_models/moses3_model/mert-work/moses.ini mark-unknown -xml-input -threads 18 exclusive

Here is the complete moses.ini (excluding comments):

[input-factors]
0

[mapping]
0 T 0

[distortion-limit]
6

[feature]
UnknownWordPenalty
WordPenalty
PhrasePenalty
PhraseDictionaryMemory name=TranslationModel0 num-features=4 path=/mnt/storage/Common/models/translation_models/moses3_model/train/model/phrase-table.gz input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/mnt/storage/Common/models/translation_models/moses3_model/train/model/reordering-table.wbe-msd-bidirectional-fe.gz
Distortion
KENLM lazyken=0 name=LM0 path=/mnt/storage/Common/models/language_models/langmod.5gram.blmorder=5

[threads]
6

[weight]

LexicalReordering0= 0.0268311 -0.0146878 0.0261305 0.0380759 0.0118265 0.07479
Distortion0= 0.074665
LM0= 0.0972206
WordPenalty0= -0.18469
PhrasePenalty0= -0.212528
TranslationModel0= 0.0184105 0.091358 0.112618 0.0161682
UnknownWordPenalty0= 1

On Wednesday, July 22, 2015, Oren <mooshif...@gmail.com <mailto:mooshif...@gmail.com>> wrote:

    Yes, we use a lot of RAM in our setup. But the improved response
    time justifies it.

    Our language is on a nfs, bit we've been working this way with
    moses 1 for some time with no problems (ten different machines
    using the same language model file over nfs). Same for the
    reordering model. Neither of them is in memory. Raising these
    models into memory will require raising our already excessive RAM
    requirements...

    Thanks again for the help.

    On Wednesday, July 22, 2015, Barry Haddow
    <bhad...@staffmail.ed.ac.uk
    <javascript:_e(%7B%7D,'cvml','bhad...@staffmail.ed.ac.uk');>> wrote:

        Hi Oren

        I'm not aware of any threading problems with
        PhraseDictionaryMemory, but not many people use it since it
        takes up too much memory. Moses command line runs
        multi-threaded using a thread pool.

        Is your language model on a local file system? Running it over
        nfs can be bad. What about your reordering model? Are you
        using the compact or the memory version?

        cheers - Barry

        On 22/07/15 15:09, Oren wrote:
        We have no swapping issues. I was asking if the use of
        an in-memory translation model might cause multithreading
        problems.


        I'm not sure how to replicate the problem on cmd moses, since
        it's purely a multithreading issue. Ca you run cmd moses
        multithreaded?

        I can't attach the complete moses.ini because it's on a
        separate network... But I copied below the stuff that looked
        relevant. I also tried to change the [threads] setting to 18,
        with no apparent effect.

        [input-factors]
        0

        [mapping]
        0 T 0

        [distortion-limit]
        6

        [feature]
        UnknownWordPenalty
        WordPenalty
        PhrasePenalty
        PhraseDictionaryMemory name=TranslationModel0 num-features=4
        path=<path>/phrase-table.gz input-factor=0 output-factor=0
        LexicalReordering <parameters>
        Distortion
        KENLM lazyken=0 name=LM0 path=<path> order=5

        [threads]
        6

        [weight]

        <weight parameters>

        On Tuesday, July 21, 2015, Hieu Hoang <hieuho...@gmail.com>
        wrote:

            is it possible you can make your moses.ini file available
            for us to see?

            do you know if the same problem occurs if you use the
            command line moses, rather than mosesserver?


            Hieu Hoang
            Researcher
            New York University, Abu Dhabi
            http://www.hoang.co.uk/hieu

            On 21 July 2015 at 18:07, Barry Haddow
            <bhad...@staffmail.ed.ac.uk> wrote:


                On 21/07/15 14:51, Oren wrote:
                I am using the in-memory mode, using about 50GB of
                RAM. (No swap issues as far as I can tell.) Could
                that cause issues?

                Yes, swapping would definitely cause issues - was
                that your question?



                I looked at the commit you linked to, but it doesn't
                seem to be something configurable beyond the
                -threads switch. Am I missing something?

                The commit enables you to set the maximum number of
                connections to be the same as the maximum number of
                threads.


                On Tuesday, July 21, 2015, Barry Haddow
                <bhad...@staffmail.ed.ac.uk> wrote:

                    Hi Oren

                    Does your host have 18 threads available? It
                    could also be that xmlrpc-c is limiting the
                    number of connections - this can now be configured:
                    
https://github.com/moses-smt/mosesdecoder/commit/b3baade7f022edbcea2969679a40616683f63523

                    Slowdowns in Moses are often caused by disk
                    access bottlenecks. You can use --minphr-memory
                    and --minlexr-memory to make sure that the
                    phrase and reordering tables are loaded in to
                    memory, rather than being access on-demand. Make
                    sure your host has enough RAM and is not
                    swapping. As I mentioned before there are
                    various ways to make your models smaller
                    (http://www.statmt.org/moses/?n=Advanced.RuleTables),
                    which can make a big difference to speed
                    depending on your setup.

                    cheers - Barry

                    On 21/07/15 09:30, Oren wrote:
                    Hi Barry,

                    Thanks for the quick response.

                    I added the switch "-threads 18" to the command
                    to raise moses server. The slowness issue
                    persists but in a different form. Most requests
                    return right away, even under heavy load, but
                    some requests (about 5%) take far longer -
                    about 15-20seconds.

                    Perhaps there are other relevant switches?

                    Thanks again.

                    On Monday, July 20, 2015, Barry Haddow
                    <bhad...@staffmail.ed.ac.uk> wrote:

                        Hi Oren

                        The threading model is different. In v1,
                        the server created a new thread for every
                        request, v3 uses a thread pool. Try
                        increasing the number of threads.

                        Also, make sure you use the compact phrase
                        table and KenLM as they are normally
                        faster, and pre-pruning your phrase table
                        can help,

                        cheers - Barry

                        On 20/07/15 12:01, Oren wrote:
                        Hi all,

                        We are in the process of migrating from
                        Moses 1 to Moses 3. We have noticed a
                        significant slowdown when sending many
                        requests at once to Moses Server. The
                        first request will actually finish about
                        25% faster that a single request using
                        Moses 1, but as more requests accumulate
                        there is a marked slowdown, until requests
                        take 5 times longer or more.

                        Is this a known issue? Is it specific to
                        Moses Server? What can we do about it?

                        Thanks!

                        Oren.


                        _______________________________________________
                        Moses-support mailing list
                        Moses-support@mit.edu
                        http://mailman.mit.edu/mailman/listinfo/moses-support



                    _______________________________________________
                    Moses-support mailing list
                    Moses-support@mit.edu
                    http://mailman.mit.edu/mailman/listinfo/moses-support



                The University of Edinburgh is a charitable body,
                registered in
                Scotland, with registration number SC005336.

                _______________________________________________
                Moses-support mailing list
                Moses-support@mit.edu
                http://mailman.mit.edu/mailman/listinfo/moses-support




The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Reply via email to