[Moses-support] memory problems running moses server
Hi everyone,
It seems like I run into memory problems when using moses server even
though I have plenty of memory available.
I isolated the problem in a little script (attached). I start the server
and send strings for translation iteratively. At a certain point it will
send the request and wait for a reply forever. The number of iterations
decreases, when I increase the length of the string.
It seems as if the already sent requests are "piling up" somewhere in
the server's or client's memory and at a certain point it doesn't have
the capacity to process the new request.
Has anyone gone through anything similar?
Cheers
Sarah
--
Sarah Schulz
University of Stuttgart
Institute for Natural Language Processing (IMS)
Pfaffenwaldring 5B, 70569 Stuttgart
Germany
http://www.ims.uni-stuttgart.de/institut/mitarbeiter/schulzsh/
import random
import xmlrpclib as x
import time
import sys
import subprocess
#initialize random port and start moses server as subprocess
port_uni= str(random.randint(4, 5))
p1 = subprocess.Popen("/mount/arbeitsdaten13/users/schulzsh/tools/Moses/mosesdecoder/bin/mosesserver -f ../static/Moses/decoder_files/SMT_Werther_uni.ini --server-port "+ port_uni,stderr=subprocess.PIPE,bufsize=8000,shell=True,env={"LD_LIBRARY_PATH":"/mount/projekte/sfb-732/inf/users/sarah/tools/xmlrpc-c/lib"})
#wait for it to start sucessfully
elapsed_time=0
start_time = time.time()
err=''
out=''
for line in iter(p1.stderr.readline, ''):
if "Listening on port" in line or elapsed_time > 150:
print "loaded uni1"
break
elapsed_time = time.time() - start_time
if elapsed_time >= 150:
print "SMT Unigram Server Mode doesn't start."
sys.exit()
#connect to ServerProxy
s_unigram = x.ServerProxy("http://localhost:"+port_uni+"/RPC2",allow_none=True,verbose=True)
#send a message to moses server for translation
#note: when message length increases, the bug appears after fewer iterations
for i in range(1000):
trans= s_unigram.translate({"text":"# I c h j k k k # k k k k k k k k k k k #","align":"true"})
print trans
print i
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] z-mert
Hi, I am using z-mert for the first time since I had to implement my own score for tuning. But when I try to run it, I get the following error while parse the param.txt file: Exception in thread "main" java.util.InputMismatchException at java.util.Scanner.throwFor(Scanner.java:864) at java.util.Scanner.next(Scanner.java:1485) at java.util.Scanner.nextDouble(Scanner.java:2413) at MertCore.processParamFile(MertCore.java:1537) at MertCore.initialize(MertCore.java:310) at MertCore.(MertCore.java:239) at ZMERT.main(ZMERT.java:44) My param.txt looks like this: lm_0 ||| 1.0 Opt 0.5 1.5 0.5 1.5 d_0 ||| 1.0 Opt 0.5 1.5 0.5 1.5 tm_0 ||| 0.3 Opt 0.25 0.75 0.25 0.75 tm_1 ||| 0.2 Opt 0.25 0.75 0.25 0.75 tm_2 ||| 0.2 Opt 0.25 0.75 0.25 0.75 tm_3 ||| 0.3 Opt 0.25 0.75 0.25 0.75 w_0 ||| 0.0 Opt -0.5 0.5 -0.5 0.5 normalization = none I was wondering if a type cast to double is missing in the code but before changing the z-mert code, I wanted to make sure I didn't get anything else wrong. Does anybody have experience with that? Cheers, Sarah ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Perplexity KenLM
Hello, I am working with kenlm models and Moses. I have several different models and want to determine which model fits my data best before I train Moses. I know that there are simple command line calls for eg SRILM to calculate perplexity but I couldn't find anything equivalent for kenlm. The only thing I could find is a python function model.score(sentence) but no information about the score itself. Does anybody know more about the score function of kenlm? Or does anybody know another way of calculating perplexity of a kenlm language model with respect to a test corpus? Thanks in advance, Sarah -- Sarah Schulz LT3 Language and Translation Technology Team Department of Translation, Interpreting and Communication Ghent University Groot-Brittanniëlaan 45 9000 Gent sarah.sch...@ugent.be ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Moses in server mode
Hello, I am running Moses in server mode using the python lib xmlrpclib. When I send long messages to Moses, it crashes. I guess it has to do with a limitation of the message size. I could find a possibility to increase the allowed message size for the XML-RPC implementation for C/C++ (which shows me that there is something like a maximum length) but not for the python implementation. That is the error I get: File /usr/lib/python2.7/xmlrpclib.py, line 1224, in __call__ return self.__send(self.__name, args) File /usr/lib/python2.7/xmlrpclib.py, line 1578, in __request verbose=self.__verbose File /usr/lib/python2.7/xmlrpclib.py, line 1264, in request return self.single_request(host, handler, request_body, verbose) File /usr/lib/python2.7/xmlrpclib.py, line 1294, in single_request response = h.getresponse(buffering=True) File /usr/lib/python2.7/httplib.py, line 1030, in getresponse response.begin() File /usr/lib/python2.7/httplib.py, line 407, in begin version, status, reason = self._read_status() File /usr/lib/python2.7/httplib.py, line 371, in _read_status raise BadStatusLine(line) httplib.BadStatusLine: '' Does anybody have experience with that? Cheers, Sarah ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
