Lewis John McGibbney created JOSHUA-318: -------------------------------------------
Summary: scripts/training/run_tuner.py should enable configurable memory usage when invioking joshua-decoder Key: JOSHUA-318 URL: https://issues.apache.org/jira/browse/JOSHUA-318 Project: Joshua Issue Type: Improvement Components: tuner Affects Versions: 6.0.5 Reporter: Lewis John McGibbney Fix For: 6.2 When I run the run_tuner.py script I can easily run into the following {code} [mert-1] rebuilding... dep=/usr/local/joshua_resources/russian_experiments/exp3/data/tune/corpus.en dep=/usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.config [CHANGED] dep=tune/model/grammar.gz.packed/slice_00000.source [CHANGED] dep=/usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.config.final [NOT FOUND] cmd=/usr/local/incubator-joshua/scripts/training/run_tuner.py /usr/local/joshua_resources/russian_experiments/exp3/data/tune/corpus.en /usr/local/joshua_resources/russian_experiments/exp3/data/tune/corpus.ru --tunedir /usr/local/joshua_resources/russian_experiments/exp3/tune --tuner mert --decoder /usr/local/joshua_resources/russian_experiments/exp3/tune/decoder_command --decoder-config /usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.config --decoder-output-file /usr/local/joshua_resources/russian_experiments/exp3/tune/output.nbest --decoder-log-file /usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.log --iterations 10 --metric 'BLEU 4 closest' JOB FAILED (return code 1) Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at org.apache.joshua.decoder.ff.tm.packed.PackedGrammar$PackedSlice.initializeFeatureStructures(PackedGrammar.java:385) at org.apache.joshua.decoder.ff.tm.packed.PackedGrammar$PackedSlice.<init>(PackedGrammar.java:368) at org.apache.joshua.decoder.ff.tm.packed.PackedGrammar.<init>(PackedGrammar.java:153) at org.apache.joshua.decoder.Decoder.initializeTranslationGrammars(Decoder.java:458) at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:389) at org.apache.joshua.decoder.Decoder.<init>(Decoder.java:128) at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69) Traceback (most recent call last): File "/usr/local/incubator-joshua/scripts/training/run_tuner.py", line 553, in <module> main(sys.argv) File "/usr/local/incubator-joshua/scripts/training/run_tuner.py", line 536, in main run_zmert(opts.tunedir, opts.source, opts.target, opts.decoder, opts.decoder_config, opts.decoder_output_file, opts) File "/usr/local/incubator-joshua/scripts/training/run_tuner.py", line 417, in run_zmert opts.metric, opts.iterations or 10) File "/usr/local/incubator-joshua/scripts/training/run_tuner.py", line 399, in setup_configs for feature,weight in get_features(config): File "/usr/local/incubator-joshua/scripts/training/run_tuner.py", line 351, in get_features output = check_output("%s/bin/joshua-decoder -c %s -show-weights -v 0" % (JOSHUA, config_file), shell=True) File "/Users/lmcgibbn/miniconda3/lib/python3.5/subprocess.py", line 626, in check_output **kwargs).stdout File "/Users/lmcgibbn/miniconda3/lib/python3.5/subprocess.py", line 708, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '/usr/local/incubator-joshua/bin/joshua-decoder -c /usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.config -show-weights -v 0' returned non-zero exit status 1 {code} This is because, by default the joshua-decoder script runs with 4g of memory. The run_runer.py script should be flexible enough to continue with the memory allocation provided when a pipe was initially invoked. This value should then be passed to the joshua-decoder script. -- This message was sent by Atlassian JIRA (v6.3.4#6332)