subject:"\[jira\] \[Commented\] \(JOSHUA\-304\) word\-align.conf alignment template file not compatible with berkeley aligner"

[jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

2016-08-29 Thread Lewis John McGibbney (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15446876#comment-15446876
 ] 

Lewis John McGibbney commented on JOSHUA-304:
-

[~post] np at all. No need for sorry.
I just tested after clean download of third party deps that this works a charm. 
Thanks for looking in to it I really appreciate it.
I am +1 for merge into master and resolve this as fixed [~post]

> word-align.conf alignment template file not compatible with berkeley aligner
> 
>
> Key: JOSHUA-304
> URL: https://issues.apache.org/jira/browse/JOSHUA-304
> Project: Joshua
>  Issue Type: Bug
>  Components: alignment, berkeley, templates
>Affects Versions: 6.0.5
>Reporter: Lewis John McGibbney
>Priority: Blocker
> Fix For: 6.1
>
>
> It takes me quite some time to debug what was going on and why pipeline's 
> were failing when using the berkeley aligner.
> It turns out that the word-align.conf template provided at
> https://github.com/apache/incubator-joshua/blob/master/scripts/training/templates/alignment/word-align.conf
> is not compatible with the berkeley aligner. 
> In particular the following lines are non compatible
> https://github.com/apache/incubator-joshua/blob/master/scripts/training/templates/alignment/word-align.conf#L12-L15
> Evidence of this is provided below
> {code}
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1 HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1, HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1 HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'JOINT JOINT'; valid choices: FORWARD|REVERSE|BOTH_INDEP|JOINT
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Exception in thread "main" java.lang.NumberFormatException: For input string: 
> "5 5"
>   at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>   at java.lang.Integer.parseInt(Integer.java:580)
>   at java.lang.Integer.parseInt(Integer.java:615)
>   at 
> edu.berkeley.nlp.fig.basic.OptInfo.interpretValue(OptionsParser.java:143)
>   at 
> edu.berkeley.nlp.fig.basic.OptInfo.interpretValue(OptionsParser.java:240)
>   at edu.berkeley.nlp.fig.basic.OptInfo.set(OptionsParser.java:294)
>   at 
> edu.berkeley.nlp.fig.basic.OptionsParser.readOptionsFile(OptionsParser.java:555)
>   at 
> edu.berkeley.nlp.fig.basic.OptionsParser.doParse(OptionsParser.java:604)
>   at edu.berkeley.nlp.fig.exec.Execution.init(Execution.java:293)
>   at edu.berkeley.nlp.wordAlignment.Main.main(Main.java:149)
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Cannot create directory: alignments/0
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

2016-08-29 Thread Matt Post (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15446657#comment-15446657
 ] 

Matt Post commented on JOSHUA-304:
--

Sorry, you have to also install the Berkeley aligner jar. See the last two 
lines of download-deps.sh. Then it should work.

> word-align.conf alignment template file not compatible with berkeley aligner
> 
>
> Key: JOSHUA-304
> URL: https://issues.apache.org/jira/browse/JOSHUA-304
> Project: Joshua
>  Issue Type: Bug
>  Components: alignment, berkeley, templates
>Affects Versions: 6.0.5
>Reporter: Lewis John McGibbney
>Priority: Blocker
> Fix For: 6.1
>
>
> It takes me quite some time to debug what was going on and why pipeline's 
> were failing when using the berkeley aligner.
> It turns out that the word-align.conf template provided at
> https://github.com/apache/incubator-joshua/blob/master/scripts/training/templates/alignment/word-align.conf
> is not compatible with the berkeley aligner. 
> In particular the following lines are non compatible
> https://github.com/apache/incubator-joshua/blob/master/scripts/training/templates/alignment/word-align.conf#L12-L15
> Evidence of this is provided below
> {code}
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1 HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1, HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1 HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'JOINT JOINT'; valid choices: FORWARD|REVERSE|BOTH_INDEP|JOINT
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Exception in thread "main" java.lang.NumberFormatException: For input string: 
> "5 5"
>   at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>   at java.lang.Integer.parseInt(Integer.java:580)
>   at java.lang.Integer.parseInt(Integer.java:615)
>   at 
> edu.berkeley.nlp.fig.basic.OptInfo.interpretValue(OptionsParser.java:143)
>   at 
> edu.berkeley.nlp.fig.basic.OptInfo.interpretValue(OptionsParser.java:240)
>   at edu.berkeley.nlp.fig.basic.OptInfo.set(OptionsParser.java:294)
>   at 
> edu.berkeley.nlp.fig.basic.OptionsParser.readOptionsFile(OptionsParser.java:555)
>   at 
> edu.berkeley.nlp.fig.basic.OptionsParser.doParse(OptionsParser.java:604)
>   at edu.berkeley.nlp.fig.exec.Execution.init(Execution.java:293)
>   at edu.berkeley.nlp.wordAlignment.Main.main(Main.java:149)
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Cannot create directory: alignments/0
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

2016-08-29 Thread Lewis John McGibbney (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15446643#comment-15446643
 ] 

Lewis John McGibbney commented on JOSHUA-304:
-

Hi [~post]
What new steps did you actually add?
I've wiped everything that was generated by Joshua. I've rebuilt JOSHUA-304 
branch. I'm getting the following

{code}
$JOSHUA/bin/pipeline.pl --type hiero --rundir 
/usr/local/jpl/xdata/joshua_experiments/fisher_callhome_experiment/0 --readme 
"Baseline Hiero run 0 --lm-gen berkeleylm --lm berkeleylm --aligner berkeley 
JOSHUA-304" --source es --target en --lm-gen berkeleylm --lm berkeleylm 
--aligner berkeley --corpus $SPANISH/corpus/asr/callhome_train --corpus 
$SPANISH/corpus/asr/fisher_train --tune  $SPANISH/corpus/asr/fisher_dev --test  
$SPANISH/corpus/asr/callhome_devtest
...
snip
...
[test-vocab-es] rebuilding...
  
dep=/usr/local/jpl/xdata/joshua_experiments/fisher_callhome_experiment/0/data/test/corpus.es
 [CHANGED]
  
dep=/usr/local/jpl/xdata/joshua_experiments/fisher_callhome_experiment/0/data/test/vocab.es
 [NOT FOUND]
  cmd=cat 
/usr/local/jpl/xdata/joshua_experiments/fisher_callhome_experiment/0/data/test/corpus.es
 | /usr/local/incubator-joshua/scripts/training/build-vocab.pl > 
/usr/local/jpl/xdata/joshua_experiments/fisher_callhome_experiment/0/data/test/vocab.es
  took 0 seconds (0s)
[test-vocab-en] rebuilding...
  
dep=/usr/local/jpl/xdata/joshua_experiments/fisher_callhome_experiment/0/data/test/corpus.en
 [CHANGED]
  
dep=/usr/local/jpl/xdata/joshua_experiments/fisher_callhome_experiment/0/data/test/vocab.en
 [NOT FOUND]
  cmd=cat 
/usr/local/jpl/xdata/joshua_experiments/fisher_callhome_experiment/0/data/test/corpus.en
 | /usr/local/incubator-joshua/scripts/training/build-vocab.pl > 
/usr/local/jpl/xdata/joshua_experiments/fisher_callhome_experiment/0/data/test/vocab.en
  took 0 seconds (0s)
[source-numlines] rebuilding...
  
dep=/usr/local/jpl/xdata/joshua_experiments/fisher_callhome_experiment/0/data/train/corpus.es
 [CHANGED]
  cmd=cat 
/usr/local/jpl/xdata/joshua_experiments/fisher_callhome_experiment/0/data/train/corpus.es
 | wc -l
  took 0 seconds (0s)
[source-numlines] retrieved cached result =>   151810
[berkeley-aligner-chunk-0] rebuilding...
  dep=alignments/0/word-align.conf [CHANGED]
  
dep=/usr/local/jpl/xdata/joshua_experiments/fisher_callhome_experiment/0/data/train/splits/corpus.es.0
 [NOT FOUND]
  
dep=/usr/local/jpl/xdata/joshua_experiments/fisher_callhome_experiment/0/data/train/splits/corpus.en.0
 [NOT FOUND]
  dep=alignments/0/training.align [NOT FOUND]
  cmd=java -d64 -Xmx10g -jar 
/usr/local/incubator-joshua/ext/berkeleyaligner/distribution/berkeleyaligner.jar
 ++alignments/0/word-align.conf
  JOB FAILED (return code 1)
[aligner-combine] rebuilding...
  dep=alignments/0/training.en-es.align [NOT FOUND]
  dep=alignments/training.align [NOT FOUND]
  cmd=cat alignments/0/training.en-es.align > alignments/training.align
  JOB FAILED (return code 1)
cat: alignments/0/training.en-es.align: No such file or directory
{code}

> word-align.conf alignment template file not compatible with berkeley aligner
> 
>
> Key: JOSHUA-304
> URL: https://issues.apache.org/jira/browse/JOSHUA-304
> Project: Joshua
>  Issue Type: Bug
>  Components: alignment, berkeley, templates
>Affects Versions: 6.0.5
>Reporter: Lewis John McGibbney
>Priority: Blocker
> Fix For: 6.1
>
>
> It takes me quite some time to debug what was going on and why pipeline's 
> were failing when using the berkeley aligner.
> It turns out that the word-align.conf template provided at
> https://github.com/apache/incubator-joshua/blob/master/scripts/training/templates/alignment/word-align.conf
> is not compatible with the berkeley aligner. 
> In particular the following lines are non compatible
> https://github.com/apache/incubator-joshua/blob/master/scripts/training/templates/alignment/word-align.conf#L12-L15
> Evidence of this is provided below
> {code}
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1 HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1, HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
>

[jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

2016-08-29 Thread Matt Post (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15446555#comment-15446555
 ] 

Matt Post commented on JOSHUA-304:
--

It's easiest if you just wipe everything, but you could remove just alignments/ 
and data/train/

> word-align.conf alignment template file not compatible with berkeley aligner
> 
>
> Key: JOSHUA-304
> URL: https://issues.apache.org/jira/browse/JOSHUA-304
> Project: Joshua
>  Issue Type: Bug
>  Components: alignment, berkeley, templates
>Affects Versions: 6.0.5
>Reporter: Lewis John McGibbney
>Priority: Blocker
> Fix For: 6.1
>
>
> It takes me quite some time to debug what was going on and why pipeline's 
> were failing when using the berkeley aligner.
> It turns out that the word-align.conf template provided at
> https://github.com/apache/incubator-joshua/blob/master/scripts/training/templates/alignment/word-align.conf
> is not compatible with the berkeley aligner. 
> In particular the following lines are non compatible
> https://github.com/apache/incubator-joshua/blob/master/scripts/training/templates/alignment/word-align.conf#L12-L15
> Evidence of this is provided below
> {code}
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1 HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1, HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1 HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'JOINT JOINT'; valid choices: FORWARD|REVERSE|BOTH_INDEP|JOINT
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Exception in thread "main" java.lang.NumberFormatException: For input string: 
> "5 5"
>   at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>   at java.lang.Integer.parseInt(Integer.java:580)
>   at java.lang.Integer.parseInt(Integer.java:615)
>   at 
> edu.berkeley.nlp.fig.basic.OptInfo.interpretValue(OptionsParser.java:143)
>   at 
> edu.berkeley.nlp.fig.basic.OptInfo.interpretValue(OptionsParser.java:240)
>   at edu.berkeley.nlp.fig.basic.OptInfo.set(OptionsParser.java:294)
>   at 
> edu.berkeley.nlp.fig.basic.OptionsParser.readOptionsFile(OptionsParser.java:555)
>   at 
> edu.berkeley.nlp.fig.basic.OptionsParser.doParse(OptionsParser.java:604)
>   at edu.berkeley.nlp.fig.exec.Execution.init(Execution.java:293)
>   at edu.berkeley.nlp.wordAlignment.Main.main(Main.java:149)
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Cannot create directory: alignments/0
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

2016-08-29 Thread lewis john mcgibbney

Hi Matt do you mean the '.cachepipe'?

On Aug 29, 2016 10:19 AM, "Matt Post (JIRA)"  wrote:

>
> [ https://issues.apache.org/jira/browse/JOSHUA-304?page=
> com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel=15446470#comment-15446470 ]
>
> Matt Post commented on JOSHUA-304:
> --
>
> I emailed days ago but don't see that it posted here. You need to wipe out
> your old run and re-run. I can see that the new versions of the commands
> were not run.
>
> > word-align.conf alignment template file not compatible with berkeley
> aligner
> > 
> 
> >
> > Key: JOSHUA-304
> > URL: https://issues.apache.org/jira/browse/JOSHUA-304
> > Project: Joshua
> >  Issue Type: Bug
> >  Components: alignment, berkeley, templates
> >Affects Versions: 6.0.5
> >Reporter: Lewis John McGibbney
> >Priority: Blocker
> > Fix For: 6.1
> >
> >
> > It takes me quite some time to debug what was going on and why
> pipeline's were failing when using the berkeley aligner.
> > It turns out that the word-align.conf template provided at
> > https://github.com/apache/incubator-joshua/blob/master/
> scripts/training/templates/alignment/word-align.conf
> > is not compatible with the berkeley aligner.
> > In particular the following lines are non compatible
> > https://github.com/apache/incubator-joshua/blob/master/
> scripts/training/templates/alignment/word-align.conf#L12-L15
> > Evidence of this is provided below
> > {code}
> > lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_
> experiment/6/alignments/0/word-align.conf
> > Invalid enum: 'MODEL1 HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|
> NONE
> > lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_
> experiment/6/alignments/0/word-align.conf
> > Invalid enum: 'MODEL1, HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|
> NONE
> > lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_
> experiment/6/alignments/0/word-align.conf
> > Invalid enum: 'MODEL1 HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|
> NONE
> > lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_
> experiment/6/alignments/0/word-align.conf
> > Invalid enum: 'JOINT JOINT'; valid choices: FORWARD|REVERSE|BOTH_INDEP|
> JOINT
> > lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_
> experiment/6/alignments/0/word-align.conf
> > Exception in thread "main" java.lang.NumberFormatException: For input
> string: "5 5"
> >   at java.lang.NumberFormatException.forInputString(
> NumberFormatException.java:65)
> >   at java.lang.Integer.parseInt(Integer.java:580)
> >   at java.lang.Integer.parseInt(Integer.java:615)
> >   at edu.berkeley.nlp.fig.basic.OptInfo.interpretValue(
> OptionsParser.java:143)
> >   at edu.berkeley.nlp.fig.basic.OptInfo.interpretValue(
> OptionsParser.java:240)
> >   at edu.berkeley.nlp.fig.basic.OptInfo.set(OptionsParser.java:294)
> >   at edu.berkeley.nlp.fig.basic.OptionsParser.readOptionsFile(
> OptionsParser.java:555)
> >   at edu.berkeley.nlp.fig.basic.OptionsParser.doParse(
> OptionsParser.java:604)
> >   at edu.berkeley.nlp.fig.exec.Execution.init(Execution.java:293)
> >   at edu.berkeley.nlp.wordAlignment.Main.main(Main.java:149)
> > lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_
> experiment/6/alignments/0/word-align.conf
> > Cannot create directory: alignments/0
> > {code}
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>

[jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

2016-08-29 Thread Matt Post (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15446470#comment-15446470
 ] 

Matt Post commented on JOSHUA-304:
--

I emailed days ago but don't see that it posted here. You need to wipe out your 
old run and re-run. I can see that the new versions of the commands were not 
run.

> word-align.conf alignment template file not compatible with berkeley aligner
> 
>
> Key: JOSHUA-304
> URL: https://issues.apache.org/jira/browse/JOSHUA-304
> Project: Joshua
>  Issue Type: Bug
>  Components: alignment, berkeley, templates
>Affects Versions: 6.0.5
>Reporter: Lewis John McGibbney
>Priority: Blocker
> Fix For: 6.1
>
>
> It takes me quite some time to debug what was going on and why pipeline's 
> were failing when using the berkeley aligner.
> It turns out that the word-align.conf template provided at
> https://github.com/apache/incubator-joshua/blob/master/scripts/training/templates/alignment/word-align.conf
> is not compatible with the berkeley aligner. 
> In particular the following lines are non compatible
> https://github.com/apache/incubator-joshua/blob/master/scripts/training/templates/alignment/word-align.conf#L12-L15
> Evidence of this is provided below
> {code}
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1 HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1, HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1 HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'JOINT JOINT'; valid choices: FORWARD|REVERSE|BOTH_INDEP|JOINT
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Exception in thread "main" java.lang.NumberFormatException: For input string: 
> "5 5"
>   at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>   at java.lang.Integer.parseInt(Integer.java:580)
>   at java.lang.Integer.parseInt(Integer.java:615)
>   at 
> edu.berkeley.nlp.fig.basic.OptInfo.interpretValue(OptionsParser.java:143)
>   at 
> edu.berkeley.nlp.fig.basic.OptInfo.interpretValue(OptionsParser.java:240)
>   at edu.berkeley.nlp.fig.basic.OptInfo.set(OptionsParser.java:294)
>   at 
> edu.berkeley.nlp.fig.basic.OptionsParser.readOptionsFile(OptionsParser.java:555)
>   at 
> edu.berkeley.nlp.fig.basic.OptionsParser.doParse(OptionsParser.java:604)
>   at edu.berkeley.nlp.fig.exec.Execution.init(Execution.java:293)
>   at edu.berkeley.nlp.wordAlignment.Main.main(Main.java:149)
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Cannot create directory: alignments/0
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

2016-08-24 Thread Matt Post

It didn't regenerate. Try wiping out your rundir and starting over. 

matt (from my phone)

> On Aug 24, 2016, at 4:08 PM, Lewis John McGibbney (JIRA)  
> wrote:
> 
> 
>[ 
> https://issues.apache.org/jira/browse/JOSHUA-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435687#comment-15435687
>  ] 
> 
> Lewis John McGibbney commented on JOSHUA-304:
> -
> 
> [~post] unfortunately my local tests are still not coming up with anything 
> fruitful.
> {code}
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua(JOSHUA-304) $ 
> $JOSHUA/bin/pipeline.pl --type hiero --rundir 8 --readme "Baseline Hiero run 
> 8 --lm-gen berkeleylm --lm berkeleylm --aligner berkeley proposed bug fixed 
> in ../../scripts/training/paralign.pl" --source es --target en --lm-gen 
> berkeleylm --lm berkeleylm --aligner berkeley --corpus 
> $SPANISH/corpus/asr/callhome_train --corpus $SPANISH/corpus/asr/fisher_train 
> --tune  $SPANISH/corpus/asr/fisher_dev --test  
> $SPANISH/corpus/asr/callhome_devtest
> [train-copy-and-filter] cached, skipping...
> [train-tokenize-es] cached, skipping...
> [train-tokenize-en] cached, skipping...
> [train-trim] cached, skipping...
> [train-lowercase-es] cached, skipping...
> [train-lowercase-en] cached, skipping...
> [train-vocab-es] cached, skipping...
> [train-vocab-en] cached, skipping...
> [tune-copy-and-filter] cached, skipping...
> [tune-tokenize-es] cached, skipping...
> [tune-tokenize-en.0] cached, skipping...
> [tune-tokenize-en.1] cached, skipping...
> [tune-tokenize-en.2] cached, skipping...
> [tune-tokenize-en.3] cached, skipping...
> [tune-lowercase-es] cached, skipping...
> [tune-lowercase-en.0] cached, skipping...
> [tune-lowercase-en.1] cached, skipping...
> [tune-lowercase-en.2] cached, skipping...
> [tune-lowercase-en.3] cached, skipping...
> [tune-vocab-es] cached, skipping...
> [tune-vocab-en.0] cached, skipping...
> [tune-vocab-en.1] cached, skipping...
> [tune-vocab-en.2] cached, skipping...
> [tune-vocab-en.3] cached, skipping...
> [test-copy-and-filter] cached, skipping...
> [test-tokenize-es] cached, skipping...
> [test-tokenize-en] cached, skipping...
> [test-lowercase-es] cached, skipping...
> [test-lowercase-en] cached, skipping...
> [test-vocab-es] cached, skipping...
> [test-vocab-en] cached, skipping...
> [source-numlines] cached, skipping...
> [source-numlines] retrieved cached result =>   151810
> [berkeley-aligner-chunk-0] rebuilding...
>  dep=alignments/0/word-align.conf [CHANGED]
>  dep=/usr/local/incubator-joshua/8/data/train/splits/corpus.es.0 [NOT FOUND]
>  dep=/usr/local/incubator-joshua/8/data/train/splits/corpus.en.0 [NOT FOUND]
>  dep=alignments/0/training.align [NOT FOUND]
>  cmd=java -d64 -Xmx10g -jar 
> /usr/local/incubator-joshua/ext/berkeleyaligner/distribution/berkeleyaligner.jar
>  ++alignments/0/word-align.conf
>  JOB FAILED (return code 1)
> [aligner-combine] rebuilding...
>  dep=alignments/0/training.en-es.align [NOT FOUND]
>  dep=alignments/training.align [CHANGED]
>  cmd=cat alignments/0/training.en-es.align > alignments/training.align
>  JOB FAILED (return code 1)
> cat: alignments/0/training.en-es.align: No such file or directory
> {code}
> 
>> word-align.conf alignment template file not compatible with berkeley aligner
>> 
>> 
>>Key: JOSHUA-304
>>URL: https://issues.apache.org/jira/browse/JOSHUA-304
>>Project: Joshua
>> Issue Type: Bug
>> Components: alignment, berkeley, templates
>>   Affects Versions: 6.0.5
>>   Reporter: Lewis John McGibbney
>>   Priority: Blocker
>>Fix For: 6.1
>> 
>> 
>> It takes me quite some time to debug what was going on and why pipeline's 
>> were failing when using the berkeley aligner.
>> It turns out that the word-align.conf template provided at
>> https://github.com/apache/incubator-joshua/blob/master/scripts/training/templates/alignment/word-align.conf
>> is not compatible with the berkeley aligner. 
>> In particular the following lines are non compatible
>> https://github.com/apache/incubator-joshua/blob/master/scripts/training/templates/alignment/word-align.conf#L12-L15
>> Evidence of this is provided below
>> {code}
>> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
>> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
>> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
>> Invalid enum: 'MODEL1 HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
>> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
>> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
>> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
>> Invalid enum: 'MODEL1, HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
>>

[jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

2016-08-24 Thread Lewis John McGibbney (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435615#comment-15435615
 ] 

Lewis John McGibbney commented on JOSHUA-304:
-

ACK will do.

> word-align.conf alignment template file not compatible with berkeley aligner
> 
>
> Key: JOSHUA-304
> URL: https://issues.apache.org/jira/browse/JOSHUA-304
> Project: Joshua
>  Issue Type: Bug
>  Components: alignment, berkeley, templates
>Affects Versions: 6.0.5
>Reporter: Lewis John McGibbney
>Priority: Blocker
> Fix For: 6.1
>
>
> It takes me quite some time to debug what was going on and why pipeline's 
> were failing when using the berkeley aligner.
> It turns out that the word-align.conf template provided at
> https://github.com/apache/incubator-joshua/blob/master/scripts/training/templates/alignment/word-align.conf
> is not compatible with the berkeley aligner. 
> In particular the following lines are non compatible
> https://github.com/apache/incubator-joshua/blob/master/scripts/training/templates/alignment/word-align.conf#L12-L15
> Evidence of this is provided below
> {code}
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1 HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1, HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1 HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'JOINT JOINT'; valid choices: FORWARD|REVERSE|BOTH_INDEP|JOINT
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Exception in thread "main" java.lang.NumberFormatException: For input string: 
> "5 5"
>   at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>   at java.lang.Integer.parseInt(Integer.java:580)
>   at java.lang.Integer.parseInt(Integer.java:615)
>   at 
> edu.berkeley.nlp.fig.basic.OptInfo.interpretValue(OptionsParser.java:143)
>   at 
> edu.berkeley.nlp.fig.basic.OptInfo.interpretValue(OptionsParser.java:240)
>   at edu.berkeley.nlp.fig.basic.OptInfo.set(OptionsParser.java:294)
>   at 
> edu.berkeley.nlp.fig.basic.OptionsParser.readOptionsFile(OptionsParser.java:555)
>   at 
> edu.berkeley.nlp.fig.basic.OptionsParser.doParse(OptionsParser.java:604)
>   at edu.berkeley.nlp.fig.exec.Execution.init(Execution.java:293)
>   at edu.berkeley.nlp.wordAlignment.Main.main(Main.java:149)
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Cannot create directory: alignments/0
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

2016-08-24 Thread Matt Post (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435601#comment-15435601
 ] 

Matt Post commented on JOSHUA-304:
--

I just pushed up some changes that should fix this. Give it a look? It's on the 
JOSHUA-309 branch. It passes my tests.

> word-align.conf alignment template file not compatible with berkeley aligner
> 
>
> Key: JOSHUA-304
> URL: https://issues.apache.org/jira/browse/JOSHUA-304
> Project: Joshua
>  Issue Type: Bug
>  Components: alignment, berkeley, templates
>Affects Versions: 6.0.5
>Reporter: Lewis John McGibbney
>Priority: Blocker
> Fix For: 6.1
>
>
> It takes me quite some time to debug what was going on and why pipeline's 
> were failing when using the berkeley aligner.
> It turns out that the word-align.conf template provided at
> https://github.com/apache/incubator-joshua/blob/master/scripts/training/templates/alignment/word-align.conf
> is not compatible with the berkeley aligner. 
> In particular the following lines are non compatible
> https://github.com/apache/incubator-joshua/blob/master/scripts/training/templates/alignment/word-align.conf#L12-L15
> Evidence of this is provided below
> {code}
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1 HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1, HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1 HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'JOINT JOINT'; valid choices: FORWARD|REVERSE|BOTH_INDEP|JOINT
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Exception in thread "main" java.lang.NumberFormatException: For input string: 
> "5 5"
>   at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>   at java.lang.Integer.parseInt(Integer.java:580)
>   at java.lang.Integer.parseInt(Integer.java:615)
>   at 
> edu.berkeley.nlp.fig.basic.OptInfo.interpretValue(OptionsParser.java:143)
>   at 
> edu.berkeley.nlp.fig.basic.OptInfo.interpretValue(OptionsParser.java:240)
>   at edu.berkeley.nlp.fig.basic.OptInfo.set(OptionsParser.java:294)
>   at 
> edu.berkeley.nlp.fig.basic.OptionsParser.readOptionsFile(OptionsParser.java:555)
>   at 
> edu.berkeley.nlp.fig.basic.OptionsParser.doParse(OptionsParser.java:604)
>   at edu.berkeley.nlp.fig.exec.Execution.init(Execution.java:293)
>   at edu.berkeley.nlp.wordAlignment.Main.main(Main.java:149)
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Cannot create directory: alignments/0
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

2016-08-24 Thread Lewis John McGibbney (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435133#comment-15435133
 ] 

Lewis John McGibbney commented on JOSHUA-304:
-

It may help for me to post the options available within the current berkeley 
aligner jar which was built when I installed Joshua
{code}
lmcgibbn@LMC-032857 /usr/local/incubator-joshua(master) $ java -jar 
./lib/berkeleyaligner.jar  -help
Usage:
  log.maxIndLevel<  int> : Maximum indent level. [10]
  log.msPerLine  <  int> : Maximum number of milliseconds 
between consecutive lines of output. [1000]
  log.file   <  str> : File to write log. []
  log.stdout < bool> : Whether to output to the console. 
[true]
  log.note   <  str> : Dummy placeholder for a comment []
  log.forcePrint < bool> : Force printing from logs* [false]
  log.maxPrintErrors <  int> : Maximum number of errors (via 
error()) to print [1]
  EMWordAligner.nullProb <  dbl> : How to assign null-word 
probabilities (=1 means 1/n) [1.0E-6]
  EMWordAligner.usePosteriorDecoding < bool> : Use posterior decoding 
(recommended for best performance). [true]
  EMWordAligner.posteriorDecodingThreshold <  dbl> : Threshold in [0,1] for 
deciding whether an alignment should exist. [0.5]
  EMWordAligner.mergeConsiderNull < bool> : When merging expected sufficient 
statistics, take into account the NULL (fix). [false]
  EMWordAligner.handleUnknownWords < bool> : Don't crash with unknown words 
(better to train on test set). [false]
  EMWordAligner.priorFraction<  dbl> : Fraction of a count to add for links 
in dictionary prior (1 works well). [0.0]
  EMWordAligner.numThreads   <  int> : Number of concurrent threads to use 
during E-step (set to number of processors). [1]
  EMWordAligner.safeConcurrency  < bool> : Safe concurrency (gets rid of 
concurrency warnings at the expense of speed) [false]
  EMWordAligner.evaluateDuringTraining < bool> : Whether to evaluate the model 
after each training iteration (slower, more memory). [false]
  TreeWalkModel.usePushProbabilities < bool> : Separate parameters for moving 
and pushing. [true]
  TreeWalkModel.conditionOnTag   < bool> : Whether to condition distortion on 
the tag types. [true]
  TreeWalkModel.cacheTreePaths   < bool> : Whether to cache paths through trees 
(uses lots of memory; faster). [false]
  Evaluator.searchForThreshold   < bool> : Evaluate using line search [false]
  Evaluator.thresholdIntervals   <  int> : Sets the number of intervals for 
posterior threshold line search [20]
  Evaluator.saveAlignmentObjects < bool> : Save object files for proposed 
alignments (large files) [false]
  Main.trainSources  < str*> : Directories or files containing 
training files. [example/train]
  Main.testSources   < str*> : Directory or file containing testing 
files. [example/test]
  Main.sentences <  int> : Maximum number of the training 
sentences to use [2147483647]
  Main.offsetTrainingSentences   <  int> : Skip this number of the first 
training sentences [0]
  Main.maxTestSentences  <  int> : Maximum number of the test sentences 
to use [2147483647]
  Main.offsetTestSentences   <  int> : Skip this number of the first test 
sentences [0]
  Main.foreignSuffix <  str> : Foreign language file suffix [f]
  Main.englishSuffix <  str> : English language file suffix [e]
  Main.itgTrainTestSplitPoint<  int> : When writing test (ITG) posteriors, 
where to divide train/test data? [0]
  Main.itgInputDir   <  str> : What directory should we dump ITG 
test data to? []
  Main.reverseAlignments < bool> : Reverse test set alignments (i.e., 
foreign to english) [false]
  Main.oneIndexed< bool> : Are alignments one-indexed (default 
== no, 0-indexed) [false]
  Main.lowercaseWords< bool> : Convert all words to lowercase 
[false]
  Main.leaveTrainingOnDisk   < bool> : Don't load and store the training 
set upfront (slower, but less memory) [false]
  Main.saveRejects   < bool> : Save rejected sentence pairs [false]
  Main.forwardModels  : Which word alignment model to use in 
the forward direction. [MODEL1 HMM]
  Main.reverseModels  : Which word alignment model to use in 
the backward direction. [MODEL1 HMM]
  Main.iters < int*> : Number of iterations to run the 
model. [5 5]
  Main.mode   : Whether to train the two models 
jointly or independently. [JOINT JOINT]
  Main.trainingCacheMaxSize  <  int> : Max sentence length for caching the 
HMM trellis (efficiency only). [100]
  Main.loadParamsDir <  str> : Directory to load parameters from. []
  Main.loadLexicalModelOnly  < bool> : When true, the

[jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

2016-08-23 Thread Lewis John McGibbney (JIRA)


[ 
https://issues.apache.org/jira/browse/JOSHUA-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434164#comment-15434164
 ] 

Lewis John McGibbney commented on JOSHUA-304:
-

It should be noted that in order for me to override the exceptions thrown above 
the template ended up looking like the following
{code}
## word-align.conf
## --
## This is an example training script for the Berkeley
## word aligner.  In this configuration it uses two HMM
## alignment models trained jointly and then decoded 
## using the competitive thresholding heuristic.

##
# Training: Defines the training regimen 
##

forwardModels   HMM
reverseModels   HMM
modeJOINT
iters   5

###
# Execution: Controls output and program flow 
###

execDir alignments/0
create
saveParams  false
numThreads  1
msPerLine   1
alignTraining

#
# Language/Data 
#

foreignSuffix   es.0
englishSuffix   en.0

# Choose the training sources, which can either be directories or files that 
list files/directories
trainSources 
/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/data/train/splits/corpus
sentencesMAX
testSources /dev/null
overwriteExecDir true

#
# 1-best output 
#

competitiveThresholding

{code}

> word-align.conf alignment template file not compatible with berkeley aligner
> 
>
> Key: JOSHUA-304
> URL: https://issues.apache.org/jira/browse/JOSHUA-304
> Project: Joshua
>  Issue Type: Bug
>  Components: alignment, berkeley, templates
>Affects Versions: 6.0.5
>Reporter: Lewis John McGibbney
>Priority: Blocker
> Fix For: 6.1
>
>
> It takes me quite some time to debug what was going on and why pipeline's 
> were failing when using the berkeley aligner.
> It turns out that the word-align.conf template provided at
> https://github.com/apache/incubator-joshua/blob/master/scripts/training/templates/alignment/word-align.conf
> is not compatible with the berkeley aligner. 
> In particular the following lines are non compatible
> https://github.com/apache/incubator-joshua/blob/master/scripts/training/templates/alignment/word-align.conf#L12-L15
> Evidence of this is provided below
> {code}
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1 HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1, HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'MODEL1 HMM'; valid choices: MODEL1|MODEL2|HMM|SYNTACTIC|NONE
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Invalid enum: 'JOINT JOINT'; valid choices: FORWARD|REVERSE|BOTH_INDEP|JOINT
> lmcgibbn@LMC-032857 /usr/local/incubator-joshua/lib(master) $ java -d64 
> -Xmx10g -jar /usr/local/incubator-joshua/lib/berkeleyaligner.jar 
> ++/usr/local/incubator-joshua/experiments/fisher_callhome_experiment/6/alignments/0/word-align.conf
> Exception in thread "main" java.lang.NumberFormatException: For input string: 
> "5 5"
>   at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>   at java.lang.Integer.parseInt(Integer.java:580)
>   at java.lang.Integer.parseInt(Integer.java:615)
>   at 
> edu.berkeley.nlp.fig.basic.OptInfo.interpretValue(OptionsParser.java:143)
>   at 
> edu.berkeley.nlp.fig.basic.OptInfo.interpretValue(OptionsParser.java:240)
>   at edu.berkeley.nlp.fig.basic.OptInfo.set(OptionsParser.java:294)
>   at 
> edu.berkeley.nlp.fig.basic.OptionsParser.readOptionsFile(OptionsParser.java:555)
>   at 
> edu.berkeley.nlp.fig.basic.OptionsParser.doParse(OptionsParser.java:604)
>   at

[jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

[jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

[jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

[jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

Re: [jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

[jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

Re: [jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

[jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

[jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

[jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

[jira] [Commented] (JOSHUA-304) word-align.conf alignment template file not compatible with berkeley aligner

11 matches

Site Navigation

Mail list logo

Footer information