[ https://issues.apache.org/jira/browse/JOSHUA-258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263253#comment-15263253 ]
ASF GitHub Bot commented on JOSHUA-258: --------------------------------------- Github user mjpost commented on the pull request: https://github.com/apache/incubator-joshua/pull/8#issuecomment-215592405 As noted there, this was moved to $JOSHUA/scripts/preparation/tokenize.pl (detokenize.pl), for the 6.1 release (part of an attempt to impose some organization on the scripts under scripts/training, which is getting unwieldy). I'll fix it there. > Add back penn-treebank-(de)tokenizer perl scripts > ------------------------------------------------- > > Key: JOSHUA-258 > URL: https://issues.apache.org/jira/browse/JOSHUA-258 > Project: Joshua > Issue Type: Task > Affects Versions: 6.0.5 > Reporter: Lewis John McGibbney > Assignee: Lewis John McGibbney > Fix For: 6.1 > > > I've been working with the > [joshua_translation_engine|https://github.com/joshua-decoder/joshua_translation_engine] > (which is friggin excellent, we will definately be standing this up on > something more heavyweight in the near future) and recently reported [issue > 15|https://github.com/joshua-decoder/joshua_translation_engine/issues/15] > This issue therefore proposes to add back in penn-treebank-(de)tokenizer perl > scripts which were removed between 6.0.4 and 6.0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)