[opennlp-sandbox] branch master updated: Compute max target length instead of fixed value in normalizer

[joern]

This is an automated email from the ASF dual-hosted git repository.

joern pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/opennlp-sandbox.git


The following commit(s) were added to refs/heads/master by this push:
     new 1870d00  Compute max target length instead of fixed value in normalizer
1870d00 is described below

commit 1870d00b659eeeb373e7f6bc83f815f8622f9cc1
Author: Jörn Kottmann <jo...@apache.org>
AuthorDate: Fri Feb 1 10:45:58 2019 +0100

    Compute max target length instead of fixed value in normalizer
---
 tf-ner-poc/src/main/python/normalizer/normalizer.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/tf-ner-poc/src/main/python/normalizer/normalizer.py 
b/tf-ner-poc/src/main/python/normalizer/normalizer.py
index bc2c6ac..a0eabe8 100644
--- a/tf-ner-poc/src/main/python/normalizer/normalizer.py
+++ b/tf-ner-poc/src/main/python/normalizer/normalizer.py
@@ -219,8 +219,12 @@ def main():
 
     batch_size = 20
 
-    # TODO: Don't hard code this ...
-    target_max_len = 9
+    target_max_len = -1
+    for token in (target_train + target_dev + target_test):
+        target_max_len = max(target_max_len, len(token))
+
+    # Increase size by one for termination char
+    target_max_len += 1
 
     train_graph = tf.Graph()
     eval_graph = tf.Graph()