[
https://issues.apache.org/jira/browse/OPENNLP-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Wiesner closed OPENNLP-1556.
-----------------------------------
Resolution: Fixed
> Improve speed of checksum computation in TwoPassDataIndexer
> -----------------------------------------------------------
>
> Key: OPENNLP-1556
> URL: https://issues.apache.org/jira/browse/OPENNLP-1556
> Project: OpenNLP
> Issue Type: Improvement
> Components: Machine Learning
> Affects Versions: 1.9.0, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.3.1, 2.3.2, 2.3.3
> Reporter: Martin Wiesner
> Assignee: Martin Wiesner
> Priority: Major
> Fix For: 2.3.4
>
>
> For training ML models, all observations (Events) are indexed via
> {{TwoPassDataIndexer#index(ObjectStream<Event> eventStream)}}.
> When #index(..) is run, a tmp file is written and read in again. For the
> purpose of checksum validation, instances of HashSumEventStream are used to
> validate the content processed.
> Based on a rather slow toString() implementation in Event, a cryptographic
> (MD5) message digest is computed. This, however, is much slower than simply
> computing a checksum (such as a CRC32c value) for both directions
> (write/read). The (slowing) effect is more problematic when larger training
> corpora are (pre-)processed, that is, indexed in advance.
> Aims:
> - Speedup the (IO-bound) indexing part prior to the actual CPU-bound training
> phase.
> - Switch from MD5 to CRC32c, as there is *no* need for a cryptographic hash
> function here; it's simply a checksum that is required to decide whether all
> bytes written are the same bytes that are read.
> - Remove the untested class HashSumEventStream which is just a wrapper for
> calling a slow toString() in Event to get some bytes to use for the
> computation of a checksum / md.
> - Provide a replacement for HashSumEventStream, e.g. ChecksumEventStream that
> makes use of the faster CRC32c checksum computation, avoiding cryptographic
> hash functions such as MD5.
> - Make sure all existing tests hold.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
