hansva opened a new pull request, #7764:
URL: https://github.com/apache/hop/pull/7764
Pipeline engine performance fixes
Profiled two representative pipelines end to end, benchmarked 13 common
transforms individually, and fixed what the measurements justified. Every
number below is a wall-clock A/B — median of 5 runs, 2M rows, fresh JVM per run
— not a profiler percentage.
1. RowMeta: lock-free reads — median +12.7%
size() and getValueMeta(int) took a ReentrantReadWriteLock read lock on
every call, for every field of every row, on a structure that's immutable once
a pipeline starts. Replaced with a volatile IValueMeta[] snapshot republished
under the write lock after each mutation.
Stream Lookup +25.9%, Select Values +24.5%, Memory Group By +19.9%, Sort
Rows +15.9%, UDJE +14.9%, Switch/Case +12.7%, Value Mapper +12.1%, Calculator
+10.5%, Merge Join +8.3%, Filter +8.2%. No regressions.
All 8 mutations of valueMetaList go through write-locked blocks and
getValueMetaList() returns a defensive copy, so no caller can mutate behind the
snapshot. The copy constructor needed an explicit refresh — it delegates to
this(...) and then fills the list, so without it every clone() published an
empty snapshot.
2. Text File Input: no-op regex per field per row — +37%
containsEscapedEscape = pol.contains(esc + esc) — with no escape character
configured (the default), esc is "" and contains("") is always true. Every
field of every row went through String.replaceAll(Pattern.quote(""), ""): a
freshly compiled \Q\E Pattern plus a full regex pass that changes nothing. 22%
of the text-file pipeline's CPU.
Text File Input in isolation: 17.2s → 10.8s, 116k → 184k rows/s. Full
pipeline ~7% (bottleneck relocates downstream).
3. Batching rowset made usable — median +32.8% (stays opt-in)
BlockingRowSet does a timed offer/poll per row: two lock acquisitions and
two condition signals every row. AbstractQueuedSynchronizer.signalNext was
45–65% of every transform thread profiled.
BlockingBatchingRowSet already fixes this but was unusable — enabled as-is
it regressed Sort Rows by 107% and gave Merge Join an 8.3s outlier among 2.4s
runs. Cause was one word: both queues were constructed fair.
- putArray = new ArrayBlockingQueue<>(BATCHSIZE, true);
- getArray = new ArrayBlockingQueue<>(BATCHSIZE, true);
+ putArray = new ArrayBlockingQueue<>(BATCHSIZE, false);
+ getArray = new ArrayBlockingQueue<>(BATCHSIZE, false);
Median +32.8% across all 13 pipelines, spreads tightened from 25–260% to
5–11%, Sort Rows now 34.6% faster. Left disabled by default — it needs the
integration suite, partitioning and the single-threaded engine before flipping;
this just makes the existing flag usable.
Also: the comment about "stalling on small amounts of rows" is stale —
setDone() already flushes the partial batch (#7742).
4. Select Values: skip clones that can't copy anything — −27% own cost
cloneValueData returns the same reference for
String/Number/Integer/Boolean/BigNumber/Serializable and all non-NORMAL
storage; only NORMAL-storage Date and Binary copy. Implemented the file's
existing TODO — precompute boolean[] needsClone once, skip both the lookup and
the virtual call. 316 → 230 ns/row.
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] Run `mvn clean install apache-rat:check` to make sure basic checks
pass. A more thorough check will be performed on your pull request
automatically.
- [ ] If you have a group of commits related to the same change, please
squash your commits into one and force push your branch using `git rebase -i`.
- [ ] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable.
To make clear that you license your contribution under the [Apache License
Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
you have to acknowledge this by using the following check-box.
- [ ] I hereby declare this contribution to be licensed under the [Apache
License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
- [ ] In any other case, please file an [Apache Individual Contributor
License Agreement](https://www.apache.org/licenses/icla.pdf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
