rmannibucau opened a new pull request, #8304:
URL: https://github.com/apache/hop/pull/8304
Idea is to try to bypass Apache POI for formula evaluation.
Since it can be a crazy huge task the philosophy is "can i optimize this
formula, if yes do it", there is also a system property to disable/enable this
feature until we are sure we don't need it anymore.
== Microbenchmark
* JMH 1.37, average-time mode (`us/op`, lower is better).
* Java 21.0.12 Temurin, single thread, `-Xmx2g`.
* Each case measured with 1 warmup iteration (30 s) and 2 measurement
iterations (30 s).
* Baseline (POI) run with
`-Dorg.apache.hop.pipeline.transforms.formula.fast.FastFormulaCompiler.enabled=false`.
* Each row evaluates a full `Formula.processRow()` (argument resolution +
evaluation + output)
The scenarios below use formulas that the fast evaluator supports (so a fast
vs POI difference is observable).
[cols="4a,1,1,1,1,1",options="header"]
|===
|Scenario |Formula |Fast (us/op) |POI (us/op) |Speedup x |Faster %
|CONCAT |`"CREATE TABLE " & [tableName] & "(FILENAME VARCHAR2(255), ROWNUM
NUMBER, " & [sqlContent] & " )" & IF([compress]="Y"," COMPRESS","")` |0.497
|8.439 |17.0 |94.1
|duration |`[heures]*60 + [minutes]` |0.207 |3.927 |19.0 |94.7
|mergeClause |`"merge into " & [outputTable] & " INPUT_TABLE using " &
[inputTable] & " STAGING_TABLE ON (" & [joinClause] & " ) WHEN NOT MATCHED THEN
INSERT ( " & [inputCols] & " ) VALUES ( STAGING_TABLE." & [outputCols] & "),"`
|0.990 |6.618 |6.7 |85.0
|nestedIf
|`IF(OR([mycolumn]="Œ",[mycolumn]="1",[mycolumn]="3",[mycolumn]="4"),"1",IF([mycolumn]="5","0",[mycolumn]))`
|0.868 |5.162 |5.9 |83.2
|renamePath |`[abcd] & [version] & [fghij] & [hhmmss] & "." & [ext]` |0.507
|4.226 |8.3 |88.0
|sqlWhere |`IF([somecolumn] <> "N"," AND WHAT = '1111111100000000'","") &
IF([othercolumn] <> "N"," AND EVER != '1'","") & IF([abcdef] <> "null"," AND
DUMMY IN (" & [ghij] & ")","")` |0.591 |6.420 |10.9 |90.8
|===
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] Run `mvn clean install apache-rat:check` to make sure basic checks
pass. A more thorough check will be performed on your pull request
automatically.
- [ ] If you have a group of commits related to the same change, please
squash your commits into one and force push your branch using `git rebase -i`.
- [-] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable.
To make clear that you license your contribution under the [Apache License
Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
you have to acknowledge this by using the following check-box.
- [X] I hereby declare this contribution to be licensed under the [Apache
License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
- [X] In any other case, please file an [Apache Individual Contributor
License Agreement](https://www.apache.org/licenses/icla.pdf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
