bamaer commented on PR #8948:
URL: https://github.com/apache/hop/pull/8948#issuecomment-6197959553
Thanks @hansva, the speed-ups are impressive. Text File Input is one of our
most used transforms, so I mainly looked at what changes for existing pipelines.
**Blockers**
- With *Running in parallel?* off, the transform now fails at init when it
runs in more than one copy and doesn't accept filenames from a previous
transform (`TextFileInput.java:89`, `TextFileInputMeta.java:968`). This breaks
existing pipelines that give each copy its own file through
`${Internal.Transform.CopyNr}` in the filename, a pattern we document for other
transforms. Following the error message and switching *Running in parallel?* on
is worse: the plan is built from one copy's file list, so the other copies'
files are silently skipped. With 2 copies and
`part_${Internal.Transform.CopyNr}.csv`, only `part_1.csv` is read, with no
error. Suggested fix: keep the current behaviour when parallel is off (at most
a `check()` warning, no init failure, and drop
`refusesSeveralCopiesWithoutParallelReading`). In parallel mode, either refuse
a filename that resolves differently per copy, or build the plan from the files
of all copies.
Verified: I ran the scenario above with your `TextFileInputHarness` on main
(each copy reads its own file) and on this PR (it fails to start; with parallel
on, one file is skipped). I also fuzzed the new `TextFileLineScanner` against
the old `TextFileLineUtil.getLine` on 400k random inputs, with identical
results. I haven't run the new integration tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
