bamaer commented on PR #8948:
URL: https://github.com/apache/hop/pull/8948#issuecomment-6216172681
Thanks @hansva, the latest commit fixes both points. With *Running in
parallel?* off, each copy reads its own file again with
`${Internal.Transform.CopyNr}`. With it on, the files of all copies are read.
What we tested on `affdb0571c`:
- The scenario from my earlier comment, plus a missing file for one copy and
the same file in every copy, run with your `TextFileInputHarness`.
- The new `TextFileLineScanner` fuzzed against the old
`TextFileLineUtil.getLine` (400k random inputs, all three file formats):
identical lines and errors.
- A full build and the integration tests through `run-tests-docker.sh`:
`transforms` 139/139 (including 0121, 0122 and 0124), `smb`/`vfs` 13/13,
`beam_directrunner` 15/15.
What we did not test: parallel reading and read-ahead against real S3, Azure
or GCS (only the container suites in CI), the `azure`, `gcp` and `spark-native`
IT projects, the new dialog tab, Hop Web, and Iceberg reads, which now go
through the new ranged reads on S3, Azure and GCS.
One behaviour change for the release notes: with parallel off, a file in the
list of more than one copy now stops the pipeline. Before, every row came out
once per copy.
LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
