hansva opened a new pull request, #8948:
URL: https://github.com/apache/hop/pull/8948
## Parallel and faster reading in Text File Input
### Goals
- Read large text files much faster, especially from cloud object storage
(S3, Azure, Google Cloud Storage), where a single connection is often limited
to a few MB/s.
- Let the copies of a Text File Input transform share the work in a large
file instead of each copy reading every file.
- Keep existing pipelines unchanged by default: one thread per copy, same
rows, same order with one copy.
- Never produce partial or truncated rows: a failed read fails the transform.
### Changes
**Text File Input**
- New **Performance** tab with three settings:
- **Running in parallel?**: the copies share one plan of chunks. Large
files are cut into chunks and each copy takes the next free chunk. A line
belongs to the chunk that holds its first byte, so no line is lost or read
twice.
- **Chunk size**: defaults to 128MB.
- **Read-ahead blocks**: defaults to 0, which means off.
- A note on the tab explains which settings start extra threads.
- Small files are read whole. The split size is `min(chunk size, max(4MB,
total size / copies))`.
- With *Allow breaks in enclosed fields?* the large files are pre-scanned
first to find exact row starts, so a chunk never starts inside an enclosed
field.
- Filenames accepted from a previous transform are collected from all copies
and planned the same way.
- Settings that can't be combined with parallel reading are reported by the
transform check and disabled in the dialog: wrapped lines, paged layout,
footer, row numbers by file, compression, stop-on-filter and repeated fields.
- Faster line reading: a buffered stream plus a new `TextFileLineScanner`
that decodes blocks of characters instead of one character at a time. Parsing
is about 2–3× faster, also with a single copy.
- A read error never returns the part of a line read before it.
**Core VFS**
- Range reads: `HopVfs.getInputStream(file, offset[, length])` and
`RangeReadRandomAccessContent`.
- New `IRangeReadableFile`: a provider reports its read block size and
serves exact byte ranges.
- New `ReadAheadInputStream`:
- keeps N blocks in flight, each one exact ranged request;
- retries a failed block up to 3 attempts in all, waiting 1 s and then 2 s;
- a block that still fails makes the stream fail permanently, and the
transform fails.
- When a chunk is smaller than the storage block size, the transform logs a
warning and uses the block size, so whole blocks aren't downloaded for every
chunk.
**VFS providers**
- Azure, S3 and GCS implement range reads and a configurable **Read block
size** (default 8MB). On Azure and S3 it's a connection setting; on GCS it's a
global option.
- S3: new **Maximum connections** setting (the Apache HTTP client pool,
which defaults to 50). Closing a stream early now aborts the request instead of
downloading the rest.
### Benchmarks
1 GB CSV, compared against the original Hop baseline.
| Storage | Original | 1 copy, new reader | Best parallel | Speedup |
|---|---|---|---|---|
| Azure | 247.3 s | 169.2 s | 13.7 s (10 copies, 8 read-ahead blocks) |
18.1× |
| S3 | 431.9 s | 325.3 s | 16.9 s (10 copies, 8 read-ahead blocks) | 25.6× |
| GCS | 95.7 s* | 41.5 s | 12.1 s (10 copies, 8 read-ahead blocks) | 7.9× |
| Local disk | 89.2 s | 35.1 s | 5.9 s (10 copies, 128MB chunks) | ~15× |
\* The GCS baseline already used 8MB reads.
### Tests
- Unit tests for the chunk plan, chunk streams, record splitter, read-ahead
(retry and permanent failure), line scanner, meta defaults, and parallel runs
(filenames, very long lines, enclosed fields far longer than a chunk).
- Provider read suites for Azurite, MinIO and fake-gcs-server through
Testcontainers. They are skipped without Docker.
- Integration test `0122-text-file-input-parallel-breaks`: line breaks
inside enclosed fields, checked against exact totals.
- About 89% line coverage of the new non-GUI code.
### Docs
- Text File Input: a new Performance tab section covering how the work is
shared, when files are cut, reading from remote storage, retries and failures,
and the settings that can't be combined.
- Azure, S3 and GCS pages: *Read block size* and *Maximum connections*.
### Also included
These are separate from the feature; the macOS test failures came up while
running the full build there.
- `HopGuiKeyHandler`: on macOS, Ctrl+Z in a script editor undid the
pipeline. Ctrl and Cmd are now both treated as belonging to the editor.
- Test fixes so the build passes on macOS:
- the dbt dialog shell image (no window icon is set on macOS);
- the AI review dialog lookup (search the dialog shell, not the active
shell);
- the file watcher timing (macOS polls every 2 s);
- Beam password-masking checks (macOS masks natively, with no echo
character).
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] Run `mvn clean install apache-rat:check` to make sure basic checks
pass. A more thorough check will be performed on your pull request
automatically.
- [ ] If you have a group of commits related to the same change, please
squash your commits into one and force push your branch using `git rebase -i`.
- [ ] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable.
To make clear that you license your contribution under the [Apache License
Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
you have to acknowledge this by using the following check-box.
- [ ] I hereby declare this contribution to be licensed under the [Apache
License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
- [ ] In any other case, please file an [Apache Individual Contributor
License Agreement](https://www.apache.org/licenses/icla.pdf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
