hansva opened a new issue, #9307: URL: https://github.com/apache/hop/issues/9307
### What needs to happen? When a pipeline runs from Hop GUI, a row sampler is attached to copy 0 of every transform. It fills the data preview shown behind the canvas data icon. The sampler runs for every row that copy writes. With several copies of a transform, copy 0 therefore runs noticeably slower than the others. With 4 copies of Generate rows and nothing downstream, copy 0 ran at about a third of the speed of copies 1–3. The cost came from three things: - Every row was cloned, even with sample type `First` after the buffer was full, and with `Random` for rows that never entered the sample. - `Last` (the default) added each row to the front of a synchronized `LinkedList` and removed the oldest, allocating a list node per row. - Each row did a map lookup for its buffer. Hop-level samples built the hop key string for every row. ### Changes - **`First`:** stops cloning once the buffer is full, then returns at once on a `volatile` flag. - **`Random`:** uses reservoir sampling with a `long` counter and clones only rows that enter the sample. The old `int` counter overflowed after about 2.1 billion rows. - **`Last`:** keeps the newest rows in a fixed ring, exposed as a list with the newest row first. Readers still lock the list itself (`synchronized (rowBuffer.getBuffer())`), so the preview dialog and painter are unchanged. - **Buffer lookup:** each listener resolves its buffer on its first row. Hop-level samplers are cached per row set, so the hop key is built once per hop. - **Sample maps:** `HopGuiPipelineGraph` now creates them as `ConcurrentHashMap`s. Transform threads add their buffer on their first row, and with a plain `HashMap` two transforms starting at the same moment could write to it concurrently. Rows are still cloned when they're kept. A later transform may change the array it receives in place (Value Mapper does), and the preview must show what the sampled transform wrote. ### Measurements Generate rows, one copy, 30M rows of 5 fields, sampler attached through `PipelineRowSamplerHelper`. Apple M2 Max, best of 3 runs. | Sample type | Before | After | |---|---|---| | None (no sampler) | 1.04 s | 1.04 s | | `First` | 1.80 s | 0.95 s | | `Random` | 1.72 s | 1.05 s | | `Last` | 2.61 s | 1.70 s | `First` and `Random` now cost nothing measurable. Most of the remaining `Last` cost is the per-row `RowMeta.cloneRow`, and most of that is its read lock (about 18 ns per call on this machine against about 6 ns for the copy itself). That is left for a separate change. ### Tests New `PipelineRowSamplerHelperTest`: - `First` keeps the first rows. - `Last` keeps the last rows, newest first, including at exactly the sample size and one row past it. - `Random` keeps every row while there are fewer rows than the sample size; past that it keeps distinct rows that were actually written, including some beyond the first ones. - For each type, kept rows are copies: changing the written array afterwards doesn't change the sample. ### Issue Priority Priority: 2 ### Issue Component Component: Hop Gui -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
