hansva opened a new issue, #9451:
URL: https://github.com/apache/hop/issues/9451

   ### Apache Hop version?
   
   2.20.0
   
   ### Java version?
   
   21
   
   ### Operating system
   
   Other
   
   ### What happened?
   
   Two bugs and a performance problem in the HTML to text transform 
(`plugins/transforms/html2text`).
   
   **Bugs**
   
   1. **Rows with empty HTML are dropped.** When the HTML field is null or 
blank, `process()` returns without writing the row, so the row disappears from 
the stream. It should pass on with an empty text.
   2. **Parallel mode can wait forever.** With "Use parallelism" on, a row that 
fails in a worker is logged, but its job is never counted down. At the end of 
the input the transform waits for a count that never reaches zero. The failure 
should stop the transform, or send the row to the error hop when error handling 
is on.
   
   **Performance**
   
   3. **A new parser for every row.** `Jsoup.parseBodyFragment(html)` builds a 
new parser and tree builder for each row. In clean mode the safelist and the 
cleaner are rebuilt for each row as well. Parsing is almost all of what this 
transform does, so that set-up is a visible share of every row.
   
   ### Proposed changes
   
   - **Empty HTML:** write the row with an empty text in the output field.
   - **Parallel mode:** always count a job down when it ends. On a failure, 
send the row to the error hop when error handling is on; otherwise stop the 
transform with the error. Count a job before it is handed to the pool, not 
after.
   - **One parser:** keep one jsoup `Parser` per transform copy and parse each 
fragment with it. A `Parser` takes a lock for every parse, so in parallel mode 
give each worker thread its own, or the workers queue up behind one parser.
   - **One safelist and cleaner:** in clean mode, build them once from the 
selected safelist.
   - **Error handling:** the conversion itself is not covered by the error hop 
today, only the row-level log line is. Put the conversion inside it.
   
   The output text does not change. Reusing the parser gives the same result as 
today for all three modes: whole text, normalised text and clean.
   
   Not part of this ticket: in clean mode the HTML is parsed, written back to 
HTML and parsed again. Cleaning the first document directly would be much 
faster, but it gives different output, so it is left as it is.
   
   ### Behavior that changes
   
   - A pipeline whose input has rows with empty HTML now gets those rows in the 
output. Before, they were silently removed.
   - A failing row in parallel mode now fails the pipeline or goes to the error 
hop. Before, the pipeline hung.
   
   ### Measured gain
   
   jsoup 1.23.2 on 200,000 small HTML documents, new parser per row against one 
reused parser:
   
   | Mode | Before | After | Gain |
   |---|---|---|---|
   | Whole text | 237k rows/s | 287k rows/s | 1.21x |
   | Normalised text | 239k rows/s | 275k rows/s | 1.15x |
   | Clean, basic safelist | 66k-77k rows/s | 83k-90k rows/s | 1.18-1.25x |
   
   Results were identical for every document in all three modes. The transform 
itself was not benchmarked yet; parsing is close to its whole cost, so about 
1.2x is expected on benchmark B150.
   
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: Transforms


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to