hansva opened a new pull request, #8442: URL: https://github.com/apache/hop/pull/8442
Some performance tweaks to reduce the number of times the gray overlay should show up: # Hop Web: stop re-rendering, re-sending and re-formatting on every request (fixes #8435) ## Symptom In Hop Web, ordinary editing greyed out the whole UI, often several times per interaction: opening the transform selector, opening a transform, closing dialogs, and — worse — while a pipeline was running with a lot of logging. The grey blocker is RAP's *wait hint*: `rwt/remote/Connection.js` shows it whenever a request takes longer than **1000 ms**. So every occurrence is a server request that crossed one second. The work was finding which ones, and why. ## Root causes All measured with a Playwright rig against a 1-CPU container. 1. **RAP has no paint coalescing.** `Canvas.redraw()` and every canvas resize fire `SWT.Paint` synchronously, and on Hop Web a paint renders the whole content to SVG on the server. 2. **RAP's text-size measurement resizes every shell twice per round trip.** After a new dialog, `TextSizeRecalculation` enlarges every shell by 1000 px and restores it, which resizes (and repaints) every open canvas — in every tab. 3. **Nothing was deduplicated on the way to the browser:** identical SVGs re-sent, toolbar JavaScript emitted for unchanged states, the log `Text` rewritten in full per appended line, a periodic forced re-download of the graph SVG. 4. **Per-second GUI refreshes did O(whole log) work:** `Pipeline.getEngineMetrics()` and the metrics grid built a `TransformStatus` per transform per second, whose constructor formats the transform's complete log out of the central buffer — only to read the speed. (Not web-specific; the desktop pays this too.) 5. **No response compression** in the Hop Web image. Concretely, before: opening the context dialog (228 transforms) = 3 requests, **15 SVG renders, 2.2 MB**; opening a 120-transform pipeline = **12 renders, 15 SVG downloads = 8.8 MB**, plus 585 kB re-downloaded every 5 s forever; a 30k-row logging run = **4–17 MB responses every second**; a busy response carried **6,635 toolbar JavaScript snippets (1.6 MB)**. ## Changes ### Context dialog - `rap/.../ContextDialogSvgRendererHandler` — a paint only *schedules* one render per request (`Display.asyncExec`, which RAP runs inside the same request before the response is written); `RemoteObject.set` is only called for properties that changed. State moved from static maps keyed by RAP widget id (per-session ids → cross-user collisions) onto the canvas via `setData`. - `ui/.../ContextDialog` — `filter()` redraws once instead of twice; `dispose()` no longer stores the settings a second time (the `SWT.Close` listener does it); on Hop Web the per-action bitmaps are no longer rasterized (the SVG path never used them). - `ui/.../ContextDialogSvgRenderer` — no pretty-printing (15 % of the payload). ### Graph canvas - `rap/.../CanvasSvgFacadeImpl` — same coalescing: a paint returns the previous result and schedules one `CoalescedRepaint`, which yields once so the graphs' own `asyncExec` redraws (updateGui, Show listener, tab activation) land first, then renders once. Background tabs are not rendered at all; Show/Resize/updateGui repaint them when they come to the front. Plugin graphs using `publishSnapshot` are untouched. - `rap/.../canvas-svg.js` — dropped the forced full re-fetch every 10th poll (the conditional 500 ms poll already recovers any missed update because every publish bumps the revision); fetches are serialised (one in flight, one pending); the hovered transform/action name is bolded client-side. - `ui/.../HopGuiPipelineGraph`, `HopGuiWorkflowGraph` — `handleWebCanvasHover` only sets the tooltip; it no longer re-renders the graph for the bold name. ### Toolbars - `rap/.../SvgLabelFacadeImpl` — remembers the last enabled/shaded state on the label and emits the enable/shade JavaScript only on change. ### Log tab (new incremental console) - `ui/.../ILogConsole`, `LogConsoleFacade` (+ rcp impl returning null, so the desktop keeps its StyledText) — a log view that takes lines incrementally. - `rap/.../WebLogConsole` + `log-console.js` — a `TextComposite` whose host composite mounts a browser-side console; a request carries only the new lines (`append`), both ends keep at most the configured maximum (20,000 when `HOP_MAX_LOG_SIZE_IN_LINES` is 0). The server keeps the same text the desktop widget holds, so copy-all, "show error lines" and the filter rebuild still read `getText()`; the browser reports the selection back (capped at 64 kB) for "copy selected text"; error lines in red, filter highlight via `<mark>`, tail-follow unless the user scrolled up. Font and colour are sent explicitly (RAP puts neither on a composite's element); native selection needs `setSelectable(true)` on the RAP widget. - `ui/.../HopGuiLogBrowser` — batches one refresh into one console call; walks the backlog backwards so only the newest `getMaxLinesPerRefresh()` (5,000) lines are formatted and sent, preceded by "… N earlier log lines are not shown here … the complete log is in the execution information …" (new key `LogBrowser.Console.LinesSkipped`). - `ui/.../HopGuiPipelineLogDelegate`, `HopGuiWorkflowLogDelegate` — create the console on web, `StyledTextComp` fallback if the facade returns null. - `rap/.../HopWeb`, `HopWebEntryPoint` — register and load `log-console.js`. ### Engine - `engine/.../TransformStatus` — new constructor `TransformStatus(component, includeLogText)`; `Pipeline.getEngineMetrics()` and `HopGuiPipelineGridDelegate.updateRowFromBaseTransform` pass `false`. Server status servlets and the remote engine keep the log text, as they need it. ### Deployment - `docker/resources/server.xml` (new) — the effective `tomcat:10` configuration with `compression="on"` and `image/svg+xml` added to the compressible types (the Hop Web icons are served as SVG). Copied into `conf/` by `web.Dockerfile` and staged by the unified builder. UserDatabase realm kept (documented Tomcat BASIC auth relies on it — verified 401/401/200). - `docs/.../hop-web.adoc` — a "Response compression" section for people deploying `hop.war` on their own Tomcat. ## Measurements 1-CPU container unless noted. | Scenario | Before | After | |---|---|---| | Context dialog, first open, 3 requests | 961 + 429 + 438 ms; 15 renders; 2,232 kB | 431 + 43 + 92 ms; 3 renders; 154 kB on the wire | | Context dialog, second open | 574 ms; 1,492 kB | 210 ms; 153 kB | | Typing "table" (5 keystrokes) | 267 ms; 1,437 kB; 2 renders each | 201 ms; 298 kB; 1 render each | | Open 120-transform pipeline | 12 renders; 15 downloads = 8,775 kB | 2 renders; 3 downloads = 45 kB | | Click a transform in it | 12 renders; 14 downloads = 8,190 kB | 3 renders; 3 downloads = 45 kB | | Idle 35 s with it open | 7 downloads = 4,095 kB | 0 | | Hover 10 transform names | ~20 renders / downloads | 0 | | Busy response while running | 6,635 toolbar JS snippets = 1.6 MB | 0 | | 30k-row WriteToLog run, 25 s | 100 MB total; 17 MB max; 1.5 s requests | 7 MB total; 307 kB max; 52 ms | | 300k-row run at Row Level (≈900k lines) | 18.5 MB response; 2.8 s; 2 overlays | 900 kB per refresh; ≤ 254 ms; 0 overlays | | Initial page load on the wire | 1,381 kB | 133 kB | At the extreme 0.25-CPU throttle the context dialog still greys out (8 s cold), but nothing crosses the 1 s line at 1 CPU anymore, and the same throttle went from 25 s to 12 s. ## Verification - Complete `web-tests` suite on an image built from the final code with all optional plugins: **297 tests, 0 failures** (13 skipped = the two hop-server API classes that need a separate hop-server), including all 253 transform dialogs, two concurrent sessions, and running a pipeline. - `ui`, `rap`, engine `Pipeline`/`TransformStatus` unit suites, spotless, RAT. - By hand via Playwright: keyboard navigation / category collapse / fixed-width toggle in the context dialog; tab switching between pipelines, dragging a transform, a workflow; console tail-follow, zoom, clear, filter, highlight, native selection and "copy selected text", the skipped-lines marker; toolbar Run/Stop/Undo states before/during/after a run; Tomcat BASIC auth via `tomcat-users.xml`. ## Notes for reviewers - The coalescing returns the *previous* render result to paints that are coalesced away. The consumers of `areaOwners`/`viewPort` within one request (hover `mouseMove` calls, batched mouse events, viewport drag) were checked; none change geometry and hit-test again in the same request, and the final render happens before the response is written. - `HopGuiLogBrowser` and `ContextDialog` behave exactly as before on the desktop; only the web branches changed. The `TransformStatus` change does affect the desktop — for the better: its own metrics refresh no longer formats whole transform logs once a second. - Pre-existing, left alone: right-clicking in the log also opens the tab folder's "Close Other / Close All" menu (identical with the old `Text`); the Open-file dialog flaked once in the full suite after the sweep left many tabs open (passes alone, unrelated code). - Not done, for later: the execution-perspective viewers paint synchronously through RAP's GC (same pattern applies); explorer `refresh()` is a full tree rebuild; the three 1 s timers per running pipeline could share a tick; `SvgExplorerFileTypeHandler` rasterizes on every paint. ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Run `mvn clean install apache-rat:check` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If you have a group of commits related to the same change, please squash your commits into one and force push your branch using `git rebase -i`. - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. To make clear that you license your contribution under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0) you have to acknowledge this by using the following check-box. - [ ] I hereby declare this contribution to be licensed under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0) - [ ] In any other case, please file an [Apache Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
