hansva opened a new issue, #9433: URL: https://github.com/apache/hop/issues/9433
### What would you like to happen? The Detect language transform always compares each text against all 75 spoken languages that Lingua supports, and loads all 75 models into memory when the transform is created. There is no way to limit the languages or trade accuracy for speed. Most pipelines know which handful of languages their data can contain. Measured with Lingua 1.2.2 on 30,000 short texts (3 to 6 words), one thread: | Setup | Model load | Rows/s | Heap | |---|---|---|---| | All spoken languages (today) | 0.8-2.1 s | 5.4k | 1.4 GB | | All spoken languages, low accuracy | 0.3 s | 9.7k | 95 MB | | 5 languages | 0.3 s | 14.7k | 174 MB | | 5 languages, low accuracy | 0.1 s | 23.1k | 39 MB | ### Proposed changes 1. **Languages to detect.** A new option that lists the languages the transform may answer. Empty means all spoken languages, which is today's behavior, so existing pipelines give the same output. With 5 languages the transform is about 2.7x faster and uses about an eighth of the memory. 2. **Low accuracy mode.** A new checkbox, off by default. Lingua then uses only its trigram models: about 1.8x faster and far less memory, but clearly less accurate on short texts. On the test data it changed the detected language of most rows, so the tooltip and the documentation must say that it is meant for longer texts. 3. **Build the detector in `init()`.** Today it is a field initializer, so every transform copy loads 1.4 GB when it is constructed. Build it in `init()` from the two options above, once per copy. 4. **Keep unloading the models when the transform finishes.** Keeping them loaded would save 1 to 2 seconds on the next run in the same JVM, at the cost of up to 1.4 GB held for as long as Hop GUI or Hop Server runs. Not worth it. The plugin does not get smaller: Lingua ships all models in one 80 MB jar. The options only change what is loaded and compared. ### Dialog and metadata - Both options are `@HopMetadataProperty` fields with `@GuiWidgetElement` annotations, in a group, per the dialog rules. Both are available for metadata injection. - The language list is stored as Lingua's language names (`ENGLISH`, `FRENCH`), the same values the transform writes to `detected_language`. An unknown name fails the transform at start with a clear message. - `check()` warns when exactly one language is listed, since Lingua needs at least two to compare. ### Behavior to document - With a language list, a text in a language that is not listed gets one of the listed languages, or `UNKNOWN` when the two best candidates are too close. This is by design. - Lingua's models are shared by every Detect language transform in the JVM. When one transform finishes and unloads them, another that is still running reloads them on demand: results stay correct, it is only slower for a moment. ### Two existing bugs to fix in the same change Both are read from the code, not reproduced yet; each needs a test that fails first. - A row whose text field is null or blank is dropped: `process()` returns without writing the row. It should pass through with `UNKNOWN` and no confidence. - With "parallelism" on, a row that fails in a worker is logged, but its job counter is never decremented, so the transform waits forever at the end. The failure should stop the transform, or go to the error hop when error handling is on. ### Tests - Default options: same detected language and confidence as before for a fixed set of sentences. - Language list: only listed languages or `UNKNOWN` come out; an unknown name fails at start. - Low accuracy mode: the transform runs and fills both output fields. - The two bug fixes above. ### Issue Priority Priority: 2 ### Issue Component Component: Transforms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
