mattcasters commented on code in PR #9434:
URL: https://github.com/apache/hop/pull/9434#discussion_r4560713275
##########
plugins/transforms/detectlanguage/src/main/java/org/apache/hop/pipeline/transforms/language/DetectLanguage.java:
##########
@@ -56,31 +57,89 @@ public DetectLanguage(
}
public static final int OVER_ALLOCATE_SIZE = 10;
+ private static final String ERROR_CODE = "DetectLanguage001";
+
private final int cores = Runtime.getRuntime().availableProcessors();
private final int maxJobs = cores * 100;
private final ExecutorService executor = commonPool();
private final AtomicInteger jobs = new AtomicInteger(0);
+ /** The first failure of a parallel job when there is no error handling to
send the row to. */
+ private final AtomicReference<Throwable> asyncFailure = new
AtomicReference<>();
+
private final Object lock = new Object();
- private final LanguageDetector detector =
- fromAllSpokenLanguages().withPreloadedLanguageModels().build();
+ /** The outcome of one detection: the language name and its confidence, null
when unknown. */
+ record Detection(String language, Double confidence) {}
+
+ /**
+ * The Lingua builder for the configured options.
+ *
+ * @param languages the languages to choose from; empty means all spoken
languages
+ * @param lowAccuracy use only the trigram models
+ */
+ static LanguageDetectorBuilder createBuilder(List<Language> languages,
boolean lowAccuracy) {
+ LanguageDetectorBuilder builder =
+ languages.isEmpty()
+ ? LanguageDetectorBuilder.fromAllSpokenLanguages()
+ : LanguageDetectorBuilder.fromLanguages(languages.toArray(new
Language[0]));
+ if (lowAccuracy) {
+ builder = builder.withLowAccuracyMode();
+ }
+ return builder.withPreloadedLanguageModels();
+ }
+
+ @Override
+ public boolean init() {
+ if (!super.init()) {
+ return false;
+ }
+ try {
+ data.detector = createBuilder(meta.resolveLanguages(this),
meta.isLowAccuracy()).build();
+ return true;
+ } catch (Exception e) {
+ logError(BaseMessages.getString(PKG,
"DetectLanguage.Error.DetectorNotBuilt"), e);
+ return false;
+ }
+ }
+
+ @Override
+ public void dispose() {
+ // Lingua keeps its models in static maps: release what this transform
asked for. Another
+ // Detect language transform that is still running loads them again on
demand.
+ if (data.detector != null) {
+ data.detector.unloadLanguageModels();
Review Comment:
This clears Lingua's static model maps for these languages, so the next run
(or the next child pipeline run of an executor) preloads everything again in
`init()`, and another Detect language transform running at the same time loses
them too. I'd rather not unload by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]