Hello,

Not sure what the problem is but , buried  deep in our parser we also use 
Optimaize, previously lang-detect. We load models once, inside a static block, 
and create a new Detector instance for every record we parse. This is very fast.

Regards,
Markus
 
-----Original message-----
> From:Sebastian Nagel <wastl.na...@googlemail.com>
> Sent: Tuesday 24th October 2017 14:11
> To: user@nutch.apache.org
> Subject: Re: Usage of Tika LanguageIdentifier in language-identifier plugin
> 
> Hi Yossi,
> 
> > does not separate the Detector object, which contains the model and should 
> > be reused, from the
> > text writer object, which should be request specific.
> 
> But shouldn't a call of reset() make it ready for re-use (the Detector object 
> including the writer)?
> 
> But I agree that a reentrant function maybe easier to integrate. Nutch 
> plugins also need to be
> thread-safe, esp. parsers and parse filters if running in a multi-threaded 
> parsing fetcher.
> Without a reentrant function and without a 100% stateless detector, the only 
> way is to use a
> ThreadLocal instance of the detector. At a first glance, the optimaize 
> detecter seems to be stateless.
> 
> > I chose optimaize mainly because Tika did. Using langid instead should be 
> > very simple, but the
> > fact that the project has not seen a single commit in the last 4 years, and 
> > the usage numbers are
> > also quite low, gives me pause...
> 
> Of course, maintenance or community around a project is an important factor. 
> CLD2 is also not really
> maintained, plus the models are fixed, no code available to retrain them.
> 
> > what I have done locally
> 
> In any case, would be great if you would open an issue on Jira and a pull 
> request on github.
> Which way to go may be discussed further.
> 
> Thanks,
> Sebastian
> 
> 
> On 10/24/2017 01:05 PM, Yossi Tamari wrote:
> > Why not LanguageDetector: The API does not separate the Detector object, 
> > which contains the model and should be reused, from the text writer object, 
> > which should be request specific. The same API Object instance contains 
> > references to both. In code terms, both loadModels() and addText() are 
> > non-static members of LanguageDetector.
> > 
> > Developing another language-identifier-optimaize is basically what I have 
> > done locally, but it seems to me having both in the Nutch repository would 
> > just be confusing for users. 99% of the code would also be duplicated (the 
> > relevant code is about 5 lines).
> > 
> > I chose optimaize mainly because Tika did. Using langid instead should be 
> > very simple, but the fact that the project has not seen a single commit in 
> > the last 4 years, and the usage numbers are also quite low, gives me 
> > pause...
> > 
> > 
> >> -----Original Message-----
> >> From: Sebastian Nagel [mailto:wastl.na...@googlemail.com]
> >> Sent: 24 October 2017 13:18
> >> To: user@nutch.apache.org
> >> Subject: Re: Usage of Tika LanguageIdentifier in language-identifier plugin
> >>
> >> Hi Yossi,
> >>
> >> sorry while fast-reading I've thought it's about the old 
> >> LanguageIdentifier.
> >>
> >>> it is not possible to initialize the detector in setConf and then reuse it
> >>
> >> Could explain why? The API/interface should allow to get an instance and 
> >> call
> >> loadModels() or not?
> >>
> >>>>> For my needs, I have modified the plugin to use
> >>>>> com.optimaize.langdetect.LanguageDetector directly, which is what
> >>
> >> Of course, that's also possible. Or just add a plugin language-identifier-
> >> optimaize.
> >>
> >> Btw., I recently had a look on various open source language identifier
> >> implementations would prefer
> >> langid (a port from Python/C) because it's faster and has a better 
> >> precision:
> >>   https://github.com/carrotsearch/langid-java.git
> >>   https://github.com/saffsd/langid.c.git
> >>   https://github.com/saffsd/langid.py.git
> >> Of course, CLD2 (https://github.com/CLD2Owners/cld2.git) is unbeaten (but 
> >> it's
> >> C++).
> >>
> >> Thanks,
> >> Sebastian
> >>
> >> On 10/24/2017 11:46 AM, Yossi Tamari wrote:
> >>> Hi Sebastian,
> >>>
> >>> Please reread the second paragraph of my email .
> >>> In short, it is not possible to initialize the detector in setConf and 
> >>> then reuse it,
> >> and initializing it per call would be extremely slow.
> >>>
> >>>   Yossi.
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Sebastian Nagel [mailto:wastl.na...@googlemail.com]
> >>>> Sent: 24 October 2017 12:41
> >>>> To: user@nutch.apache.org
> >>>> Subject: Re: Usage of Tika LanguageIdentifier in language-identifier 
> >>>> plugin
> >>>>
> >>>> Hi Yossi,
> >>>>
> >>>> why not port it to use
> >>>>
> >>>>
> >> http://tika.apache.org/1.16/api/org/apache/tika/language/detect/LanguageDe
> >>>> tector.html
> >>>>
> >>>> The upgrade to Tika 1.16 is already in progress (NUTCH-2439).
> >>>>
> >>>> Sebastian
> >>>>
> >>>> On 10/24/2017 11:26 AM, Yossi Tamari wrote:
> >>>>> Hi
> >>>>>
> >>>>>
> >>>>>
> >>>>> The language-identifier plugin uses
> >>>>> org.apache.tika.language.LanguageIdentifier for extracting the
> >>>>> language from the document text. There are two issues with that:
> >>>>>
> >>>>> 1.      LanguageIdentifier is deprecated in Tika.
> >>>>> 2.      It does not support CJK language (and I suspect a lot of other
> >>>>> languages -
> >>>>> https://wiki.apache.org/nutch/LanguageIdentifierPlugin#Implemented_Lan
> >>>>> guages _and_their_ISO_636_Codes), and it doesn't even fail gracefully
> >>>>> with them - in my experience Chinese was recognized as Italian.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Since in Tika LanguageIdentifier was superseded by
> >>>>> org.apache.tika.language.detect.LanguageDetector, it seems obvious to
> >>>>> make that change in the plugin as well. However, because the design of
> >>>>> LanguageDetector is terrible, it makes the implementation not
> >>>>> reentrant, meaning the full language model would have to be reloaded
> >>>>> on each call to the detector.
> >>>>>
> >>>>>
> >>>>>
> >>>>> For my needs, I have modified the plugin to use
> >>>>> com.optimaize.langdetect.LanguageDetector directly, which is what
> >>>>> Tika's LanguageDetector uses internally (at least by default). My
> >>>>> question is whether that is a change that should be made to the official
> >> plugin.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>>                Yossi.
> >>>>>
> >>>>>
> >>>
> >>>
> > 
> > 
> 
> 

Reply via email to