Re: Lexical analysis tools for German language data
On Thu, Apr 12, 2012 at 03:46:56PM +, Michael Ludwig wrote: Von: Walter Underwood German noun decompounding is a little more complicated than it might seem. There can be transformations or inflections, like the s in Weinachtsbaum (Weinachten/Baum). I remember from my linguistics studies that the terminus technicus for these is Fugenmorphem (interstitial or joint morpheme) [...] IANAL (I am not a linguist -- pun intended ;) but I've always read that as a genitive. Any pointers? Regards -- Tomás Zerolo Axel Springer AG Axel Springer media Systems BILD Produktionssysteme Axel-Springer-Straße 65 10888 Berlin Tel.: +49 (30) 2591-72875 tomas.zer...@axelspringer.de www.axelspringer.de Axel Springer AG, Sitz Berlin, Amtsgericht Charlottenburg, HRB 4998 Vorsitzender des Aufsichtsrats: Dr. Giuseppe Vita Vorstand: Dr. Mathias Döpfner (Vorsitzender) Jan Bayer, Ralph Büchi, Lothar Lanz, Dr. Andreas Wiele
AW: Lexical analysis tools for German language data
Von: Tomas Zerolo There can be transformations or inflections, like the s in Weinachtsbaum (Weinachten/Baum). I remember from my linguistics studies that the terminus technicus for these is Fugenmorphem (interstitial or joint morpheme) [...] IANAL (I am not a linguist -- pun intended ;) but I've always read that as a genitive. Any pointers? Admittedly, that's what you'd think, and despite linguistics telling me otherwise I'd maintain there's some truth in it. For this case, however, consider: die Weihnacht declines like die Nacht, so: nom. die Weihnacht gen. der Weihnacht dat. der Weihnacht akk. die Weihnacht As you can see, there's no s to be found anywhere, not even in the genitive. But my gut feeling, like yours, is that this should indicate genitive, and I would make a point of well-argued gut feeling being at least as relevant as formalist analysis. Michael
Lexical analysis tools for German language data
Given an input of Windjacke (probably wind jacket in English), I'd like the code that prepares the data for the index (tokenizer etc) to understand that this is a Jacke (jacket) so that a query for Jacke would include the Windjacke document in its result set. It appears to me that such an analysis requires a dictionary-backed approach, which doesn't have to be perfect at all; a list of the most common 2000 words would probably do the job and fulfil a criterion of reasonable usefulness. Do you know of any implementation techniques or working implementations to do this kind of lexical analysis for German language data? (Or other languages, for that matter?) What are they, where can I find them? I'm sure there is something out (commercial or free) because I've seen lots of engines grokking German and the way it builds words. Failing that, what are the proper terms do refer to these techniques so you can search more successfully? Michael
AW: Lexical analysis tools for German language data
Given an input of Windjacke (probably wind jacket in English), I'd like the code that prepares the data for the index (tokenizer etc) to understand that this is a Jacke (jacket) so that a query for Jacke would include the Windjacke document in its result set. It appears to me that such an analysis requires a dictionary- backed approach, which doesn't have to be perfect at all; a list of the most common 2000 words would probably do the job and fulfil a criterion of reasonable usefulness. A simple approach would obviously be a word list and a regular expression. There will, however, be nuts and bolts to take care of. A more sophisticated and tested approach might be known to you. Michael
Re: Lexical analysis tools for German language data
Michael, I'm on this list and the lucene list since several years and have not found this yet. It's been one neglected topics to my taste. There is a CompoundAnalyzer but it requires the compounds to be dictionary based, as you indicate. I am convinced there's a way to build the de-compounding words efficiently from a broad corpus but I have never seen it (and the experts at DFKI I asked for for also told me they didn't know of one). paul Le 12 avr. 2012 à 11:52, Michael Ludwig a écrit : Given an input of Windjacke (probably wind jacket in English), I'd like the code that prepares the data for the index (tokenizer etc) to understand that this is a Jacke (jacket) so that a query for Jacke would include the Windjacke document in its result set. It appears to me that such an analysis requires a dictionary-backed approach, which doesn't have to be perfect at all; a list of the most common 2000 words would probably do the job and fulfil a criterion of reasonable usefulness. Do you know of any implementation techniques or working implementations to do this kind of lexical analysis for German language data? (Or other languages, for that matter?) What are they, where can I find them? I'm sure there is something out (commercial or free) because I've seen lots of engines grokking German and the way it builds words. Failing that, what are the proper terms do refer to these techniques so you can search more successfully? Michael
Re: Lexical analysis tools for German language data
You might have a look at: http://www.basistech.com/lucene/ Am 12.04.2012 11:52, schrieb Michael Ludwig: Given an input of Windjacke (probably wind jacket in English), I'd like the code that prepares the data for the index (tokenizer etc) to understand that this is a Jacke (jacket) so that a query for Jacke would include the Windjacke document in its result set. It appears to me that such an analysis requires a dictionary-backed approach, which doesn't have to be perfect at all; a list of the most common 2000 words would probably do the job and fulfil a criterion of reasonable usefulness. Do you know of any implementation techniques or working implementations to do this kind of lexical analysis for German language data? (Or other languages, for that matter?) What are they, where can I find them? I'm sure there is something out (commercial or free) because I've seen lots of engines grokking German and the way it builds words. Failing that, what are the proper terms do refer to these techniques so you can search more successfully? Michael
Re: Lexical analysis tools for German language data
If you want that query jacke matches a document containing the word windjacke or kinderjacke, you could use a custom update processor. This processor could search the indexed text for words matching the pattern .*jacke and inject the word jacke into an additional field which you can search against. You would need a whole list of possible suffixes, of course. It would slow down the update process but you don't need to split words during search. Best, Valeriy On Thu, Apr 12, 2012 at 12:39 PM, Paul Libbrecht p...@hoplahup.net wrote: Michael, I'm on this list and the lucene list since several years and have not found this yet. It's been one neglected topics to my taste. There is a CompoundAnalyzer but it requires the compounds to be dictionary based, as you indicate. I am convinced there's a way to build the de-compounding words efficiently from a broad corpus but I have never seen it (and the experts at DFKI I asked for for also told me they didn't know of one). paul Le 12 avr. 2012 à 11:52, Michael Ludwig a écrit : Given an input of Windjacke (probably wind jacket in English), I'd like the code that prepares the data for the index (tokenizer etc) to understand that this is a Jacke (jacket) so that a query for Jacke would include the Windjacke document in its result set. It appears to me that such an analysis requires a dictionary-backed approach, which doesn't have to be perfect at all; a list of the most common 2000 words would probably do the job and fulfil a criterion of reasonable usefulness. Do you know of any implementation techniques or working implementations to do this kind of lexical analysis for German language data? (Or other languages, for that matter?) What are they, where can I find them? I'm sure there is something out (commercial or free) because I've seen lots of engines grokking German and the way it builds words. Failing that, what are the proper terms do refer to these techniques so you can search more successfully? Michael
Re: Lexical analysis tools for German language data
Bernd, can you please say a little more? I think this list is ok to contain some description for commercial solutions that satisfy a request formulated on list. Is there any product at BASIS Tech that provides a compound-analyzer with a big dictionary of decomposed compounds in German? If yes, for which domain? The Google Search result (I wonder if this is politically correct to not have yours ;-)) shows me that there's an amount of job done in this direction (e.g. Gärten to match Garten) but being precise for this question would be more helpful! paul Le 12 avr. 2012 à 12:46, Bernd Fehling a écrit : You might have a look at: http://www.basistech.com/lucene/ Am 12.04.2012 11:52, schrieb Michael Ludwig: Given an input of Windjacke (probably wind jacket in English), I'd like the code that prepares the data for the index (tokenizer etc) to understand that this is a Jacke (jacket) so that a query for Jacke would include the Windjacke document in its result set. It appears to me that such an analysis requires a dictionary-backed approach, which doesn't have to be perfect at all; a list of the most common 2000 words would probably do the job and fulfil a criterion of reasonable usefulness. Do you know of any implementation techniques or working implementations to do this kind of lexical analysis for German language data? (Or other languages, for that matter?) What are they, where can I find them? I'm sure there is something out (commercial or free) because I've seen lots of engines grokking German and the way it builds words. Failing that, what are the proper terms do refer to these techniques so you can search more successfully? Michael
Re: Lexical analysis tools for German language data
Paul, nearly two years ago I requested an evaluation license and tested BASIS Tech Rosette for Lucene Solr. Was working excellent but the price much much to high. Yes, they also have compound analysis for several languages including German. Just configure your pipeline in solr and setup the processing pipeline in Rosette Language Processing (RLP) and thats it. Example from my very old schema.xml config: fieldtype name=text_rlp class=solr.TextField positionIncrementGap=100 analyzer type=index tokenizer class=com.basistech.rlp.solr.RLPTokenizerFactory rlpContext=solr/conf/rlp-index-context.xml postPartOfSpeech=false postLemma=true postStem=true postCompoundComponents=true/ filter class=solr.LowerCaseFilterFactory/ filter class=org.apache.solr.analysis.RemoveDuplicatesTokenFilterFactory/ /analyzer analyzer type=query tokenizer class=com.basistech.rlp.solr.RLPTokenizerFactory rlpContext=solr/conf/rlp-query-context.xml postPartOfSpeech=false postLemma=true postCompoundComponents=true/ filter class=solr.LowerCaseFilterFactory/ filter class=org.apache.solr.analysis.RemoveDuplicatesTokenFilterFactory/ /analyzer /fieldtype So you just point tokenizer to RLP and have two RLP pipelines configured, one for indexing (rlp-index-context.xml) and one for querying (rlp-query-context.xml). Example form my rlp-index-context.xml config: contextconfig properties property name=com.basistech.rex.optimize value=false/ property name=com.basistech.ela.retokenize_for_rex value=true/ /properties languageprocessors languageprocessorUnicode Converter/languageprocessor languageprocessorLanguage Identifier/languageprocessor languageprocessorEncoding and Character Normalizer/languageprocessor languageprocessorEuropean Language Analyzer/languageprocessor !--languageprocessorScript Region Locator/languageprocessor languageprocessorJapanese Language Analyzer/languageprocessor languageprocessorChinese Language Analyzer/languageprocessor languageprocessorKorean Language Analyzer/languageprocessor languageprocessorSentence Breaker/languageprocessor languageprocessorWord Breaker/languageprocessor languageprocessorArabic Language Analyzer/languageprocessor languageprocessorPersian Language Analyzer/languageprocessor languageprocessorUrdu Language Analyzer/languageprocessor -- languageprocessorStopword Locator/languageprocessor languageprocessorBase Noun Phrase Locator/languageprocessor !--languageprocessorStatistical Entity Extractor/languageprocessor -- languageprocessorExact Match Entity Extractor/languageprocessor languageprocessorPattern Match Entity Extractor/languageprocessor languageprocessorEntity Redactor/languageprocessor languageprocessorREXML Writer/languageprocessor /languageprocessors /contextconfig As you can see I used the European Language Analyzer. Bernd Am 12.04.2012 12:58, schrieb Paul Libbrecht: Bernd, can you please say a little more? I think this list is ok to contain some description for commercial solutions that satisfy a request formulated on list. Is there any product at BASIS Tech that provides a compound-analyzer with a big dictionary of decomposed compounds in German? If yes, for which domain? The Google Search result (I wonder if this is politically correct to not have yours ;-)) shows me that there's an amount of job done in this direction (e.g. Gärten to match Garten) but being precise for this question would be more helpful! paul
AW: Lexical analysis tools for German language data
Von: Valeriy Felberg If you want that query jacke matches a document containing the word windjacke or kinderjacke, you could use a custom update processor. This processor could search the indexed text for words matching the pattern .*jacke and inject the word jacke into an additional field which you can search against. You would need a whole list of possible suffixes, of course. Merci, Valeriy - I agree on the feasability of such an approach. The list would likely have to be composed of the most frequently used terms for your specific domain. In our case, it's things people would buy in shops. Reducing overly complicated and convoluted product descriptions to proper basic terms - that would do the job. It's like going to a restaurant boasting fancy and unintelligible names for the dishes you may order when they are really just ordinary stuff like pork and potatoes. Thinking some more about it, giving sufficient boost to the attached category data might also do the job. That would shift the burden of supplying proper semantics to the guys doing the categorization. It would slow down the update process but you don't need to split words during search. Le 12 avr. 2012 à 11:52, Michael Ludwig a écrit : Given an input of Windjacke (probably wind jacket in English), I'd like the code that prepares the data for the index (tokenizer etc) to understand that this is a Jacke (jacket) so that a query for Jacke would include the Windjacke document in its result set. A query for Windjacke or Kinderjacke would probably not have to be de-specialized to Jacke because, well, that's the user input and users looking for specific things are probably doing so for a reason. If no matches are found you can still tell them to just broaden their search. Michael
Re: Lexical analysis tools for German language data
Hi, We've done a lot of tests with the HyphenationCompoundWordTokenFilter using a from TeX generated FOP XML file for the Dutch language and have seen decent results. A bonus was that now some tokens can be stemmed properly because not all compounds are listed in the dictionary for the HunspellStemFilter. It does introduce a recall/precision problem but it at least returns results for those many users that do not properly use compounds in their search query. There seem to be a small issue with the filter where minSubwordSize=N yields subwords of size N-1. Cheers, On Thursday 12 April 2012 12:39:44 Paul Libbrecht wrote: Michael, I'm on this list and the lucene list since several years and have not found this yet. It's been one neglected topics to my taste. There is a CompoundAnalyzer but it requires the compounds to be dictionary based, as you indicate. I am convinced there's a way to build the de-compounding words efficiently from a broad corpus but I have never seen it (and the experts at DFKI I asked for for also told me they didn't know of one). paul Le 12 avr. 2012 à 11:52, Michael Ludwig a écrit : Given an input of Windjacke (probably wind jacket in English), I'd like the code that prepares the data for the index (tokenizer etc) to understand that this is a Jacke (jacket) so that a query for Jacke would include the Windjacke document in its result set. It appears to me that such an analysis requires a dictionary-backed approach, which doesn't have to be perfect at all; a list of the most common 2000 words would probably do the job and fulfil a criterion of reasonable usefulness. Do you know of any implementation techniques or working implementations to do this kind of lexical analysis for German language data? (Or other languages, for that matter?) What are they, where can I find them? I'm sure there is something out (commercial or free) because I've seen lots of engines grokking German and the way it builds words. Failing that, what are the proper terms do refer to these techniques so you can search more successfully? Michael -- Markus Jelsma - CTO - Openindex
AW: Lexical analysis tools for German language data
Von: Markus Jelsma We've done a lot of tests with the HyphenationCompoundWordTokenFilter using a from TeX generated FOP XML file for the Dutch language and have seen decent results. A bonus was that now some tokens can be stemmed properly because not all compounds are listed in the dictionary for the HunspellStemFilter. Thank you for pointing me to these two filter classes. It does introduce a recall/precision problem but it at least returns results for those many users that do not properly use compounds in their search query. Could you define what the term recall should be taken to mean in this context? I've also encountered it on the BASIStech website. Okay, I found a definition: http://en.wikipedia.org/wiki/Precision_and_recall Dank je wel! Michael
Re: Lexical analysis tools for German language data
German noun decompounding is a little more complicated than it might seem. There can be transformations or inflections, like the s in Weinachtsbaum (Weinachten/Baum). Internal nouns should be recapitalized, like Baum above. Some compounds probably should not be decompounded, like Fahrrad (farhren/Rad). With a dictionary-based stemmer, you might decide to avoid decompounding for words in the dictionary. Verbs get more complicated inflections, and might need to be decapitalized, like farhren above. Und so weiter. Note that highlighting gets pretty weird when you are matching only part of a word. Luckily, a lot of compounds are simple, and you could well get a measurable improvement with a very simple algorithm. There isn't anything complicated about compounds like Orgelmusik or Netzwerkbetreuer. The Basis Technology linguistic analyzers aren't cheap or small, but they work well. wunder On Apr 12, 2012, at 3:58 AM, Paul Libbrecht wrote: Bernd, can you please say a little more? I think this list is ok to contain some description for commercial solutions that satisfy a request formulated on list. Is there any product at BASIS Tech that provides a compound-analyzer with a big dictionary of decomposed compounds in German? If yes, for which domain? The Google Search result (I wonder if this is politically correct to not have yours ;-)) shows me that there's an amount of job done in this direction (e.g. Gärten to match Garten) but being precise for this question would be more helpful! paul Le 12 avr. 2012 à 12:46, Bernd Fehling a écrit : You might have a look at: http://www.basistech.com/lucene/ Am 12.04.2012 11:52, schrieb Michael Ludwig: Given an input of Windjacke (probably wind jacket in English), I'd like the code that prepares the data for the index (tokenizer etc) to understand that this is a Jacke (jacket) so that a query for Jacke would include the Windjacke document in its result set. It appears to me that such an analysis requires a dictionary-backed approach, which doesn't have to be perfect at all; a list of the most common 2000 words would probably do the job and fulfil a criterion of reasonable usefulness. Do you know of any implementation techniques or working implementations to do this kind of lexical analysis for German language data? (Or other languages, for that matter?) What are they, where can I find them? I'm sure there is something out (commercial or free) because I've seen lots of engines grokking German and the way it builds words. Failing that, what are the proper terms do refer to these techniques so you can search more successfully? Michael
AW: Lexical analysis tools for German language data
Von: Walter Underwood German noun decompounding is a little more complicated than it might seem. There can be transformations or inflections, like the s in Weinachtsbaum (Weinachten/Baum). I remember from my linguistics studies that the terminus technicus for these is Fugenmorphem (interstitial or joint morpheme). But there's not many of them - phrased in a regex, it's /e?[ns]/. The Weinachtsbaum in the example above is from the singular (die Weihnacht), then s, then Baum. Still, it's much more complex then, say, English or Italian. Internal nouns should be recapitalized, like Baum above. Casing won't matter for indexing, I think. The way I would go about obtaining stems from compound words is by using a dictionary of stems and a regex. We'll see how far that'll take us. Some compounds probably should not be decompounded, like Fahrrad (farhren/Rad). With a dictionary-based stemmer, you might decide to avoid decompounding for words in the dictionary. Good point. Note that highlighting gets pretty weird when you are matching only part of a word. Guess it'll be a weird when you get it wrong, like Noten in Notentriegelung. Luckily, a lot of compounds are simple, and you could well get a measurable improvement with a very simple algorithm. There isn't anything complicated about compounds like Orgelmusik or Netzwerkbetreuer. Exactly. The Basis Technology linguistic analyzers aren't cheap or small, but they work well. We will consider our needs and options. Thanks for your thoughts. Michael
Re: AW: Lexical analysis tools for German language data
Le 12 avr. 2012 à 17:46, Michael Ludwig a écrit : Some compounds probably should not be decompounded, like Fahrrad (farhren/Rad). With a dictionary-based stemmer, you might decide to avoid decompounding for words in the dictionary. Good point. More or less, Fahrrad is generally abbreviated as Rad. (even though Rad can mean wheel and bike) Note that highlighting gets pretty weird when you are matching only part of a word. Guess it'll be a weird when you get it wrong, like Noten in Notentriegelung. This decomposition should not happen because Noten-triegelung does not have a correct second term. The Basis Technology linguistic analyzers aren't cheap or small, but they work well. We will consider our needs and options. Thanks for your thoughts. My question remains as to which domain it aims at covering. We had such need for mathematics texts... I would be pleasantly surprised if, for example, Differenzen-quotient would be decompounded. paul
Re: AW: Lexical analysis tools for German language data
On Apr 12, 2012, at 8:46 AM, Michael Ludwig wrote: I remember from my linguistics studies that the terminus technicus for these is Fugenmorphem (interstitial or joint morpheme). That is some excellent linguistic jargon. I'll file that with hapax legomenon. If you don't highlight, you can get good results with pretty rough analyzers, but highlighting exposes those, even when they don't affect relevance. For example, you can get good relevance just indexing bigrams in Chinese, but it looks awful when you highlight them. As soon as you highlight, you need a dictionary-based segmenter. wunder -- Walter Underwood wun...@wunderwood.org
Re: AW: Lexical analysis tools for German language data
On Thursday 12 April 2012 18:00:14 Paul Libbrecht wrote: Le 12 avr. 2012 à 17:46, Michael Ludwig a écrit : Some compounds probably should not be decompounded, like Fahrrad (farhren/Rad). With a dictionary-based stemmer, you might decide to avoid decompounding for words in the dictionary. Good point. More or less, Fahrrad is generally abbreviated as Rad. (even though Rad can mean wheel and bike) Note that highlighting gets pretty weird when you are matching only part of a word. Guess it'll be a weird when you get it wrong, like Noten in Notentriegelung. This decomposition should not happen because Noten-triegelung does not have a correct second term. The Basis Technology linguistic analyzers aren't cheap or small, but they work well. We will consider our needs and options. Thanks for your thoughts. My question remains as to which domain it aims at covering. We had such need for mathematics texts... I would be pleasantly surprised if, for example, Differenzen-quotient would be decompounded. The HyphenationCompoundWordTokenFilter can do those things but those words must be listed in the dictionary or you'll get strange results. It still yields strange results when it emits tokens that are subwords of a subword. paul -- Markus Jelsma - CTO - Openindex
Re: AW: Lexical analysis tools for German language data
On Apr 12, 2012, at 9:00 AM, Paul Libbrecht wrote: More or less, Fahrrad is generally abbreviated as Rad. (even though Rad can mean wheel and bike) A synonym could handle this, since farhren would not be a good match. It is judgement call, but this seems more like an equivalence Fahrrad = Rad than decompounding. wunder -- Walter Underwood wun...@wunderwood.org
