Re: Analysis of Japanese characters
On 4/7/2014 2:07 PM, T. Kuro Kurosaka wrote: Tom, You should be using JapaneseAnalyzer (kuromoji). Neither CJK nor ICU tokenize at word boundaries. Is JapaneseAnalyzer configurable with regard to what it does with non-japanese text? If it's not, it won't work for me. We use a combination of tokenizers and filters because there are no full analyzers that do what we require. My analysis chain (for our index that's primarily english) has evolved over the last few years into its current form: http://apaste.info/xa5 For our Japanese customer, we have recently changed from ICUFoldingFilter to ASCIIFoldingFilter and ICUNormalizer2Filter, because they do not want us to fold accent marks on Japanese characters. I do not understand enough about Japanese to have an opinion on this, beyond the general "we should normalize EVERYTHING" approach. The data from this customer is not purely Japanese - there is a lot of English as well, and quite possibly a small amount of other languages. Thanks, Shawn
Re: Analysis of Japanese characters
Tom, You should be using JapaneseAnalyzer (kuromoji). Neither CJK nor ICU tokenize at word boundaries. On 04/02/2014 10:33 AM, Tom Burton-West wrote: Hi Shawn, I'm not sure I understand the problem and why you need to solve it at the ICUTokenizer level rather than the CJKBigramFilter Can you perhaps give a few examples of the problem? Have you looked at the flags for the CJKBigramfilter? You can tell it to make bigrams of different Japanese character sets. For example the config given in the JavaDocs tells it to make bigrams across 3 of the different Japanese character sets. (Is the issue related to Romaji?) http://lucene.apache.org/core/4_7_1/analyzers-common/org/apache/lucene/analysis/cjk/CJKBigramFilterFactory.html Tom On Wed, Apr 2, 2014 at 1:19 PM, Shawn Heisey wrote: My company is setting up a system for a customer from Japan. We have an existing system that handles primarily English. Here's my general text analysis chain: http://apaste.info/xa5 After talking to the customer about problems they are encountering with search, we have determined that some of the problems are caused because ICUTokenizer splits on *any* character set change, including changes between different Japanase character sets. Knowing the risk of this being an XY problem, here's my question: Can someone help me develop a rule file for the ICU Tokenizer that will *not* split when the character set changes from one of the japanese character sets to another japanese character set, but still split on other character set changes? Thanks, Shawn
Re: Analysis of Japanese characters
No specific answers, but have you read the detailed CJK article collection: http://discovery-grindstone.blogspot.ca/ . There is a lot of information there. Regards, Alex. Personal website: http://www.outerthoughts.com/ Current project: http://www.solr-start.com/ - Accelerating your Solr proficiency On Thu, Apr 3, 2014 at 12:19 AM, Shawn Heisey wrote: > My company is setting up a system for a customer from Japan. We have an > existing system that handles primarily English. > > Here's my general text analysis chain: > > http://apaste.info/xa5 > > After talking to the customer about problems they are encountering with > search, we have determined that some of the problems are caused because > ICUTokenizer splits on *any* character set change, including changes between > different Japanase character sets. > > Knowing the risk of this being an XY problem, here's my question: Can > someone help me develop a rule file for the ICU Tokenizer that will *not* > split when the character set changes from one of the japanese character sets > to another japanese character set, but still split on other character set > changes? > > Thanks, > Shawn >
Re: Analysis of Japanese characters
Hi Shawn, >>For an input of 田中角栄 the bigram filter works like you described, and what I would expect. If I add a space at the point where the ICU >>tokenizer would have split them anyway, the bigram filter output is very different. If I'm understanding what you are reporting, I suspect this is behavior as designed. My guess is that the bigram filter figures that if there was space in the original input (to the whole filter chain), it should not create a bigram across it. Tom BTW: if you can show a few examples of Japanese queries the show the original problem and the reason its a problem (without of course showing anything proprietary), I'd love to see them. I'm always interested in learning more about Japanese query processing.
Re: Analysis of Japanese characters
On 4/2/2014 2:19 PM, Tom Burton-West wrote: Hi Shawn, I may still be missing your point. Below is an example where the ICUTokenizer splits Now, I'm beginning to wonder if I really understand what those flags on the CJKBigramFilter do. The ICUTokenizer spits out unigrams and the CJKBigramFilter will put them back together into bigrams. I thought if you set han=true, hiragana=true You would get this kind of result where the third bigram is composed of a hirigana and han character It looks like you are right. I did not notice that the bigram filter was putting the tokens back together, even though the tokenizer was splitting them apart. I might be worrying over nothing! Thank you for taking some time to point out the obvious. I did notice something odd, though. Keep in mind that I have absolutely no idea what I am writing here, so I have no idea if this is valid at all: For an input of 田中角栄 the bigram filter works like you described, and what I would expect. If I add a space at the point where the ICU tokenizer would have split them anyway, the bigram filter output is very different. Best guess: It notices that the end/start values from the original input are not consecutive, and therefore doesn't combine them. Like I said above, I may have nothing at all to worry about here. Thanks, Shawn
Re: Analysis of Japanese characters
Hi Shawn, I may still be missing your point. Below is an example where the ICUTokenizer splits Now, I'm beginning to wonder if I really understand what those flags on the CJKBigramFilter do. The ICUTokenizer spits out unigrams and the CJKBigramFilter will put them back together into bigrams. I thought if you set han=true, hiragana=true You would get this kind of result where the third bigram is composed of a hirigana and han character いろは革命歌 =>“いろ” ”ろは“ “は革” ”革命” “命歌” Hopefully the e-mail hasn't munged the output of the Solr analysis panel below: I can see this in our query processing where outpugUnigrams=false: org.apache.solr.analysis.ICUTokenizerFactory {luceneMatchVersion=LUCENE_36} Splits into unigrams term text いろは革命歌 org.apache.solr.analysis.CJKBigramFilterFactory {hangul=false, outputUnigrams=false, katakana=false, han=true, hiragana=true, luceneMatchVersion=LUCENE_36} makes bigrams including the middle one which is one character hirigana and one han term text いろろはは革革命命歌 It appears that if you include outputUnigrams=true (as we both do in the indexing configuration) that this doesn't happen. org.apache.solr.analysis.CJKBigramFilterFactory {hangul=false, outputUnigrams=true, katakana=false, han=true, hiragana=true , luceneMatchVersion=LUCENE_36} いろは革命歌 革命命歌 type Not sure what happens for katakana as the ICUTokenizer doesn't convert it to unigrams and our configuration is set to katakana=false. I'll play around on the test machine when I have time. Tom
Re: Analysis of Japanese characters
On 4/2/2014 11:33 AM, Tom Burton-West wrote: Hi Shawn, I'm not sure I understand the problem and why you need to solve it at the ICUTokenizer level rather than the CJKBigramFilter Can you perhaps give a few examples of the problem? Have you looked at the flags for the CJKBigramfilter? You can tell it to make bigrams of different Japanese character sets. For example the config given in the JavaDocs tells it to make bigrams across 3 of the different Japanese character sets. (Is the issue related to Romaji?) http://lucene.apache.org/core/4_7_1/analyzers-common/org/apache/lucene/analysis/cjk/CJKBigramFilterFactory.html Tom On Wed, Apr 2, 2014 at 1:19 PM, Shawn Heisey wrote: My company is setting up a system for a customer from Japan. We have an existing system that handles primarily English. Here's my general text analysis chain: http://apaste.info/xa5 After talking to the customer about problems they are encountering with search, we have determined that some of the problems are caused because ICUTokenizer splits on *any* character set change, including changes between different Japanase character sets. Knowing the risk of this being an XY problem, here's my question: Can someone help me develop a rule file for the ICU Tokenizer that will *not* split when the character set changes from one of the japanese character sets to another japanese character set, but still split on other character set changes? Because of what ICUTokenizer does, by the time it makes it to the bigram filter, they're already separate terms. Simplifying to english, let's pretend that upper and lowercase letters are in different character sets. Original term is abCD. You expect that by the end of the analysis, you'll have ab bC CD. With the ICUTokenizer, you end up with just ab CD. The index side is more complex because of outputUnigrams. We are still deciding whether we want to keep that parameter set, but that's a separate issue, one that we know how to resolve without help. Thanks, Shawn
Re: Analysis of Japanese characters
Hi Shawn, I'm not sure I understand the problem and why you need to solve it at the ICUTokenizer level rather than the CJKBigramFilter Can you perhaps give a few examples of the problem? Have you looked at the flags for the CJKBigramfilter? You can tell it to make bigrams of different Japanese character sets. For example the config given in the JavaDocs tells it to make bigrams across 3 of the different Japanese character sets. (Is the issue related to Romaji?) http://lucene.apache.org/core/4_7_1/analyzers-common/org/apache/lucene/analysis/cjk/CJKBigramFilterFactory.html Tom On Wed, Apr 2, 2014 at 1:19 PM, Shawn Heisey wrote: > My company is setting up a system for a customer from Japan. We have an > existing system that handles primarily English. > > Here's my general text analysis chain: > > http://apaste.info/xa5 > > After talking to the customer about problems they are encountering with > search, we have determined that some of the problems are caused because > ICUTokenizer splits on *any* character set change, including changes > between different Japanase character sets. > > Knowing the risk of this being an XY problem, here's my question: Can > someone help me develop a rule file for the ICU Tokenizer that will *not* > split when the character set changes from one of the japanese character > sets to another japanese character set, but still split on other character > set changes? > > Thanks, > Shawn > >
Analysis of Japanese characters
My company is setting up a system for a customer from Japan. We have an existing system that handles primarily English. Here's my general text analysis chain: http://apaste.info/xa5 After talking to the customer about problems they are encountering with search, we have determined that some of the problems are caused because ICUTokenizer splits on *any* character set change, including changes between different Japanase character sets. Knowing the risk of this being an XY problem, here's my question: Can someone help me develop a rule file for the ICU Tokenizer that will *not* split when the character set changes from one of the japanese character sets to another japanese character set, but still split on other character set changes? Thanks, Shawn