Hi Everyone,
My question is related to the arabic analysis package under:
org.apache.lucene.analysis.ar
It is cool and it is doing a great job, but it uses a special tokenizer:
ArabicLetterTokenizer
The problem with this tokenizer is that it fails to handle emails, urls
and acronyms the same way the StandardTokenizer does.
Also the problem of the StandardTokenizer is that it fails to handle
arabic diacritics right. so it splits words which shouldn't be splitted.
Arabic diacritics are: (as mentioned in the class:
org.apache.lucene.analysis.ar.ArabicNormalizer)
FATHATAN = '\u064B';
DAMMATAN = '\u064C';
KASRATAN = '\u064D';
FATHA = '\u064E';
DAMMA = '\u064F';
KASRA = '\u0650';
SHADDA = '\u0651';
SUKUN = '\u0652';
so it is the range [\u064B-\u0652]
Is it possible to modify the StandardTokenizerImp to consider these
diacritics as normal letters.
I guess it should be done the same way its is done for Chinese and
Japanese in this line in the file StandardTokenizerImp.jflex
// Chinese and Japanese (but NOT Korean, which is included in [:letter:])
CJ =
[\u3100-\u312f\u3040-\u309F\u30A0-\u30FF\u31F0-\u31FF\u3300-\u337f\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff\uff65-\uff9f]
so it can be something like:
AR = [\u064B-\u0652]
then modify this line also to include our new group of characters:
// From the JFlex manual: "the expression that matches everything of <a>
not matched by <b> is !(!<a>|<b>)"
LETTER = !(![:letter:]|{CJ}|{AR})
Am I right?! and am I going in the right direction?!! Comments are very
welcome.
Regards..
Yusuf
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org