[ https://issues.apache.org/jira/browse/LUCENE-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929556#action_12929556 ]
M Alexander commented on LUCENE-2745: ------------------------------------- {quote} I think that ArabicLetterTokenizer, which is the tokenizer used by ArabicAnalyzer, is obsolete (as of version 3.1), since StandardTokenizer, which implements the Unicode word segmentation rules from UAX#29, should be able to properly tokenize Arabic. StandardTokenizer recognizes email addresses, hostnames, and URLs, so your concern would be addressed. (See LUCENE-2167, though, which was just reopened to turn off full URL output.) You can test this by composing your own analyzer, if you're willing to try using using as-yet-unreleased branch_3X, from which 3.1 will be cut (hopefully fairly soon): just copy ArabicAnalyzer class and swap in StandardTokenizer for ArabicLetterTokenizer {quote} I tried to test this and failed (miserably). I think I struggled to patch LUCENE-2167 correctly through my eclipse. I might just wait for branch_3X release to make my life easier. I will then create my own Analyzer to perform Arabic Text Analysis and another one for Farsi Text Analysis. Both Analyzers will have the ability to handle diacritics as well as email addresses, hostnames and so on. I will colse this issue for now (will re-open in the future if needed). Quick question - any thoughts of handling Arabic email addresses and hostnames in the future? Thanks to both of you for the time taken and I shall wait for the branch release to solve my issue. > ArabicAnalyzer - the ability to recognise email addresses host names and so on > ------------------------------------------------------------------------------ > > Key: LUCENE-2745 > URL: https://issues.apache.org/jira/browse/LUCENE-2745 > Project: Lucene - Java > Issue Type: Improvement > Components: contrib/analyzers > Affects Versions: 2.9.2, 2.9.3, 3.0, 3.0.1, 3.0.2 > Environment: All > Reporter: M Alexander > > The ArabicAnalyzer does not recognise email addresses, hostnames and so on. > For example, > a...@hotmail.com > will be tokenised to [adam] [hotmail] [com] > It would be great if the ArabicAnalyzer can tokenises this to > [a...@hotmail.com]. The same applies to hostnames and so on. > Can this be resolved? I hope so > Thanks > MAA -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org