On 30/09/2015 10:13, Zheng Lin Edwin Yeo wrote:
Hi Charlie,
Hi Edwin,
Thanks for your reply. Seems like quite a number of the chinese tokenizers
are not really compatible with the newer versions of Solr
I'm also looking at HMMChineseTokenizer and JiebaTokenizer to see if they
are suitable to be used for Solr 5.x too.
I think there is a general lack of knowledge (at least in the
non-Chinese-speaking community) about the best way to analyze Chinese
content with Lucene/Solr - so if you can write up your experiences that
would be great!
Cheers
Charlie
Regards,
Edwin
On 30 September 2015 at 16:20, Charlie Hull <char...@flax.co.uk> wrote:
On 30/09/2015 04:09, Zheng Lin Edwin Yeo wrote:
Hi Charlie,
Hi,
I've checked that Paoding's code is written for Solr 3 and Solr 4
versions.
It is not written for Solr 5, thus I was unable to use it in my Solr 5.x
version.
I'm pretty sure we had to recompile it for v4.6 as well....it has been a
little painful.
Have you tried to use HMMChineseTokenizer and JiebaTokenizer as well?
I don't think so.
Charlie
Regards,
Edwin
On 25 September 2015 at 18:46, Charlie Hull <char...@flax.co.uk> wrote:
On 25/09/2015 11:43, Zheng Lin Edwin Yeo wrote:
Hi Charlie,
Thanks for your comment. I faced the compatibility issues with Paoding
when
I tried it in Solr 5.1.0 and Solr 5.2.1, and I found out that the code
was
optimised for Solr 3.6.
Which version of Solr are you using when you tried on the Paoding?
Solr v4.6 I believe.
Charlie
Regards,
Edwin
On 25 September 2015 at 16:43, Charlie Hull <char...@flax.co.uk> wrote:
On 23/09/2015 16:23, Alexandre Rafalovitch wrote:
You may find the following articles interesting:
http://discovery-grindstone.blogspot.ca/2014/01/searching-in-solr-analyzing-results-and.html
( a whole epic journey)
https://dzone.com/articles/indexing-chinese-solr
The latter article is great and we drew on it when helping a recent
client
with Chinese indexing. However, if you do use Paoding bear in mind that
it
has few if any tests and all the comments are in Chinese. We found a
problem with it recently (it breaks the Lucene highlighters) and have
submitted a patch:
http://git.oschina.net/zhzhenqin/paoding-analysis/issues/1
Cheers
Charlie
Regards,
Alex.
----
Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
http://www.solr-start.com/
On 23 September 2015 at 10:41, Zheng Lin Edwin Yeo <
edwinye...@gmail.com>
wrote:
Hi,
Would like to check, will StandardTokenizerFactory works well for
indexing
both English and Chinese (Bilingual) documents, or do we need
tokenizers
that are customised for chinese (Eg: HMMChineseTokenizerFactory)?
Regards,
Edwin
--
Charlie Hull
Flax - Open Source Enterprise Search
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828
web: www.flax.co.uk
--
Charlie Hull
Flax - Open Source Enterprise Search
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828
web: www.flax.co.uk
--
Charlie Hull
Flax - Open Source Enterprise Search
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828
web: www.flax.co.uk
--
Charlie Hull
Flax - Open Source Enterprise Search
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828
web: www.flax.co.uk
