[lucy-user] Chinese support?
Hi all, I use the StandardTokenizer. search by English word work, but in Chinese give me strange results. my $tokenizer = Lucy::Analysis::StandardTokenizer->new; my $raw_type = Lucy::Plan::FullTextType->new( analyzer => $tokenizer, ); also, I was going to use the EasyAnalyzer ( https://metacpan.org/pod/distribution/Lucy/lib/Lucy/Analysis/EasyAnalyzer.pod ) , but chinese in not supported. What is the simple way to use lucy with chinese doc? Thanks. Best, Hao Wu
Re: [lucy-user] Chinese support?
Thanks. Get it work.
code pasted below in case anyone have similar question.
package ChineseAnalyzer;
use Jieba;
use v5.10;
use Encode qw(decode_utf8);
use base qw( Lucy::Analysis::Analyzer );
sub new {
my $self = shift->SUPER::new;
return $self;
}
sub transform {
my ($self, $inversion)= @_;
return $inversion;
}
sub transform_text {
my ($self, $text) = @_;
my $inversion = Lucy::Analysis::Inversion->new;
my @tokens = Jieba::jieba_tokenize(decode_utf8($text));
$inversion->append(
Lucy::Analysis::Token->new(text =>$_->[0],
start_offset=> $_->[1] ,
end_offset=>$_->[2]
)
) for @tokens;
return $inversion;
}
1;
package Jieba;
use v5.10;
sub jieba_tokenize {
jieba_tokenize_python(shift);
}
# TODO:
#result = jieba.tokenize(u'永和服装饰品有限公司', mode='search')
use Inline Python => <<'END_OF_PYTHON_CODE';
from jieba import tokenize
def jieba_tokenize_python(text):
seg_list = tokenize(text, mode='search')
return(list(seg_list))
END_OF_PYTHON_CODE
1;
On Fri, Feb 17, 2017 at 6:29 PM, Peter Karman wrote:
> Hao Wu wrote on 2/17/17 4:44 PM:
>
>> Hi all,
>>
>> I use the StandardTokenizer. search by English word work, but in
>> Chinese give me strange results.
>>
>> my $tokenizer = Lucy::Analysis::StandardTokenizer->new;
>> my $raw_type = Lucy::Plan::FullTextType->new(
>> analyzer => $tokenizer,
>> );
>>
>> also, I was going to use the EasyAnalyzer (
>> https://metacpan.org/pod/distribution/Lucy/lib/Lucy/Analysis
>> /EasyAnalyzer.pod
>> )
>> , but chinese in not supported.
>>
>> What is the simple way to use lucy with chinese doc? Thanks.
>>
>
> There is currently no equivalent of
> https://lucene.apache.org/core/4_0_0/analyzers-common/org/
> apache/lucene/analysis/cjk/CJKTokenizer.html
> within core Lucy.
>
> Furthermore, there is no automatic language detection in Lucy. You'll note
> in https://metacpan.org/pod/distribution/Lucy/lib/Lucy/Analysis
> /EasyAnalyzer.pod
> that the language must be explicitly specified, and that is for the
> stemming analyzer. Also, Chinese is not among the supported languages
> listed.
>
> Maybe something wrapped around https://metacpan.org/pod/Lingu
> a::CJK::Tokenizer would work as a custom analyzer.
>
> You can see an example in the documentation here
> https://metacpan.org/pod/Lucy::Analysis::Analyzer#new
>
>
>
> --
> Peter Karman . https://peknet.com/ . https://keybase.io/peterkarman
>
Re: [lucy-user] Chinese support?
Still have problem when I try to update the index using the custom analyzer.
If I comment out the
truncate => 1
rerun I got the following errror.
'body' assigned conflicting FieldType
LUCY_Schema_Spec_Field_IMP at cfcore/Lucy/Plan/Schema.c line 124
at /home/hwu/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/Lucy.pm
line 118.
Lucy::Index::Indexer::new('Lucy::Index::Indexer', 'index',
'/home/hwu/data/lucy/mitbbs.index', 'schema',
'Lucy::Plan::Schema=SCALAR(0x211c758)', 'create', 1) called at
mitbbs_index.pl line 26
*** Error in `perl': corrupted double-linked list: 0x021113a0 ***
If I switch the analyzer to Lucy::Analysis::StandardTokenize. works fine.
a new seg_2 is created.
my $tokenizer = Lucy::Analysis::StandardTokenizer->new;
my $raw_type = Lucy::Plan::FullTextType->new(
analyzer => $tokenizer,
);
So I guess I must miss something in the custom Chinese Analyzer.
--my script
#!/usr/local/bin/perl
#TODO: update doc, instead create everytime
use DBI;
use File::Spec::Functions qw( catfile );
use Lucy::Plan::Schema;
use Lucy::Plan::FullTextType;
use Lucy::Index::Indexer;
use ChineseAnalyzer;
my $path_to_index = '/home/hwu/data/lucy/mitbbs.index';
# Create Schema.
my $schema = Lucy::Plan::Schema->new;
my $chinese = ChineseAnalyzer->new();
my $raw_type = Lucy::Plan::FullTextType->new(
analyzer => $chinese,
);
$schema->spec_field( name => 'body', type => $raw_type);
# Create an Indexer object.
my $indexer = Lucy::Index::Indexer->new(
index=> $path_to_index,
schema => $schema,
create => 1,
truncate => 1,
);
my $driver = "SQLite";
my $database = "/home/hwu/data/mitbbs.db";
my $dsn = "DBI:$driver:dbname=$database";
my $dbh = DBI->connect($dsn,{ RaiseError => 1 }) or die $DBI::errstr;
my $stmt = qq(SELECT id, text from post where id >= 100 and id < 200;);
#my $stmt = qq(SELECT id, text from post where id < 100;);
my $sth = $dbh->prepare( $stmt );
my $rv = $sth->execute() or die $DBI::errstr;
while(my @row = $sth->fetchrow_array()) {
print "id = ". $row[0] . "\n";
print $row[1];
my $doc = { body => $row[1] };
$indexer->add_doc($doc);
}
$indexer->commit;
print "Finished.\n";
On Sat, Feb 18, 2017 at 6:46 AM, Nick Wellnhofer
wrote:
> On 18/02/2017 07:22, Hao Wu wrote:
>
>> Thanks. Get it work.
>>
>
> Lucy's StandardTokenizer breaks up the text at the word boundaries defined
> in Unicode Standard Annex #29. Then we treat every Alphabetic character
> that doesn't have a Word_Break property as a single term. These are
> characters that match \p{Ideographic}, \p{Script: Hiragana}, or
> \p{Line_Break: Complex_Context}. This should work for Chinese but as Peter
> mentioned, we don't support n-grams.
>
> If you're using QueryParser, you're likely to run into problems, though.
> QueryParser will turn a sequence of Chinese characters into a PhraseQuery
> which is obviously wrong. A quick hack is to insert a space after every
> Chinese character before passing a query string to QueryParser:
>
> $query_string =~ s/\p{Ideographic}/$& /g;
>
> Nick
>
>
Re: [lucy-user] Chinese support?
Hi Peter,
Thanks for reply.
That could be a problem. But probably not in my case.
I removed the old index.
run the program with 'ChineseAnalyzer' and truncate => 0 twice. the second
time, will give me the error.
'body' assigned conflicting FieldType
LUCY_Schema_Spec_Field_IMP at cfcore/Lucy/Plan/Schema.c line 124
at /home/hwu/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/Lucy.pm
line 118.
Lucy::Index::Indexer::new('Lucy::Index::Indexer', 'index',
'/home/hwu/data/lucy/mitbbs.index', 'schema',
'Lucy::Plan::Schema=SCALAR(0x1c56798)', 'create', 1) called at
mitbbs_index.pl line 26
run the program with 'ChineseAnalyzer' and truncate => 0 twice, no error.
but I want to update the index.
run the program with 'StandardTokenizer', with truncate 0 or 1, both work
fine.
So, this make me think I must miss something in the 'ChineseAnalyzer' I
have.
On Mon, Feb 20, 2017 at 6:47 PM, Peter Karman wrote:
> Hao Wu wrote on 2/20/17 6:12 PM:
>
>> Still have problem when I try to update the index using the custom
>> analyzer.
>>
>> If I comment out the
>>truncate => 1
>>
>> rerun I got the following errror.
>>
>>
>> 'body' assigned conflicting FieldType
>> LUCY_Schema_Spec_Field_IMP at cfcore/Lucy/Plan/Schema.c line 124
>> at /home/hwu/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/Lucy
>> .pm
>> line 118.
>> Lucy::Index::Indexer::new('Lucy::Index::Indexer', 'index',
>> '/home/hwu/data/lucy/mitbbs.index', 'schema',
>> 'Lucy::Plan::Schema=SCALAR(0x211c758)', 'create', 1) called at
>> mitbbs_index.pl line 26
>> *** Error in `perl': corrupted double-linked list: 0x021113a0 ***
>>
>> If I switch the analyzer to Lucy::Analysis::StandardTokenize. works
>> fine.
>> a new seg_2 is created.
>>
>> my $tokenizer = Lucy::Analysis::StandardTokenizer->new;
>> my $raw_type = Lucy::Plan::FullTextType->new(
>> analyzer => $tokenizer,
>> );
>>
>> So I guess I must miss something in the custom Chinese Analyzer.
>>
>>
> since you changed the field definition with a new analyzer, you must
> create a new index. You cannot update an existing index with 2 different
> field definitions in the same schema.
>
>
>
> --
> Peter Karman . https://peknet.com/ . https://keybase.io/peterkarman
>
Re: [lucy-user] Chinese support?
Hi Peter,
Thanks for spending time in the script.
I clean it up a bit, so there is no dependency now.
https://gist.github.com/swuecho/1b960ae17a1f47466be006fd14e3b7ff
still do not work.
On Mon, Feb 20, 2017 at 9:03 PM, Peter Karman wrote:
> Hao Wu wrote on 2/20/17 10:18 PM:
>
>> Hi Peter,
>>
>> Thanks for reply.
>>
>> That could be a problem. But probably not in my case.
>>
>> I removed the old index.
>>
>> run the program with 'ChineseAnalyzer' and truncate => 0 twice. the
>> second
>> time, will give me the error.
>>
>> 'body' assigned conflicting FieldType
>> LUCY_Schema_Spec_Field_IMP at cfcore/Lucy/Plan/Schema.c line 124
>> at /home/hwu/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/Lucy.pm
>> line 118.
>> Lucy::Index::Indexer::new('Lucy::Index::Indexer', 'index',
>> '/home/hwu/data/lucy/mitbbs.index', 'schema',
>> 'Lucy::Plan::Schema=SCALAR(0x1c56798)', 'create', 1) called at
>> mitbbs_index.pl
>> <http://mitbbs_index.pl> line 26
>>
>> run the program with 'ChineseAnalyzer' and truncate => 0 twice, no
>> error. but I
>> want to update the index.
>>
>> run the program with 'StandardTokenizer', with truncate 0 or 1, both
>> work fine.
>>
>> So, this make me think I must miss something in the 'ChineseAnalyzer' I
>> have.
>>
>>
>
> This is not your default, I don't think. This seems like a bug.
>
> Here's a smaller gist demonstrating the problem:
>
> https://gist.github.com/karpet/d8fe12085246b8419f9e4ab44930c1cc
>
> With the 2 files in the gist, I get this result:
>
> [karpet@pekmac:~/tmp/chinese-analyzer]$ perl indexer.pl test-index
> Building prefix dict from the default dictionary ...
> Loading model from cache /var/folders/r3/yk7hmbb9125fns
> df9bqs6lrmgp/T/jieba.cache
> Loading model cost 0.553 seconds.
> Prefix dict has been built succesfully.
> Finished.
>
> [karpet@pekmac:~/tmp/chinese-analyzer]$ perl indexer.pl test-index
> 'body' assigned conflicting FieldType
> LUCY_Schema_Spec_Field_IMP at cfcore/Lucy/Plan/Schema.c line 124
> at /usr/local/perl/5.24.0/lib/site_perl/5.24.0/darwin-2level/Lucy.pm
> line 118.
> Lucy::Index::Indexer::new("Lucy::Index::Indexer", "index",
> "test-index", "schema", Lucy::Plan::Schema=SCALAR(0x7f9b0b004a18),
> "create", 1) called at indexer.pl line 23
> Segmentation fault: 11
>
>
>
> I would expect the code to work as you wrote it, so maybe someone else can
> spot what's going wrong.
>
> Here's what the schema_1.json file looks like after the initial index
> creation:
>
> {
> "_class": "Lucy::Plan::Schema",
> "analyzers": [
> null,
> {
> "_class": "ChineseAnalyzer"
> }
> ],
> "fields": {
> "body": {
> "analyzer": "1",
> "type": "fulltext"
>
> }
> }
> }
>
>
> --
> Peter Karman . https://peknet.com/ . https://keybase.io/peterkarman
>
Re: [lucy-user] Custom Analyzer [was Chinese support?]
Hi Peter,
works great. The document is significantly better now.
Thanks everyone for taking care of this issues.
Best,
Hao
On Thu, Feb 23, 2017 at 8:23 AM, Peter Karman wrote:
> Nick Wellnhofer wrote on 2/23/17 6:49 AM:
>
>> On 23/02/2017 04:52, Peter Karman wrote:
>>
>>> package MyAnalyzer {
>>> use base qw( Lucy::Analysis::Analyzer );
>>> sub transform { $_[1] }
>>> }
>>>
>>
>> Every Analyzer needs an `equals` method. For simple Analyzers, it can
>> simply
>> check whether the class of the other object matches:
>>
>> package MyAnalyzer {
>> use base qw( Lucy::Analysis::Analyzer );
>> sub transform { $_[1] }
>> sub equals { $_[1]->isa(__PACKAGE__) }
>> }
>>
>> If the Analyzer uses (inside-out) member variables, you'll also need dump
>> and
>> load methods. Unfortunately, we don't have good documentation for writing
>> custom
>> analyzers yet.
>>
>>
>
> Thanks for the quick response and accurate diagnosis, Nick. I see you've
> already committed changes to the POD so that will be very helpful in future.
>
> Hao, if you add the `sub equals` method to your ChineseAnalyzer, I think
> that should fix your problem. I have confirmed that locally with my own
> tests.
>
>
>
> --
> Peter Karman . https://peknet.com/ . https://keybase.io/peterkarman
>
