[ 
https://issues.apache.org/jira/browse/CASSANDRA-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381446#comment-16381446
 ] 

mck edited comment on CASSANDRA-14247 at 3/1/18 3:10 AM:
---------------------------------------------------------

Approaches to (2) are found 
[here|https://github.com/thelastpickle/cassandra/commit/0d6c8117120ef444e1aa52e49ab66aafa159677e]
 and 
[here|https://github.com/thelastpickle/cassandra/commit/c1f66d7c389ab5816b36d7d02ca2b8043bab0ecf].

The former was just my first attempt at removing the overhead of the 
{{string.split(..)}} call. The second re-codes it to use nio buffers.
It's the latter i presume we are aiming for. Is it what you had in mind 
[~mkjellman]?
A few quick stress test showed that it was 60% (±5%) faster than the original 
patches above, working with {{world_cities_a.csv}} as input.

{quote}iterate the text left or right or right{quote}
Can we put that in the too-hard basket for now? 
I would think a better next step (in a new ticket) would be to improve the 
other analysers to also use ByteBuffers. as there's an obvious win here.


was (Author: michaelsembwever):
Approaches to (2) are found 
[here|https://github.com/thelastpickle/cassandra/commit/0d6c8117120ef444e1aa52e49ab66aafa159677e]
 and 
[here|https://github.com/thelastpickle/cassandra/commit/c1f66d7c389ab5816b36d7d02ca2b8043bab0ecf].

The former was just my first attempt at removing the overhead of the 
{{string.split(..)}} call. The second re-codes it to use nio buffers.
It's the latter i presume we are aiming for. Is it what you had in mind 
[~mkjellman]?
A few quick stress test showed that it was 60% (±5%) faster than the original 
patches above, working with {{world_cities_a.csv}} as input.

{quote}iterate the text left or right or right{quote}
Can we put that in the too-hard basket for now? 
I think the next step would be to improve the other analysers to also use 
ByteBuffers. as there's an obvious win here.

> SASI tokenizer for simple delimiter based entries
> -------------------------------------------------
>
>                 Key: CASSANDRA-14247
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14247
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: sasi
>            Reporter: mck
>            Assignee: mck
>            Priority: Major
>             Fix For: 4.0, 3.11.x
>
>
> Currently SASI offers only two tokenizer options:
>  - NonTokenizerAnalyser
>  - StandardAnalyzer
> The latter is built upon Snowball, powerful for human languages but overkill 
> for simple tokenization.
> A simple tokenizer is proposed here. The need for this arose as a workaround 
> of CASSANDRA-11182, and to avoid the disk usage explosion when having to 
> resort to {{CONTAINS}}. See https://github.com/openzipkin/zipkin/issues/1861
> Example use of this would be:
> {code}
> CREATE CUSTOM INDEX span_annotation_query_idx 
>     ON zipkin2.span (annotation_query) USING 
> 'org.apache.cassandra.index.sasi.SASIIndex' 
>     WITH OPTIONS = {
>         'analyzer_class': 
> 'org.apache.cassandra.index.sasi.analyzer.DelimiterAnalyzer', 
>         'delimiter': '░',
>         'case_sensitive': 'true', 
>         'mode': 'prefix', 
>         'analyzed': 'true'};
> {code}
> Original credit for this work goes to https://github.com/zuochangan



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

Reply via email to