[ https://issues.apache.org/jira/browse/CASSANDRA-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16376253#comment-16376253 ]
mck edited comment on CASSANDRA-14247 at 2/26/18 12:35 AM: ----------------------------------------------------------- [~mkjellman], have forced pushed the branch again. (let me know if you want to be adding checkpoint commits rather than overwriting the existing commit.) This adds the test file {{test/resources/tokenization/world_cities_a.csv}}, and a unit test to match. The other unit test methods have been updated to use different delimiters as appropriate for the existing test data files. Example corridor testing… {code:java} create table test ( one text, two int, three text, PRIMARY KEY (one,two) ); # insert a new row, with the contents of test/resources/tokenization/world_cities_a.csv going into column 'three'. create CUSTOM INDEX on test (three) USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = { 'analyzer_class': 'org.apache.cassandra.index.sasi.analyzer.DelimiterAnalyzer', 'delimiter': ',', 'mode': 'prefix', 'analyzed': 'true'}; select one,two from test where three LIKE 'azzazl' ALLOW FILTERING; {code} Aside: this tokenizer raises the need for a "exact" mode. Querying a csv inside a column like this is one example where the user may never require wildcarding LIKE clause (using %) and an 'exact' mode would be significantly more performant and use less disk. (btw I'm suspecting that {{is_literal: false}} would have the same impact as an 'exact' mode…) was (Author: michaelsembwever): [~mkjellman], have forced pushed the branch again. (let me know if you want to be adding checkpoint commits rather than overwriting the existing commit.) This adds the test file {{test/resources/tokenization/world_cities_a.csv}}, and a unit test to match. The other unit test methods have been updated to use different delimiters as appropriate for the existing test data files. Example corridor testing… {code:java} create table test ( one text, two int, three text, PRIMARY KEY (one,two) ); # insert a new row, with the contents of test/resources/tokenization/world_cities_a.csv going into column 'three'. create CUSTOM INDEX on test (three) USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = { 'analyzer_class': 'org.apache.cassandra.index.sasi.analyzer.DelimiterAnalyzer', 'delimiter': ',', 'mode': 'prefix', 'analyzed': 'true'}; select one,two from test where three LIKE 'azzazl' ALLOW FILTERING; {code} Aside: this tokenizer raises the need for a "exact" mode. Querying a csv inside a column like this is one example where the user may never require wildcarding LIKE clause (using %) and an 'exact' mode would be significantly more performant and use less disk. > SASI tokenizer for simple delimiter based entries > ------------------------------------------------- > > Key: CASSANDRA-14247 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14247 > Project: Cassandra > Issue Type: Improvement > Components: sasi > Reporter: mck > Assignee: mck > Priority: Major > Fix For: 4.0, 3.11.x > > > Currently SASI offers only two tokenizer options: > - NonTokenizerAnalyser > - StandardAnalyzer > The latter is built upon Snowball, powerful for human languages but overkill > for simple tokenization. > A simple tokenizer is proposed here. The need for this arose as a workaround > of CASSANDRA-11182, and to avoid the disk usage explosion when having to > resort to {{CONTAINS}}. See https://github.com/openzipkin/zipkin/issues/1861 > Example use of this would be: > {code} > CREATE CUSTOM INDEX span_annotation_query_idx > ON zipkin2.span (annotation_query) USING > 'org.apache.cassandra.index.sasi.SASIIndex' > WITH OPTIONS = { > 'analyzer_class': > 'org.apache.cassandra.index.sasi.analyzer.DelimiterAnalyzer', > 'delimiter': '░', > 'case_sensitive': 'true', > 'mode': 'prefix', > 'analyzed': 'true'}; > {code} > Original credit for this work goes to https://github.com/zuochangan -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org