Indhumathi27 commented on a change in pull request #3689: [CARBONDATA-3680]Add Secondary Index Document URL: https://github.com/apache/carbondata/pull/3689#discussion_r401374771
########## File path: docs/index/secondary-index-guide.md ########## @@ -0,0 +1,155 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData Secondary Index + +* [Quick Example](#quick-example) +* [Secondary Index Table](#Secondary-Index-Introduction) +* [Loading Data](#loading-data) +* [Querying Data](#querying-data) +* [Compaction](#compacting-SI-table) +* [Data Management](#data-management-with-SI-tables) + +## Quick example + +Start spark-sql in terminal and run the following queries, +``` +CREATE TABLE maintable(a int, b string, c string) stored as carbondata; +insert into maintable select 1, 'ab', 'cd'; +CREATE index inex1 on table maintable(c) AS 'carbondata'; +SELECT a from maintable where c = 'cd'; +// NOTE: run explain query and check if query hits the SI table from the plan +EXPLAIN SELECT a from maintable where c = 'cd'; +``` + +## Secondary Index Introduction + Sencondary index tables are created as a child table and manager as tables internally by Carbondata. + User can create the SI tables based on the column position in table(Recommended for right columns) + and the queries should have filter on that column to improve the filter query performance. + + SI tables will always be loaded non-lazy way. Once SI table is created, Carbondata's + CarbonOptimizer with the help of CarbonSITransformationRule, transforms the query plan to hit the + SI table based on the filter condition or set of filter conditions present in the query. + So first level of pruning will be done on SI table as it stores blocklets and main table/parent + table pruning will be based on the SI output, which helps in giving the faster query results with + better pruning. + + For instance, main table called **sales** which is defined as + + ``` + CREATE TABLE sales ( + order_time timestamp, + user_id string, + sex string, + country string, + quantity int, + price bigint) + STORED AS carbondata + ``` + + User can create SI table using the Create Index DDL + + ``` + CREATE INDEX index_sales + ON TABLE sales(user_id) + AS + 'carbondata' + TBLPROPERTIES('table_blocksize'='1') + ``` +**NOTE**: + * Secondary Index tables are registered with hive and stored n HiveSERDEPROPERTIES s json formatted + value. Maximum characters supported for SERDEPROPERTIES by hive is 4000 characters which cannot be + changed. + + +#### How SI tables are selected + +When a user filter query is submitted, during query planning phase, CarbonData with help of Review comment: ```suggestion When a user executes a filter query, during query planning phase, CarbonData with help of ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
