[jira] [Created] (CARBONDATA-527) Greater than/less-than/Like filters optimization for dictionary columns
Sujith created CARBONDATA-527: - Summary: Greater than/less-than/Like filters optimization for dictionary columns Key: CARBONDATA-527 URL: https://issues.apache.org/jira/browse/CARBONDATA-527 Project: CarbonData Issue Type: Improvement Reporter: Sujith Current design In greater than/less-than/Like filters, system first iterates each row present in the dictionary cache for identifying valid filter actual members by applying the filter expression , once evaluation done system will hold the list of identified valid filter actual member values(String), now in next step again system will look up the dictionary cache in order to identify the dictionary surrogate values of the identified members. this look up is an additional cost to our system even though the look up methodology is an binary search in dictionary cache. Proposed design/solution: Identify the dictionary surrogate values in filter expression evaluation step itself when actual dictionary values will be scanned for identifying valid filter members . Keep a dictionary counter variable which will be increased when system iterates through the dictionary cache in order to retrieve each actual member stored in dictionary cache , after this system will evaluate each row against the filter expression to identify whether its a valid filter member or not, while doing this process itself counter value can be taken as valid selected dictionary value since the actual member values and its dictionary values will be kept in same order in dictionary cache as the iteration order. thus it will eliminate the further dictionary look up step which is required to retrieve the dictionary surrogate value against identified actual valid filter member. this can also increase significantly the filter query performance of such filter queries which require expression evaluation to identify it the filter members by looking up dictionary cache, like greater than/less-than/Like filters . Note : this optimization is applicable for dictionary columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-382) Like Filter Query Optimization for Dictionary Columns
Sujith created CARBONDATA-382: - Summary: Like Filter Query Optimization for Dictionary Columns Key: CARBONDATA-382 URL: https://issues.apache.org/jira/browse/CARBONDATA-382 Project: CarbonData Issue Type: Improvement Reporter: Sujith Assignee: Sujith Like Filter Query Optimization for Dictionary Columns a) Added pushdown mechanism for the Like filters like startsWith,endsWith and contains so that the respective filters will be processed in Carbon layer itself. b) This mechanism can provide significant gain in the performance of Like filter queries applied in the dictionary since block and blocklet level pruning will be done in the carbon layer before applying the filter. c) Since three new expressions has been added in carbon layer the carbon will be applying the expression only once for startsWith/endsWith/contains filter query, this will make the dictionary lookup also once for applying the expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-278) IS NULL and IS NOT NULL shall be push down to carbon
Sujith created CARBONDATA-278: - Summary: IS NULL and IS NOT NULL shall be push down to carbon Key: CARBONDATA-278 URL: https://issues.apache.org/jira/browse/CARBONDATA-278 Project: CarbonData Issue Type: Improvement Reporter: Sujith Assignee: Sujith Priority: Minor IS NULL and IS NOT NULL shall be push down to carbon layer since carbon layer can process these filters faster using block/block-let pruning , also while processing filters in executers carbon is applying binary search for applying filter values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-270) [Filter Optimization] double data type value comparison optimization
Sujith created CARBONDATA-270: - Summary: [Filter Optimization] double data type value comparison optimization Key: CARBONDATA-270 URL: https://issues.apache.org/jira/browse/CARBONDATA-270 Project: CarbonData Issue Type: Improvement Components: core Reporter: Sujith Assignee: Sujith Priority: Minor EqualsToExpression evaluation for double values first check for the equality of nan values and then the double value comparison happens, since nan comparison scenarios are rare we can push the comparison of nan after the double value comparison. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-114) Decimal Precision and scale getting lost for Complex type columns while describing and querying
Sujith created CARBONDATA-114: - Summary: Decimal Precision and scale getting lost for Complex type columns while describing and querying Key: CARBONDATA-114 URL: https://issues.apache.org/jira/browse/CARBONDATA-114 Project: CarbonData Issue Type: Bug Reporter: Sujith Assignee: Sujith While describing/querying the complex type dimension decimal columns system was not able to maintain the scale/precision, always default precision has been taken which needs to be handled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-99) Complex type column filters with like and not like not working
Sujith created CARBONDATA-99: Summary: Complex type column filters with like and not like not working Key: CARBONDATA-99 URL: https://issues.apache.org/jira/browse/CARBONDATA-99 Project: CarbonData Issue Type: Bug Reporter: Sujith Assignee: Sujith Priority: Minor For complex type filter queries if query contains filter expression rather than BinaryExpression the system was not able to get the dimensions which are involved in the particular filter expression for executing complex type filter column expressions. because of this reason filter was failing Sample Query select test3[1] from complex_filter where test4[1] not like'%1%' order by test1 select test2[0] from complex_filter where test3[0] like '%1234%' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CARBONDATA-76) Not Equals filter display even the null members while filtering non null values
[ https://issues.apache.org/jira/browse/CARBONDATA-76?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sujith reassigned CARBONDATA-76: Assignee: Sujith Will handle > Not Equals filter display even the null members while filtering non null > values > --- > > Key: CARBONDATA-76 > URL: https://issues.apache.org/jira/browse/CARBONDATA-76 > Project: CarbonData > Issue Type: Bug >Reporter: Sujith >Assignee: Sujith >Priority: Minor > > if the user was applying filte to list down non null members. When user > applies Not Equals filter in any non null members the system shall not > display null members in report as per Hive compatibility. > Query Sample: > select > imei,name,country,city,productdate,enddate,age,task,sale,num,level,quest,pointid,score > from big_int where age != 2147; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-76) Not Equals filter display even the null members while filtering non null values
[ https://issues.apache.org/jira/browse/CARBONDATA-76?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sujith updated CARBONDATA-76: - Priority: Minor (was: Major) Description: if the user was applying filte to list down non null members. When user applies Not Equals filter in any non null members the system shall not display null members in report as per Hive compatibility. Query Sample: select imei,name,country,city,productdate,enddate,age,task,sale,num,level,quest,pointid,score from big_int where age != 2147; was:if the user was applying filte to list down non null members. When user applies Not Equals filter in any non null members the system shall not display null members in report as per Hive compatibility. > Not Equals filter display even the null members while filtering non null > values > --- > > Key: CARBONDATA-76 > URL: https://issues.apache.org/jira/browse/CARBONDATA-76 > Project: CarbonData > Issue Type: Bug >Reporter: Sujith >Priority: Minor > > if the user was applying filte to list down non null members. When user > applies Not Equals filter in any non null members the system shall not > display null members in report as per Hive compatibility. > Query Sample: > select > imei,name,country,city,productdate,enddate,age,task,sale,num,level,quest,pointid,score > from big_int where age != 2147; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CARBONDATA-66) Filter was failing when join condition is been applied between two tables
[ https://issues.apache.org/jira/browse/CARBONDATA-66?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sujith reassigned CARBONDATA-66: Assignee: Sujith Will handle this issue regarding join. > Filter was failing when join condition is been applied between two tables > - > > Key: CARBONDATA-66 > URL: https://issues.apache.org/jira/browse/CARBONDATA-66 > Project: CarbonData > Issue Type: Bug >Reporter: Sujith >Assignee: Sujith >Priority: Minor > > Filter was failing when join condition is been applied between two tables in > case of timestamp,bigint and big decimal data type columns. > Query Sample: > select b.* from big_int_basicc a join big_int_basic11 b on > a.productdate=b.productdate").show() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-66) Filter was failing when join condition is been applied between two tables
Sujith created CARBONDATA-66: Summary: Filter was failing when join condition is been applied between two tables Key: CARBONDATA-66 URL: https://issues.apache.org/jira/browse/CARBONDATA-66 Project: CarbonData Issue Type: Bug Reporter: Sujith Priority: Minor Filter was failing when join condition is been applied between two tables in case of timestamp,bigint and big decimal data type columns. Query Sample: select b.* from big_int_basicc a join big_int_basic11 b on a.productdate=b.productdate").show() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
