[GitHub] carbondata issue #1584: [CARBONDATA-1827][WIP] Added S3 Implementation
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1584 Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1178/ ---
[GitHub] carbondata issue #1584: [CARBONDATA-1827][WIP] Added S3 Implementation
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1584 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2401/ ---
[jira] [Created] (CARBONDATA-1947) fix select * issue after compaction, delete and clean files operation
Akash R Nilugal created CARBONDATA-1947: --- Summary: fix select * issue after compaction, delete and clean files operation Key: CARBONDATA-1947 URL: https://issues.apache.org/jira/browse/CARBONDATA-1947 Project: CarbonData Issue Type: Bug Reporter: Akash R Nilugal Assignee: Akash R Nilugal Priority: Minor All data is deleted from compacted segment if a record is deleted and clean file command is run. 1: create table tt2(id int,name string) stored by 'carbondata'; 2: insert into tt2 select 1,'abc'; 3: insert into tt2 select 2,'pqr'; 4: insert into tt2 select 3,'mno'; 5: insert into tt2 select 4,'ghi' 6: Alter table tt2 compact 'minor'; 7: clean files for table tt2; 8: delete from tt2 where id=3; 9: clean files for table tt2; 10: select * from tt2; select query gives empty result -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata pull request #1735: [CARBONDATA-1947]fix select * issue after com...
GitHub user akashrn5 opened a pull request: https://github.com/apache/carbondata/pull/1735 [CARBONDATA-1947]fix select * issue after compaction, delete and clean files operation **Problem**: fix select * issue after compaction, delete and clean files operation **Analysis**: during compaction the timestamp of fact file and the load starttime present in metadata details are different for the same segments, so when the delete operation is executed and after that clean files is called, since there is a timestamp difference, it will delete the carbondata file which inturn gives empty result for select query. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [X] Any interfaces changed? NA - [X] Any backward compatibility impacted? NA - [X] Document update required? NA - [X] Testing done Please provide details on - Units test cases are added to test the scenario - [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/akashrn5/incubator-carbondata clean_after_delete Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1735.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1735 commit 47e9a7753473936607f54321d7b0602fcf50e90a Author: akashrn5 Date: 2017-12-28T08:13:15Z fix select * issue after compaction, delete and clean files operation ---
[GitHub] carbondata issue #1734: [CARBONDATA-1912] Handling lock issues for alter ren...
Github user ManoharVanam commented on the issue: https://github.com/apache/carbondata/pull/1734 retest sdv please ---
[GitHub] carbondata issue #1734: [CARBONDATA-1912] Handling lock issues for alter ren...
Github user ManoharVanam commented on the issue: https://github.com/apache/carbondata/pull/1734 retest this please ---
[GitHub] carbondata issue #1731: [CARBONDATA-1943][PARTITION] Fix Load static partiti...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1731 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2582/ ---
[GitHub] carbondata issue #1735: [CARBONDATA-1947]fix select * issue after compaction...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1735 Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1179/ ---
[GitHub] carbondata issue #1678: [CARBONDATA-1903] Fix code issues in carbondata
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/1678 retest this please ---
[GitHub] carbondata issue #1732: [CARBONDATA-1946] Exception thrown after alter data ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1732 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2583/ ---
[GitHub] carbondata issue #1723: [CARBONDATA-1939] Added show segments validation tes...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/1723 LGTM ---
[GitHub] carbondata pull request #1723: [CARBONDATA-1939] Added show segments validat...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/1723 ---
[jira] [Resolved] (CARBONDATA-1939) Added show segments validation test case
[ https://issues.apache.org/jira/browse/CARBONDATA-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manish Gupta resolved CARBONDATA-1939. -- Resolution: Fixed Fix Version/s: 1.3.0 > Added show segments validation test case > > > Key: CARBONDATA-1939 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1939 > Project: CarbonData > Issue Type: Improvement >Reporter: dhatchayani >Assignee: dhatchayani >Priority: Minor > Fix For: 1.3.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > (1) Modified headers of show segments > (2) Modified SDV test cases for validating headers and result -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1703: [CARBONDATA-1917] While loading, check for stale dic...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/1703 @dhatchayani ...From my perspective I think this PR is not required. This is a case when user deletes the dictionary and sort index file intentionally from back end from the system and keeps only the dictionary meta file. But in any case user should not be allowed to delete any of the carbon store files. Therefore I think we should not handle this scenario. ---
[GitHub] carbondata issue #1735: [CARBONDATA-1947]fix select * issue after compaction...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1735 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2402/ ---
[GitHub] carbondata issue #1735: [CARBONDATA-1947]fix select * issue after compaction...
Github user akashrn5 commented on the issue: https://github.com/apache/carbondata/pull/1735 retest this please ---
[GitHub] carbondata pull request #1703: [CARBONDATA-1917] While loading, check for st...
Github user dhatchayani closed the pull request at: https://github.com/apache/carbondata/pull/1703 ---
[GitHub] carbondata issue #1718: [CARBONDATA-1929][Validation]carbon property configu...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/1718 @mohammadshahidkhan ..IDG update is required for the min and max values of the properties in this PR ---
[GitHub] carbondata issue #1734: [CARBONDATA-1912] Handling lock issues for alter ren...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1734 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2403/ ---
[GitHub] carbondata issue #1678: [CARBONDATA-1903] Fix code issues in carbondata
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1678 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2404/ ---
[GitHub] carbondata issue #1733: [CARBONDATA-1944][PARTITION]Special character like c...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1733 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2584/ ---
[jira] [Assigned] (CARBONDATA-1905) alter streaming table to normal table
[ https://issues.apache.org/jira/browse/CARBONDATA-1905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] QiangCai reassigned CARBONDATA-1905: Assignee: QiangCai > alter streaming table to normal table > - > > Key: CARBONDATA-1905 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1905 > Project: CarbonData > Issue Type: Sub-task >Reporter: QiangCai >Assignee: QiangCai > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (CARBONDATA-1904) auto handoff streaming segment
[ https://issues.apache.org/jira/browse/CARBONDATA-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] QiangCai reassigned CARBONDATA-1904: Assignee: QiangCai > auto handoff streaming segment > -- > > Key: CARBONDATA-1904 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1904 > Project: CarbonData > Issue Type: Sub-task >Reporter: QiangCai >Assignee: QiangCai > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata pull request #1736: [CARBONDATA-1904][CARBONDATA-1905] Support au...
GitHub user QiangCai opened a pull request: https://github.com/apache/carbondata/pull/1736 [CARBONDATA-1904][CARBONDATA-1905] Support auto handoff and close streaming 1. auto handoff streaming segment 2. alter streaming table to normal table alter table compact 'close_streaming' - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/QiangCai/carbondata auto_compaction Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1736.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1736 commit bfc37e358a0d872cc9ed57fa46f992bb2ffe126e Author: QiangCai Date: 2017-12-28T10:43:43Z support auto handoff and close streaming ---
[GitHub] carbondata pull request #1707: [CARBONDATA-1839] [DataLoad] Fix bugs and opt...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1707#discussion_r158927577
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/holder/UnsafeSortTempFileChunkHolder.java
---
@@ -296,11 +200,12 @@ private void fillDataForPrefetch() {
* @throws CarbonSortKeyAndGroupByException
*/
private Object[] getRowFromStream() throws
CarbonSortKeyAndGroupByException {
-Object[] row = new Object[dimensionCount + measureCount];
+Object[] row = new Object[sortParameters.getDimColCount()
++ sortParameters.getComplexDimColCount() +
sortParameters.getMeasureColCount()];
--- End diff --
Fine~ I also find the same issues in other files and will fix it at the
same time.
---
[GitHub] carbondata pull request #1707: [CARBONDATA-1839] [DataLoad] Fix bugs and opt...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1707#discussion_r158927598
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/holder/UnsafeSortTempFileChunkHolder.java
---
@@ -312,7 +217,7 @@ private void fillDataForPrefetch() {
}
// write complex dimensions here.
- for (; dimCount < dimensionCount; dimCount++) {
+ for (; dimCount < sortParameters.getComplexDimColCount();
dimCount++) {
--- End diff --
yeah, I made a mistake when applying the changes to this PR.
---
[GitHub] carbondata issue #1707: [CARBONDATA-1839] [DataLoad] Fix bugs and optimize i...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/1707 @ravipesala Thanks for your carefully review. I've fixed the issues and tested in my local cluster to make sure it's OK. ---
[GitHub] carbondata issue #1735: [CARBONDATA-1947]fix select * issue after compaction...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1735 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2405/ ---
[GitHub] carbondata issue #1735: [CARBONDATA-1947]fix select * issue after compaction...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1735 Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1180/ ---
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1694#discussion_r158930636 --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggregateExpressions.scala --- @@ -0,0 +1,44 @@ +package org.apache.carbondata.integration.spark.testsuite.preaggregate --- End diff -- add license header ---
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158930733
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggregateExpressions.scala
---
@@ -0,0 +1,44 @@
+package org.apache.carbondata.integration.spark.testsuite.preaggregate
+
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+class TestPreAggregateExpressions extends QueryTest with BeforeAndAfterAll
{
+
+ override def beforeAll: Unit = {
+sql("drop table if exists mainTable")
+sql("CREATE TABLE mainTable(id int, name string, city string, age
string) STORED BY 'org.apache.carbondata.format'")
+sql("create datamap agg0 on table mainTable using 'preaggregate' as
select name,count(age) from mainTable group by name")
+sql("create datamap agg1 on table mainTable using 'preaggregate' as
select name,sum(case when age=35 then id else 0 end) from mainTable group by
name")
+sql("create datamap agg2 on table mainTable using 'preaggregate' as
select name,sum(case when age=35 then id else 0 end),city from mainTable group
by name,city")
--- End diff --
format the SQL string to make it good nice, they should be capital and
separate into different lines correctly
---
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158930929
--- Diff:
integration/spark-common/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchemaCommon.scala
---
@@ -467,10 +474,13 @@ class TableNewProcessor(cm: TableModel) {
// Sort columns should be at the begin of all columns
cm.sortKeyDims.get.foreach { keyDim =>
val field = cm.dimCols.find(keyDim equals _.column).get
- val encoders = if (cm.parentTable.isDefined &&
cm.dataMapRelation.get.get(field).isDefined) {
+ val encoders = if (cm.parentTable.isDefined &&
+ cm.dataMapRelation.get.get(field).isDefined &&
+
cm.dataMapRelation.get.get(field).get.columnTableRelationList.size==1 ) {
--- End diff --
add space before and after `=`
---
[GitHub] carbondata issue #1584: [CARBONDATA-1827][WIP] Added S3 Implementation
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1584 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2406/ ---
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158931027
--- Diff:
integration/spark-common/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchemaCommon.scala
---
@@ -492,11 +502,12 @@ class TableNewProcessor(cm: TableModel) {
val sortField = cm.sortKeyDims.get.find(field.column equals _)
if (sortField.isEmpty) {
val encoders = if (cm.parentTable.isDefined &&
- cm.dataMapRelation.get.get(field).isDefined) {
+ cm.dataMapRelation.get.get(field).isDefined &&
+
cm.dataMapRelation.get.get(field).get.columnTableRelationList.size==1) {
--- End diff --
add space before and after =
---
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158931327
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala
---
@@ -126,19 +126,33 @@ object PreAggregateUtil {
attr.aggregateFunction,
parentTableName,
parentDatabaseName,
- parentTableId)
+ parentTableId,
+ "column_" + counter)
+counter = counter + 1
case attr: AttributeReference =>
+val columnRelation = getColumnRelation(attr.name,
+ parentTableId,
+ parentTableName,
+ parentDatabaseName,
+ carbonTable)
+val arrayBuffer = new ArrayBuffer[ColumnTableRelation]()
+arrayBuffer += columnRelation
fieldToDataMapFieldMap += getField(attr.name,
attr.dataType,
- parentColumnId = carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
parentTableName = parentTableName,
- parentDatabaseName = parentDatabaseName, parentTableId =
parentTableId)
+ columnTableRelationList = arrayBuffer.toList)
case Alias(attr: AttributeReference, _) =>
+val columnRelation = getColumnRelation(attr.name,
--- End diff --
move `attr.name` to next line, do the same for all place in this function
---
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158931409
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala
---
@@ -147,6 +161,34 @@ object PreAggregateUtil {
fieldToDataMapFieldMap
}
+ /**
+ * Below method will be used to get the column relation
+ * with the parent column which will be used during query and data
loading
+ * @param parentColumnName
+ * parent column name
--- End diff --
why not keep in same line?
---
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158931680
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggregateExpressions.scala
---
@@ -0,0 +1,44 @@
+package org.apache.carbondata.integration.spark.testsuite.preaggregate
+
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+class TestPreAggregateExpressions extends QueryTest with BeforeAndAfterAll
{
+
+ override def beforeAll: Unit = {
+sql("drop table if exists mainTable")
+sql("CREATE TABLE mainTable(id int, name string, city string, age
string) STORED BY 'org.apache.carbondata.format'")
+sql("create datamap agg0 on table mainTable using 'preaggregate' as
select name,count(age) from mainTable group by name")
+sql("create datamap agg1 on table mainTable using 'preaggregate' as
select name,sum(case when age=35 then id else 0 end) from mainTable group by
name")
+sql("create datamap agg2 on table mainTable using 'preaggregate' as
select name,sum(case when age=35 then id else 0 end),city from mainTable group
by name,city")
+sql("create datamap agg3 on table mainTable using 'preaggregate' as
select name,sum(case when age=27 then id else 0 end) from mainTable group by
name")
--- End diff --
I think these `CREATE DATAMAP` statement should be run inside the test
function, not in beforeAll
---
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158932043
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggregateExpressions.scala
---
@@ -0,0 +1,44 @@
+package org.apache.carbondata.integration.spark.testsuite.preaggregate
+
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+class TestPreAggregateExpressions extends QueryTest with BeforeAndAfterAll
{
+
+ override def beforeAll: Unit = {
+sql("drop table if exists mainTable")
+sql("CREATE TABLE mainTable(id int, name string, city string, age
string) STORED BY 'org.apache.carbondata.format'")
+sql("create datamap agg0 on table mainTable using 'preaggregate' as
select name,count(age) from mainTable group by name")
+sql("create datamap agg1 on table mainTable using 'preaggregate' as
select name,sum(case when age=35 then id else 0 end) from mainTable group by
name")
+sql("create datamap agg2 on table mainTable using 'preaggregate' as
select name,sum(case when age=35 then id else 0 end),city from mainTable group
by name,city")
+sql("create datamap agg3 on table mainTable using 'preaggregate' as
select name,sum(case when age=27 then id else 0 end) from mainTable group by
name")
+sql("create datamap agg4 on table mainTable using 'preaggregate' as
select name,sum(case when age=27 then id else 0 end), sum(case when age=35 then
id else 0 end) from mainTable group by name")
+sql(s"LOAD DATA LOCAL INPATH
'$resourcesPath/measureinsertintotest.csv' into table mainTable")
+ }
+
+ test("test pre agg create table with expression 1") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg0"), true,
"maintable_age_count")
+ }
+
+ test("test pre agg create table with expression 2") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg1"), true,
"maintable_column_0_sum")
+ }
+
+ test("test pre agg create table with expression 3") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg2"), true,
"maintable_column_0_sum")
+ }
+
+ test("test pre agg create table with expression 4") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg3"), true,
"maintable_column_0_sum")
+ }
+
+ test("test pre agg create table with expression 5") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg4"), true,
"maintable_column_0_sum")
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg4"), true,
"maintable_column_1_sum")
+ }
+
--- End diff --
Can you add some more test case:
1. nested case when (case when inside case when)
2. case when and with filter when creating datamap. Filter column is part
of the group by column
---
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158932166
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggregateExpressions.scala
---
@@ -0,0 +1,44 @@
+package org.apache.carbondata.integration.spark.testsuite.preaggregate
+
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+class TestPreAggregateExpressions extends QueryTest with BeforeAndAfterAll
{
+
+ override def beforeAll: Unit = {
+sql("drop table if exists mainTable")
+sql("CREATE TABLE mainTable(id int, name string, city string, age
string) STORED BY 'org.apache.carbondata.format'")
+sql("create datamap agg0 on table mainTable using 'preaggregate' as
select name,count(age) from mainTable group by name")
+sql("create datamap agg1 on table mainTable using 'preaggregate' as
select name,sum(case when age=35 then id else 0 end) from mainTable group by
name")
+sql("create datamap agg2 on table mainTable using 'preaggregate' as
select name,sum(case when age=35 then id else 0 end),city from mainTable group
by name,city")
+sql("create datamap agg3 on table mainTable using 'preaggregate' as
select name,sum(case when age=27 then id else 0 end) from mainTable group by
name")
+sql("create datamap agg4 on table mainTable using 'preaggregate' as
select name,sum(case when age=27 then id else 0 end), sum(case when age=35 then
id else 0 end) from mainTable group by name")
+sql(s"LOAD DATA LOCAL INPATH
'$resourcesPath/measureinsertintotest.csv' into table mainTable")
+ }
+
+ test("test pre agg create table with expression 1") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg0"), true,
"maintable_age_count")
+ }
+
+ test("test pre agg create table with expression 2") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg1"), true,
"maintable_column_0_sum")
+ }
+
+ test("test pre agg create table with expression 3") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg2"), true,
"maintable_column_0_sum")
+ }
+
+ test("test pre agg create table with expression 4") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg3"), true,
"maintable_column_0_sum")
--- End diff --
I think you need to verify the select query will hit the pre-agg table, not
just check it is exist.
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Github user geetikagupta16 commented on the issue: https://github.com/apache/carbondata/pull/1650 retest this please ---
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158932405
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala
---
@@ -166,127 +208,160 @@ object PreAggregateUtil {
aggFunctions: AggregateFunction,
parentTableName: String,
parentDatabaseName: String,
- parentTableId: String) : scala.collection.mutable.ListBuffer[(Field,
DataMapField)] = {
+ parentTableId: String,
+ newColumnName: String) : scala.collection.mutable.ListBuffer[(Field,
DataMapField)] = {
--- End diff --
what is this field? please add comment of this function
---
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158932436
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala
---
@@ -166,127 +208,160 @@ object PreAggregateUtil {
aggFunctions: AggregateFunction,
parentTableName: String,
parentDatabaseName: String,
- parentTableId: String) : scala.collection.mutable.ListBuffer[(Field,
DataMapField)] = {
+ parentTableId: String,
+ newColumnName: String) : scala.collection.mutable.ListBuffer[(Field,
DataMapField)] = {
val list = scala.collection.mutable.ListBuffer.empty[(Field,
DataMapField)]
aggFunctions match {
- case sum@Sum(attr: AttributeReference) =>
-list += getField(attr.name,
- attr.dataType,
- sum.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case sum@Sum(Cast(attr: AttributeReference, changeDataType:
DataType)) =>
-list += getField(attr.name,
+ case sum@Sum(MatchCastExpression(exp: Expression, changeDataType:
DataType)) =>
+list += getFieldForAggregateExpression(exp,
changeDataType,
- sum.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case count@Count(Seq(attr: AttributeReference)) =>
-list += getField(attr.name,
- attr.dataType,
- count.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case count@Count(Seq(Cast(attr: AttributeReference, _))) =>
-list += getField(attr.name,
- attr.dataType,
- count.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case min@Min(attr: AttributeReference) =>
-list += getField(attr.name,
- attr.dataType,
- min.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case min@Min(Cast(attr: AttributeReference, changeDataType:
DataType)) =>
-list += getField(attr.name,
+ carbonTable,
+ newColumnName,
+ sum.prettyName)
+ case sum@Sum(exp: Expression) =>
+list += getFieldForAggregateExpression(exp,
--- End diff --
move first parameter to next line
---
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158932905
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala
---
@@ -126,19 +126,33 @@ object PreAggregateUtil {
attr.aggregateFunction,
parentTableName,
parentDatabaseName,
- parentTableId)
+ parentTableId,
+ "column_" + counter)
+counter = counter + 1
case attr: AttributeReference =>
+val columnRelation = getColumnRelation(attr.name,
+ parentTableId,
+ parentTableName,
+ parentDatabaseName,
+ carbonTable)
+val arrayBuffer = new ArrayBuffer[ColumnTableRelation]()
+arrayBuffer += columnRelation
fieldToDataMapFieldMap += getField(attr.name,
attr.dataType,
- parentColumnId = carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
parentTableName = parentTableName,
- parentDatabaseName = parentDatabaseName, parentTableId =
parentTableId)
+ columnTableRelationList = arrayBuffer.toList)
case Alias(attr: AttributeReference, _) =>
+val columnRelation = getColumnRelation(attr.name,
--- End diff --
I think `getColumnRelation` is no need, you can create a new
ColumnTableRelation directly here, parameter is almost the same
---
[GitHub] carbondata issue #1678: [CARBONDATA-1903] Fix code issues in carbondata
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1678 Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1181/ ---
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158933435
--- Diff:
integration/spark-common/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchemaCommon.scala
---
@@ -79,7 +79,7 @@ case class Field(column: String, var dataType:
Option[String], name: Option[Stri
}
case class DataMapField(var aggregateFunction: String = "",
-columnTableRelation: Option[ColumnTableRelation] = None) {
+columnTableRelationList: Option[List[ColumnTableRelation]] = None) {
--- End diff --
Use Seq instead of List
---
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158933775
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala
---
@@ -166,127 +208,160 @@ object PreAggregateUtil {
aggFunctions: AggregateFunction,
parentTableName: String,
parentDatabaseName: String,
- parentTableId: String) : scala.collection.mutable.ListBuffer[(Field,
DataMapField)] = {
+ parentTableId: String,
+ newColumnName: String) : scala.collection.mutable.ListBuffer[(Field,
DataMapField)] = {
val list = scala.collection.mutable.ListBuffer.empty[(Field,
DataMapField)]
aggFunctions match {
- case sum@Sum(attr: AttributeReference) =>
-list += getField(attr.name,
- attr.dataType,
- sum.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case sum@Sum(Cast(attr: AttributeReference, changeDataType:
DataType)) =>
-list += getField(attr.name,
+ case sum@Sum(MatchCastExpression(exp: Expression, changeDataType:
DataType)) =>
+list += getFieldForAggregateExpression(exp,
changeDataType,
- sum.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case count@Count(Seq(attr: AttributeReference)) =>
-list += getField(attr.name,
- attr.dataType,
- count.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case count@Count(Seq(Cast(attr: AttributeReference, _))) =>
-list += getField(attr.name,
- attr.dataType,
- count.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case min@Min(attr: AttributeReference) =>
-list += getField(attr.name,
- attr.dataType,
- min.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case min@Min(Cast(attr: AttributeReference, changeDataType:
DataType)) =>
-list += getField(attr.name,
+ carbonTable,
+ newColumnName,
+ sum.prettyName)
+ case sum@Sum(exp: Expression) =>
+list += getFieldForAggregateExpression(exp,
+ sum.dataType,
+ carbonTable,
+ newColumnName,
+ sum.prettyName)
+ case count@Count(Seq(MatchCastExpression(exp: Expression,
changeDataType: DataType))) =>
+list += getFieldForAggregateExpression(exp,
changeDataType,
- min.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case max@Max(attr: AttributeReference) =>
-list += getField(attr.name,
- attr.dataType,
- max.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case max@Max(Cast(attr: AttributeReference, changeDataType:
DataType)) =>
-list += getField(attr.name,
+ carbonTable,
+ newColumnName,
+ count.prettyName)
+ case count@Count(Seq(expression: Expression)) =>
+list += getFieldForAggregateExpression(expression,
+ count.dataType,
+ carbonTable,
+ newColumnName,
+ count.prettyName)
+ case min@Min(MatchCastExpression(exp: Expression, changeDataType:
DataType)) =>
+list += getFieldForAggregateExpression(exp,
changeDataType,
- max.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case Average(attr: AttributeReference) =>
-list += getField(attr.name,
- attr.dataType,
- "sum",
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
-
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158933818
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala
---
@@ -166,127 +208,160 @@ object PreAggregateUtil {
aggFunctions: AggregateFunction,
parentTableName: String,
parentDatabaseName: String,
- parentTableId: String) : scala.collection.mutable.ListBuffer[(Field,
DataMapField)] = {
+ parentTableId: String,
+ newColumnName: String) : scala.collection.mutable.ListBuffer[(Field,
DataMapField)] = {
val list = scala.collection.mutable.ListBuffer.empty[(Field,
DataMapField)]
aggFunctions match {
- case sum@Sum(attr: AttributeReference) =>
-list += getField(attr.name,
- attr.dataType,
- sum.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case sum@Sum(Cast(attr: AttributeReference, changeDataType:
DataType)) =>
-list += getField(attr.name,
+ case sum@Sum(MatchCastExpression(exp: Expression, changeDataType:
DataType)) =>
+list += getFieldForAggregateExpression(exp,
changeDataType,
- sum.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case count@Count(Seq(attr: AttributeReference)) =>
-list += getField(attr.name,
- attr.dataType,
- count.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case count@Count(Seq(Cast(attr: AttributeReference, _))) =>
-list += getField(attr.name,
- attr.dataType,
- count.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case min@Min(attr: AttributeReference) =>
-list += getField(attr.name,
- attr.dataType,
- min.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case min@Min(Cast(attr: AttributeReference, changeDataType:
DataType)) =>
-list += getField(attr.name,
+ carbonTable,
+ newColumnName,
+ sum.prettyName)
+ case sum@Sum(exp: Expression) =>
+list += getFieldForAggregateExpression(exp,
+ sum.dataType,
+ carbonTable,
+ newColumnName,
+ sum.prettyName)
+ case count@Count(Seq(MatchCastExpression(exp: Expression,
changeDataType: DataType))) =>
+list += getFieldForAggregateExpression(exp,
changeDataType,
- min.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case max@Max(attr: AttributeReference) =>
-list += getField(attr.name,
- attr.dataType,
- max.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case max@Max(Cast(attr: AttributeReference, changeDataType:
DataType)) =>
-list += getField(attr.name,
+ carbonTable,
+ newColumnName,
+ count.prettyName)
+ case count@Count(Seq(expression: Expression)) =>
+list += getFieldForAggregateExpression(expression,
+ count.dataType,
+ carbonTable,
+ newColumnName,
+ count.prettyName)
+ case min@Min(MatchCastExpression(exp: Expression, changeDataType:
DataType)) =>
+list += getFieldForAggregateExpression(exp,
changeDataType,
- max.prettyName,
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
- parentDatabaseName, parentTableId = parentTableId)
- case Average(attr: AttributeReference) =>
-list += getField(attr.name,
- attr.dataType,
- "sum",
- carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
- parentTableName,
-
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158933879
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala
---
@@ -126,19 +126,33 @@ object PreAggregateUtil {
attr.aggregateFunction,
parentTableName,
parentDatabaseName,
- parentTableId)
+ parentTableId,
+ "column_" + counter)
+counter = counter + 1
case attr: AttributeReference =>
+val columnRelation = getColumnRelation(attr.name,
+ parentTableId,
+ parentTableName,
+ parentDatabaseName,
+ carbonTable)
+val arrayBuffer = new ArrayBuffer[ColumnTableRelation]()
+arrayBuffer += columnRelation
fieldToDataMapFieldMap += getField(attr.name,
attr.dataType,
- parentColumnId = carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
parentTableName = parentTableName,
- parentDatabaseName = parentDatabaseName, parentTableId =
parentTableId)
+ columnTableRelationList = arrayBuffer.toList)
case Alias(attr: AttributeReference, _) =>
+val columnRelation = getColumnRelation(attr.name,
+ parentTableId,
+ parentTableName,
+ parentDatabaseName,
+ carbonTable)
+val arrayBuffer = new ArrayBuffer[ColumnTableRelation]()
--- End diff --
This is not needed after changing List to Seq
---
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158934001
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala
---
@@ -508,7 +583,10 @@ object PreAggregateUtil {
val headers =
dataMapSchemas.find(_.getChildSchema.getTableName.equalsIgnoreCase(
dataMapIdentifier.table)) match {
case Some(dataMapSchema) =>
-
dataMapSchema.getChildSchema.getListOfColumns.asScala.sortBy(_.getSchemaOrdinal).map(
+val columns = dataMapSchema.getChildSchema.getListOfColumns.asScala
+ .filter{f =>
--- End diff --
change `f` to a meaningful variable, add space before `{`
---
[GitHub] carbondata pull request #1728: [CARBONDATA-1926][Pre-Aggregate] Expression s...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1728#discussion_r158934173
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggregateExpressions.scala
---
@@ -0,0 +1,105 @@
+package org.apache.carbondata.integration.spark.testsuite.preaggregate
+
+import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Row}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.hive.CarbonRelation
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+class TestPreAggregateExpressions extends QueryTest with BeforeAndAfterAll
{
+
+ override def beforeAll: Unit = {
+sql("drop table if exists mainTable")
+sql("CREATE TABLE mainTable(id int, name string, city string, age
string) STORED BY 'org.apache.carbondata.format'")
+sql("create datamap agg0 on table mainTable using 'preaggregate' as
select name,count(age) from mainTable group by name")
+sql("create datamap agg1 on table mainTable using 'preaggregate' as
select name,sum(case when age=35 then id else 0 end) from mainTable group by
name")
+sql("create datamap agg2 on table mainTable using 'preaggregate' as
select name,sum(case when age=35 then id else 0 end),city from mainTable group
by name,city")
+sql("create datamap agg3 on table mainTable using 'preaggregate' as
select name,sum(case when age=27 then id else 0 end) from mainTable group by
name")
+sql("create datamap agg4 on table mainTable using 'preaggregate' as
select name,sum(case when age=27 then id else 0 end), sum(case when age=35 then
id else 0 end) from mainTable group by name")
+sql(s"LOAD DATA LOCAL INPATH
'$resourcesPath/measureinsertintotest.csv' into table mainTable")
+ }
+
+ test("test pre agg create table with expression 1") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg0"), true,
"maintable_age_count")
+ }
+
+ test("test pre agg create table with expression 2") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg1"), true,
"maintable_column_0_sum")
+ }
+
+ test("test pre agg create table with expression 3") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg2"), true,
"maintable_column_0_sum")
+ }
+
+ test("test pre agg create table with expression 4") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg3"), true,
"maintable_column_0_sum")
+ }
+
+ test("test pre agg create table with expression 5") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg4"), true,
"maintable_column_0_sum")
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg4"), true,
"maintable_column_1_sum")
+ }
+
+ test("test pre agg table selection with expression 1") {
+val df = sql("select name as NewName, count(age) as sum from mainTable
group by name order by name")
+preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0")
+ }
+
+
+ test("test pre agg table selection with expression 2") {
+val df = sql("select name as NewName, sum(case when age=35 then id
else 0 end) as sum from mainTable group by name order by name")
+preAggTableValidator(df.queryExecution.analyzed, "maintable_agg1")
+ }
+
+ test("test pre agg table selection with expression 3") {
+val df = sql("select sum(case when age=35 then id else 0 end) from
maintable")
+checkAnswer(df, Seq(Row(6.0)))
+ }
+
+ test("test pre agg table selection with expression 4") {
+val df = sql("select sum(case when age=27 then id else 0 end) from
maintable")
+preAggTableValidator(df.queryExecution.analyzed, "maintable_agg3")
+checkAnswer(df, Seq(Row(2.0)))
+ }
+
+ test("test pre agg table selection with expression 5") {
+val df = sql("select sum(case when age=27 then id else 0 end),
sum(case when age=35 then id else 0 end) from maintable")
+preAggTableValidator(df.queryExecution.analyzed, "maintable_agg4")
+checkAnswer(df, Seq(Row(2.0,6.0)))
+ }
+
+ def preAggTableValidator(plan: LogicalPlan, actualTableName: String) :
Unit ={
--- End diff --
add comment for this function
---
[GitHub] carbondata issue #1736: [CARBONDATA-1904][CARBONDATA-1905] Support auto hand...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1736 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2407/ ---
[GitHub] carbondata pull request #1728: [CARBONDATA-1926][Pre-Aggregate] Expression s...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1728#discussion_r158934260
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggregateExpressions.scala
---
@@ -0,0 +1,105 @@
+package org.apache.carbondata.integration.spark.testsuite.preaggregate
+
+import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Row}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.hive.CarbonRelation
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+class TestPreAggregateExpressions extends QueryTest with BeforeAndAfterAll
{
+
+ override def beforeAll: Unit = {
+sql("drop table if exists mainTable")
+sql("CREATE TABLE mainTable(id int, name string, city string, age
string) STORED BY 'org.apache.carbondata.format'")
+sql("create datamap agg0 on table mainTable using 'preaggregate' as
select name,count(age) from mainTable group by name")
+sql("create datamap agg1 on table mainTable using 'preaggregate' as
select name,sum(case when age=35 then id else 0 end) from mainTable group by
name")
+sql("create datamap agg2 on table mainTable using 'preaggregate' as
select name,sum(case when age=35 then id else 0 end),city from mainTable group
by name,city")
+sql("create datamap agg3 on table mainTable using 'preaggregate' as
select name,sum(case when age=27 then id else 0 end) from mainTable group by
name")
+sql("create datamap agg4 on table mainTable using 'preaggregate' as
select name,sum(case when age=27 then id else 0 end), sum(case when age=35 then
id else 0 end) from mainTable group by name")
+sql(s"LOAD DATA LOCAL INPATH
'$resourcesPath/measureinsertintotest.csv' into table mainTable")
+ }
+
+ test("test pre agg create table with expression 1") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg0"), true,
"maintable_age_count")
+ }
+
+ test("test pre agg create table with expression 2") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg1"), true,
"maintable_column_0_sum")
+ }
+
+ test("test pre agg create table with expression 3") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg2"), true,
"maintable_column_0_sum")
+ }
+
+ test("test pre agg create table with expression 4") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg3"), true,
"maintable_column_0_sum")
+ }
+
+ test("test pre agg create table with expression 5") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg4"), true,
"maintable_column_0_sum")
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg4"), true,
"maintable_column_1_sum")
+ }
+
+ test("test pre agg table selection with expression 1") {
+val df = sql("select name as NewName, count(age) as sum from mainTable
group by name order by name")
+preAggTableValidator(df.queryExecution.analyzed, "maintable_agg0")
+ }
+
+
+ test("test pre agg table selection with expression 2") {
+val df = sql("select name as NewName, sum(case when age=35 then id
else 0 end) as sum from mainTable group by name order by name")
+preAggTableValidator(df.queryExecution.analyzed, "maintable_agg1")
+ }
+
+ test("test pre agg table selection with expression 3") {
+val df = sql("select sum(case when age=35 then id else 0 end) from
maintable")
+checkAnswer(df, Seq(Row(6.0)))
+ }
+
+ test("test pre agg table selection with expression 4") {
+val df = sql("select sum(case when age=27 then id else 0 end) from
maintable")
+preAggTableValidator(df.queryExecution.analyzed, "maintable_agg3")
+checkAnswer(df, Seq(Row(2.0)))
+ }
+
+ test("test pre agg table selection with expression 5") {
+val df = sql("select sum(case when age=27 then id else 0 end),
sum(case when age=35 then id else 0 end) from maintable")
+preAggTableValidator(df.queryExecution.analyzed, "maintable_agg4")
+checkAnswer(df, Seq(Row(2.0,6.0)))
+ }
+
--- End diff --
add a testcase for subquery also
---
[GitHub] carbondata issue #1584: [CARBONDATA-1827][WIP] Added S3 Implementation
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1584 Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1182/ ---
[GitHub] carbondata pull request #1728: [CARBONDATA-1926][Pre-Aggregate] Expression s...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1728#discussion_r158934363 --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonPreAggregateRules.scala --- @@ -126,16 +127,17 @@ case class CarbonPreAggregateQueryRules(sparkSession: SparkSession) extends Rule aggregateExp, carbonTable, tableName, -list) - carbonTable +list, +aggregateExpressions) + (carbonTable, logicalRelation) // below case for handling filter query // When plan has grouping expression, aggregate expression // filter expression case Aggregate(groupingExp, - aggregateExp, - Filter(filterExp, - CarbonSubqueryAlias(_, logicalRelation: LogicalRelation))) +aggregateExp, --- End diff -- unnecessary change ---
[GitHub] carbondata issue #1709: [CARBONDATA-1774] [PrestoIntegration] Not able to fe...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1709 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2585/ ---
[jira] [Assigned] (CARBONDATA-1941) Document update for Lock Retry
[ https://issues.apache.org/jira/browse/CARBONDATA-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gururaj Shetty reassigned CARBONDATA-1941: -- Assignee: Shahid K I > Document update for Lock Retry > -- > > Key: CARBONDATA-1941 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1941 > Project: CarbonData > Issue Type: Task >Reporter: Gururaj Shetty >Assignee: Shahid K I >Priority: Minor > Time Spent: 1h 50m > Remaining Estimate: 0h > > Add documentation for lock retry > Property Name: carbon.lock.retries > Default Value: 3 > Details: Specifies the maximum number of retries to obtain the lock for any > operations other than load. > Poperty Name: carbon.lock.retry.timeout.sec > Default Value: 5 > Details: Specifies the interval between the retries to obtain the lock for > any operation other than load. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (CARBONDATA-1942) Documentation for Concurrent Lock Retries
[ https://issues.apache.org/jira/browse/CARBONDATA-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gururaj Shetty reassigned CARBONDATA-1942: -- Assignee: Ayush Anubhava (was: Gururaj Shetty) > Documentation for Concurrent Lock Retries > - > > Key: CARBONDATA-1942 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1942 > Project: CarbonData > Issue Type: Task >Reporter: Gururaj Shetty >Assignee: Ayush Anubhava >Priority: Minor > Fix For: 1.3.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Add documenation for concurrent lock retries > Property Name: carbon.concurrent.lock.retries > Default Value: 100 > Deatils: Specifies the maximum number of retries to obtain the lock for > concurrent operations. This is used for concurrent loading. > > Property Name: carbon.concurrent.lock.retry.timeout.sec > Default Value: 1 > Details: Specifies the interval between the retries to obtain the lock for > concurrent operations. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158935184
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggregateExpressions.scala
---
@@ -0,0 +1,44 @@
+package org.apache.carbondata.integration.spark.testsuite.preaggregate
+
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+class TestPreAggregateExpressions extends QueryTest with BeforeAndAfterAll
{
+
+ override def beforeAll: Unit = {
+sql("drop table if exists mainTable")
+sql("CREATE TABLE mainTable(id int, name string, city string, age
string) STORED BY 'org.apache.carbondata.format'")
+sql("create datamap agg0 on table mainTable using 'preaggregate' as
select name,count(age) from mainTable group by name")
+sql("create datamap agg1 on table mainTable using 'preaggregate' as
select name,sum(case when age=35 then id else 0 end) from mainTable group by
name")
+sql("create datamap agg2 on table mainTable using 'preaggregate' as
select name,sum(case when age=35 then id else 0 end),city from mainTable group
by name,city")
+sql("create datamap agg3 on table mainTable using 'preaggregate' as
select name,sum(case when age=27 then id else 0 end) from mainTable group by
name")
+sql("create datamap agg4 on table mainTable using 'preaggregate' as
select name,sum(case when age=27 then id else 0 end), sum(case when age=35 then
id else 0 end) from mainTable group by name")
+sql(s"LOAD DATA LOCAL INPATH
'$resourcesPath/measureinsertintotest.csv' into table mainTable")
+ }
+
+ test("test pre agg create table with expression 1") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg0"), true,
"maintable_age_count")
+ }
+
+ test("test pre agg create table with expression 2") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg1"), true,
"maintable_column_0_sum")
+ }
+
+ test("test pre agg create table with expression 3") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg2"), true,
"maintable_column_0_sum")
+ }
+
+ test("test pre agg create table with expression 4") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg3"), true,
"maintable_column_0_sum")
--- End diff --
This PR is only for create and load support for expression inside aggregate
function...Verify the query result and which pre aggregate table it will it
will be validated as a part of different pr
---
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158935305
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggregateExpressions.scala
---
@@ -0,0 +1,44 @@
+package org.apache.carbondata.integration.spark.testsuite.preaggregate
+
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+class TestPreAggregateExpressions extends QueryTest with BeforeAndAfterAll
{
+
+ override def beforeAll: Unit = {
+sql("drop table if exists mainTable")
+sql("CREATE TABLE mainTable(id int, name string, city string, age
string) STORED BY 'org.apache.carbondata.format'")
+sql("create datamap agg0 on table mainTable using 'preaggregate' as
select name,count(age) from mainTable group by name")
+sql("create datamap agg1 on table mainTable using 'preaggregate' as
select name,sum(case when age=35 then id else 0 end) from mainTable group by
name")
+sql("create datamap agg2 on table mainTable using 'preaggregate' as
select name,sum(case when age=35 then id else 0 end),city from mainTable group
by name,city")
+sql("create datamap agg3 on table mainTable using 'preaggregate' as
select name,sum(case when age=27 then id else 0 end) from mainTable group by
name")
+sql("create datamap agg4 on table mainTable using 'preaggregate' as
select name,sum(case when age=27 then id else 0 end), sum(case when age=35 then
id else 0 end) from mainTable group by name")
+sql(s"LOAD DATA LOCAL INPATH
'$resourcesPath/measureinsertintotest.csv' into table mainTable")
+ }
+
+ test("test pre agg create table with expression 1") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg0"), true,
"maintable_age_count")
+ }
+
+ test("test pre agg create table with expression 2") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg1"), true,
"maintable_column_0_sum")
+ }
+
+ test("test pre agg create table with expression 3") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg2"), true,
"maintable_column_0_sum")
+ }
+
+ test("test pre agg create table with expression 4") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg3"), true,
"maintable_column_0_sum")
+ }
+
+ test("test pre agg create table with expression 5") {
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg4"), true,
"maintable_column_0_sum")
+checkExistence(sql("DESCRIBE FORMATTED mainTable_agg4"), true,
"maintable_column_1_sum")
+ }
+
--- End diff --
Filter Column part of group by column testcases already present in
TestPreAggregateTableSelection
---
[GitHub] carbondata issue #1727: [CARBONDATA-1702] [Segment-Reading] added doc for "q...
Github user sgururajshetty commented on the issue: https://github.com/apache/carbondata/pull/1727 @praveenmeenakshi56 please follow the same structure like ALTER TABLE Heading Description Syntax Properties (if any) Example ---
[GitHub] carbondata issue #1709: [CARBONDATA-1774] [PrestoIntegration] Not able to fe...
Github user anubhav100 commented on the issue: https://github.com/apache/carbondata/pull/1709 retest this please ---
[GitHub] carbondata issue #1707: [CARBONDATA-1839] [DataLoad] Fix bugs and optimize i...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1707 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2408/ ---
[GitHub] carbondata issue #1736: [CARBONDATA-1904][CARBONDATA-1905] Support auto hand...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1736 Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1183/ ---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1650 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2409/ ---
[jira] [Commented] (CARBONDATA-1572) Support Streaming Ingest
[ https://issues.apache.org/jira/browse/CARBONDATA-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16305430#comment-16305430 ] zhouguangcheng commented on CARBONDATA-1572: [~QiangCai] why streaming table donot support delete segment ? I think it is a usefully feature. > Support Streaming Ingest > > > Key: CARBONDATA-1572 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1572 > Project: CarbonData > Issue Type: New Feature >Reporter: QiangCai >Assignee: QiangCai > Attachments: CarbonData Streaming Ingest_v1.4.pdf, CarbonData > Streaming Ingest_v1.5.pdf > > Time Spent: 14h 20m > Remaining Estimate: 0h > > CarbonData should support streaming ingest. > [^CarbonData Streaming Ingest_v1.5.pdf] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata pull request #1737: [CARBONDATA-1922] Documentation - Updated doc...
GitHub user brahmareddybattula opened a pull request: https://github.com/apache/carbondata/pull/1737 [CARBONDATA-1922] Documentation - Updated document with Ignoring empty line OPTION Documentation - Updated document with Ignoring empty line OPTION You can merge this pull request into a Git repository by running: $ git pull https://github.com/brahmareddybattula/carbondata master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1737.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1737 commit 69361d8144c3b49f8c898e060c45129ebfab3e86 Author: Brahma Reddy Battula Date: 2017-12-28T13:09:12Z [CARBONDATA-1922] Documentation - Updated document with Ignoring empty line OPTION Updated product document with Ignoring empty line OPTION commit 55e3483af01915f43465fa39ad531f69618fddba Author: Brahma Reddy Battula Date: 2017-12-28T13:14:32Z [CARBONDATA-1922] Documentation - Updated document with Ignoring empty line OPTION Updated product document with Ignoring empty line OPTION ---
[GitHub] carbondata issue #1737: [CARBONDATA-1922] Documentation - Updated document w...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1737 Can one of the admins verify this patch? ---
[GitHub] carbondata issue #1709: [CARBONDATA-1774] [PrestoIntegration] Not able to fe...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1709 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2586/ ---
[GitHub] carbondata issue #1707: [CARBONDATA-1839] [DataLoad] Fix bugs and optimize i...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1707 Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1184/ ---
[GitHub] carbondata issue #1709: [CARBONDATA-1774] [PrestoIntegration] Not able to fe...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1709 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2410/ ---
[GitHub] carbondata issue #1737: [CARBONDATA-1922] Documentation - Updated document w...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1737 Can one of the admins verify this patch? ---
[GitHub] carbondata issue #1709: [CARBONDATA-1774] [PrestoIntegration] Not able to fe...
Github user anubhav100 commented on the issue: https://github.com/apache/carbondata/pull/1709 retest this please ---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1650 Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1185/ ---
[GitHub] carbondata issue #1737: [CARBONDATA-1922] Documentation - Updated document w...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1737 Can one of the admins verify this patch? ---
[GitHub] carbondata issue #1668: [CARBONDATA-1787] Updated data-management-on-carbond...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1668 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2411/ ---
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158948765
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala
---
@@ -166,127 +208,160 @@ object PreAggregateUtil {
aggFunctions: AggregateFunction,
parentTableName: String,
parentDatabaseName: String,
- parentTableId: String) : scala.collection.mutable.ListBuffer[(Field,
DataMapField)] = {
+ parentTableId: String,
+ newColumnName: String) : scala.collection.mutable.ListBuffer[(Field,
DataMapField)] = {
--- End diff --
fixed
---
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1694#discussion_r158948738
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala
---
@@ -126,19 +126,33 @@ object PreAggregateUtil {
attr.aggregateFunction,
parentTableName,
parentDatabaseName,
- parentTableId)
+ parentTableId,
+ "column_" + counter)
+counter = counter + 1
case attr: AttributeReference =>
+val columnRelation = getColumnRelation(attr.name,
+ parentTableId,
+ parentTableName,
+ parentDatabaseName,
+ carbonTable)
+val arrayBuffer = new ArrayBuffer[ColumnTableRelation]()
+arrayBuffer += columnRelation
fieldToDataMapFieldMap += getField(attr.name,
attr.dataType,
- parentColumnId = carbonTable.getColumnByName(parentTableName,
attr.name).getColumnId,
parentTableName = parentTableName,
- parentDatabaseName = parentDatabaseName, parentTableId =
parentTableId)
+ columnTableRelationList = arrayBuffer.toList)
case Alias(attr: AttributeReference, _) =>
+val columnRelation = getColumnRelation(attr.name,
+ parentTableId,
+ parentTableName,
+ parentDatabaseName,
+ carbonTable)
+val arrayBuffer = new ArrayBuffer[ColumnTableRelation]()
--- End diff --
fixed
---
[GitHub] carbondata pull request #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to...
Github user kumarvishal09 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1694#discussion_r158948779 --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggregateExpressions.scala --- @@ -0,0 +1,44 @@ +package org.apache.carbondata.integration.spark.testsuite.preaggregate --- End diff -- fixed ---
[GitHub] carbondata issue #1688: [CARBONDATA-1916]Correct the database location path ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1688 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2412/ ---
[GitHub] carbondata issue #1709: [CARBONDATA-1774] [PrestoIntegration] Not able to fe...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1709 Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1186/ ---
[GitHub] carbondata issue #1703: [CARBONDATA-1917] While loading, check for stale dic...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1703 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2587/ ---
[GitHub] carbondata issue #1718: [CARBONDATA-1929][Validation]carbon property configu...
Github user mohammadshahidkhan commented on the issue: https://github.com/apache/carbondata/pull/1718 retest this please ---
[GitHub] carbondata issue #1733: [CARBONDATA-1944][PARTITION]Special character like c...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/1733 Can you separate this PR into two? It is better to move the CarbonInputFormat deletion into another PR. ---
[GitHub] carbondata pull request #1729: [CARBONDATA-1936][PARTITION] Corrected bad re...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1729#discussion_r158954562
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
---
@@ -484,161 +485,147 @@ case class CarbonLoadDataCommand(
// converted to hive standard fomat to let spark understand the data
to partition.
val serializationNullFormat =
carbonLoadModel.getSerializationNullFormat.split(CarbonCommonConstants.COMMA,
2)(1)
-val failAction =
- carbonLoadModel.getBadRecordsAction.split(",")(1).equalsIgnoreCase(
-CarbonCommonConstants.CARBON_BAD_RECORDS_ACTION_DEFAULT)
-val ignoreAction =
-
carbonLoadModel.getBadRecordsAction.split(",")(1).equalsIgnoreCase("ignore")
-val query: LogicalPlan = if (dataFrame.isDefined) {
- var timeStampformatString =
CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT
- val timeStampFormat = new SimpleDateFormat(timeStampformatString)
- var dateFormatString =
CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT
- val dateFormat = new SimpleDateFormat(dateFormatString)
- val delimiterLevel1 = carbonLoadModel.getComplexDelimiterLevel1
- val delimiterLevel2 = carbonLoadModel.getComplexDelimiterLevel2
- val serializationNullFormat =
-
carbonLoadModel.getSerializationNullFormat.split(CarbonCommonConstants.COMMA,
2)(1)
- val attributes =
-StructType(dataFrame.get.schema.fields.map(_.copy(dataType =
StringType))).toAttributes
- val len = attributes.length
- val rdd = dataFrame.get.rdd.map { f =>
-val data = new Array[Any](len)
-var i = 0
-while (i < len) {
- data(i) =
-UTF8String.fromString(
- CarbonScalaUtil.getString(f.get(i),
-serializationNullFormat,
-delimiterLevel1,
-delimiterLevel2,
-timeStampFormat,
-dateFormat))
- i = i + 1
+val badRecordAction =
+ carbonLoadModel.getBadRecordsAction.split(",")(1)
+var timeStampformatString = carbonLoadModel.getTimestampformat
+if (timeStampformatString.isEmpty) {
+ timeStampformatString = carbonLoadModel.getDefaultTimestampFormat
+}
+val timeStampFormat = new SimpleDateFormat(timeStampformatString)
+var dateFormatString = carbonLoadModel.getDateFormat
+if (dateFormatString.isEmpty) {
+ dateFormatString = carbonLoadModel.getDefaultDateFormat
+}
+val dateFormat = new SimpleDateFormat(dateFormatString)
+
CarbonSession.threadSet(CarbonLoadOptionConstants.CARBON_OPTIONS_DATEFORMAT,
dateFormatString)
+CarbonSession.threadSet(
+ CarbonLoadOptionConstants.CARBON_OPTIONS_TIMESTAMPFORMAT,
+ timeStampformatString)
+CarbonSession.threadSet(
+ CarbonLoadOptionConstants.CARBON_OPTIONS_SERIALIZATION_NULL_FORMAT,
+ serializationNullFormat)
+CarbonSession.threadSet(
+ CarbonLoadOptionConstants.CARBON_OPTIONS_BAD_RECORDS_ACTION,
+ badRecordAction)
+CarbonSession.threadSet(
+ CarbonLoadOptionConstants.CARBON_OPTIONS_IS_EMPTY_DATA_BAD_RECORD,
+ carbonLoadModel.getIsEmptyDataBadRecord.split(",")(1))
+try {
+ val query: LogicalPlan = if (dataFrame.isDefined) {
+val delimiterLevel1 = carbonLoadModel.getComplexDelimiterLevel1
+val delimiterLevel2 = carbonLoadModel.getComplexDelimiterLevel2
+val attributes =
+ StructType(dataFrame.get.schema.fields.map(_.copy(dataType =
StringType))).toAttributes
+val len = attributes.length
+val rdd = dataFrame.get.rdd.map { f =>
+ val data = new Array[Any](len)
+ var i = 0
+ while (i < len) {
+data(i) =
+ UTF8String.fromString(
+CarbonScalaUtil.getString(f.get(i),
+ serializationNullFormat,
+ delimiterLevel1,
+ delimiterLevel2,
+ timeStampFormat,
+ dateFormat))
+i = i + 1
+ }
+ InternalRow.fromSeq(data)
}
-InternalRow.fromSeq(data)
- }
- if (updateModel.isDefined) {
-sparkSession.sparkContext.setLocalProperty(EXECUTION_ID_KEY, null)
-// In case of update, we don't need the segmrntid column in case
of partitioning
-val dropAttributes = attributes.dropRight(1)
-val finalOutput = catalogTable.schema.map { attr =>
- dropAttributes.find { d =>
-val index = d.name.lastIndexOf("-updatedColumn")
-
[GitHub] carbondata pull request #1729: [CARBONDATA-1936][PARTITION] Corrected bad re...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1729#discussion_r158954662
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
---
@@ -484,161 +485,147 @@ case class CarbonLoadDataCommand(
// converted to hive standard fomat to let spark understand the data
to partition.
val serializationNullFormat =
carbonLoadModel.getSerializationNullFormat.split(CarbonCommonConstants.COMMA,
2)(1)
-val failAction =
- carbonLoadModel.getBadRecordsAction.split(",")(1).equalsIgnoreCase(
-CarbonCommonConstants.CARBON_BAD_RECORDS_ACTION_DEFAULT)
-val ignoreAction =
-
carbonLoadModel.getBadRecordsAction.split(",")(1).equalsIgnoreCase("ignore")
-val query: LogicalPlan = if (dataFrame.isDefined) {
- var timeStampformatString =
CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT
- val timeStampFormat = new SimpleDateFormat(timeStampformatString)
- var dateFormatString =
CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT
- val dateFormat = new SimpleDateFormat(dateFormatString)
- val delimiterLevel1 = carbonLoadModel.getComplexDelimiterLevel1
- val delimiterLevel2 = carbonLoadModel.getComplexDelimiterLevel2
- val serializationNullFormat =
-
carbonLoadModel.getSerializationNullFormat.split(CarbonCommonConstants.COMMA,
2)(1)
- val attributes =
-StructType(dataFrame.get.schema.fields.map(_.copy(dataType =
StringType))).toAttributes
- val len = attributes.length
- val rdd = dataFrame.get.rdd.map { f =>
-val data = new Array[Any](len)
-var i = 0
-while (i < len) {
- data(i) =
-UTF8String.fromString(
- CarbonScalaUtil.getString(f.get(i),
-serializationNullFormat,
-delimiterLevel1,
-delimiterLevel2,
-timeStampFormat,
-dateFormat))
- i = i + 1
+val badRecordAction =
+ carbonLoadModel.getBadRecordsAction.split(",")(1)
+var timeStampformatString = carbonLoadModel.getTimestampformat
+if (timeStampformatString.isEmpty) {
+ timeStampformatString = carbonLoadModel.getDefaultTimestampFormat
+}
+val timeStampFormat = new SimpleDateFormat(timeStampformatString)
+var dateFormatString = carbonLoadModel.getDateFormat
+if (dateFormatString.isEmpty) {
+ dateFormatString = carbonLoadModel.getDefaultDateFormat
+}
+val dateFormat = new SimpleDateFormat(dateFormatString)
+
CarbonSession.threadSet(CarbonLoadOptionConstants.CARBON_OPTIONS_DATEFORMAT,
dateFormatString)
+CarbonSession.threadSet(
+ CarbonLoadOptionConstants.CARBON_OPTIONS_TIMESTAMPFORMAT,
+ timeStampformatString)
+CarbonSession.threadSet(
+ CarbonLoadOptionConstants.CARBON_OPTIONS_SERIALIZATION_NULL_FORMAT,
+ serializationNullFormat)
+CarbonSession.threadSet(
+ CarbonLoadOptionConstants.CARBON_OPTIONS_BAD_RECORDS_ACTION,
+ badRecordAction)
+CarbonSession.threadSet(
+ CarbonLoadOptionConstants.CARBON_OPTIONS_IS_EMPTY_DATA_BAD_RECORD,
+ carbonLoadModel.getIsEmptyDataBadRecord.split(",")(1))
+try {
+ val query: LogicalPlan = if (dataFrame.isDefined) {
+val delimiterLevel1 = carbonLoadModel.getComplexDelimiterLevel1
+val delimiterLevel2 = carbonLoadModel.getComplexDelimiterLevel2
+val attributes =
+ StructType(dataFrame.get.schema.fields.map(_.copy(dataType =
StringType))).toAttributes
+val len = attributes.length
+val rdd = dataFrame.get.rdd.map { f =>
+ val data = new Array[Any](len)
+ var i = 0
+ while (i < len) {
+data(i) =
+ UTF8String.fromString(
+CarbonScalaUtil.getString(f.get(i),
+ serializationNullFormat,
+ delimiterLevel1,
+ delimiterLevel2,
+ timeStampFormat,
+ dateFormat))
+i = i + 1
+ }
+ InternalRow.fromSeq(data)
}
-InternalRow.fromSeq(data)
- }
- if (updateModel.isDefined) {
-sparkSession.sparkContext.setLocalProperty(EXECUTION_ID_KEY, null)
-// In case of update, we don't need the segmrntid column in case
of partitioning
-val dropAttributes = attributes.dropRight(1)
-val finalOutput = catalogTable.schema.map { attr =>
- dropAttributes.find { d =>
-val index = d.name.lastIndexOf("-updatedColumn")
-
[GitHub] carbondata issue #1731: [CARBONDATA-1943][PARTITION] Fix Load static partiti...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/1731 retest this please ---
[GitHub] carbondata issue #1709: [CARBONDATA-1774] [PrestoIntegration] Not able to fe...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1709 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2413/ ---
[GitHub] carbondata pull request #1734: [CARBONDATA-1912] Handling lock issues for al...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1734#discussion_r158955988
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableRenameCommand.scala
---
@@ -156,15 +156,20 @@ private[sql] case class CarbonAlterTableRenameCommand(
} catch {
case e: Exception =>
LOGGER.error(e, "Rename table failed: " + e.getMessage)
-if (carbonTable != null) {
- AlterTableUtil
-.revertRenameTableChanges(oldTableIdentifier,
- newTableName,
- carbonTable.getTablePath,
- carbonTable.getCarbonTableIdentifier.getTableId,
- timeStamp)(
- sparkSession)
- renameBadRecords(newTableName, oldTableName, oldDatabaseName)
+try {
+ if (carbonTable != null) {
+AlterTableUtil
+ .revertRenameTableChanges(oldTableIdentifier,
--- End diff --
please follow code convention:
```
AlterTableUtil. revertRenameTableChanges(
xxx,
yyy
)
```
---
[GitHub] carbondata pull request #1734: [CARBONDATA-1912] Handling lock issues for al...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1734#discussion_r158956093
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableRenameCommand.scala
---
@@ -156,15 +156,20 @@ private[sql] case class CarbonAlterTableRenameCommand(
} catch {
case e: Exception =>
LOGGER.error(e, "Rename table failed: " + e.getMessage)
-if (carbonTable != null) {
- AlterTableUtil
-.revertRenameTableChanges(oldTableIdentifier,
- newTableName,
- carbonTable.getTablePath,
- carbonTable.getCarbonTableIdentifier.getTableId,
- timeStamp)(
- sparkSession)
- renameBadRecords(newTableName, oldTableName, oldDatabaseName)
+try {
+ if (carbonTable != null) {
+AlterTableUtil
+ .revertRenameTableChanges(oldTableIdentifier,
+newTableName,
+carbonTable.getTablePath,
+carbonTable.getCarbonTableIdentifier.getTableId,
+timeStamp)(
+sparkSession)
+renameBadRecords(newTableName, oldTableName, oldDatabaseName)
+ }
+} catch {
+ case e: Exception =>
--- End diff --
use finally, without catch
---
[GitHub] carbondata pull request #1730: [CARBONDATA-1937][PARTITION] Fix partition fe...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1730#discussion_r158956384 --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java --- @@ -1501,6 +1501,15 @@ public static final String TIMESERIES_HIERARCHY = "timeseries.hierarchy"; + /** + * It allows queries on hive metastore directly along with filter information, otherwise first + * fetches all partitions from hive and apply filters on it. --- End diff -- Can you mention how to decide whether to set it to false or true? ---
[GitHub] carbondata pull request #1730: [CARBONDATA-1937][PARTITION] Fix partition fe...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1730#discussion_r158956569
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonFilters.scala
---
@@ -405,13 +407,60 @@ object CarbonFilters {
}
}
+ /**
+ * Fetches partition information from hive
+ * @param partitionFilters
+ * @param sparkSession
+ * @param identifier
+ * @return
+ */
def getPartitions(partitionFilters: Seq[Expression],
sparkSession: SparkSession,
identifier: TableIdentifier): Seq[String] = {
-val partitions =
- sparkSession.sessionState.catalog.listPartitionsByFilter(identifier,
partitionFilters)
+val partitions = {
+ try {
+if (CarbonProperties.getInstance().
+
getProperty(CarbonCommonConstants.CARBON_READ_PARTITION_HIVE_DIRECT,
+
CarbonCommonConstants.CARBON_READ_PARTITION_HIVE_DIRECT_DEFAULT).toBoolean) {
+
sparkSession.sessionState.catalog.listPartitionsByFilter(identifier,
partitionFilters)
+} else {
+ getPartitionsAlternate(partitionFilters, sparkSession,
identifier)
+}
+ } catch {
+case e: Exception =>
+ // Get partition information alternatively.
+ getPartitionsAlternate(partitionFilters, sparkSession,
identifier)
+ }
+}
partitions.toList.flatMap { partition =>
partition.spec.seq.map{case (column, value) => column + "=" + value}
}.toSet.toSeq
}
+
+ /**
+ * This is alternate way of getting partition information. It first
fetches all partitions from
+ * hive and then apply filter instead of querying hive along with
filters.
+ * @param partitionFilters
--- End diff --
give comment for parameter
---
[GitHub] carbondata pull request #1730: [CARBONDATA-1937][PARTITION] Fix partition fe...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1730#discussion_r158956514
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonFilters.scala
---
@@ -405,13 +407,60 @@ object CarbonFilters {
}
}
+ /**
+ * Fetches partition information from hive
+ * @param partitionFilters
+ * @param sparkSession
+ * @param identifier
+ * @return
+ */
def getPartitions(partitionFilters: Seq[Expression],
sparkSession: SparkSession,
identifier: TableIdentifier): Seq[String] = {
-val partitions =
- sparkSession.sessionState.catalog.listPartitionsByFilter(identifier,
partitionFilters)
+val partitions = {
+ try {
+if (CarbonProperties.getInstance().
+
getProperty(CarbonCommonConstants.CARBON_READ_PARTITION_HIVE_DIRECT,
+
CarbonCommonConstants.CARBON_READ_PARTITION_HIVE_DIRECT_DEFAULT).toBoolean) {
+
sparkSession.sessionState.catalog.listPartitionsByFilter(identifier,
partitionFilters)
+} else {
+ getPartitionsAlternate(partitionFilters, sparkSession,
identifier)
+}
+ } catch {
+case e: Exception =>
+ // Get partition information alternatively.
+ getPartitionsAlternate(partitionFilters, sparkSession,
identifier)
+ }
+}
partitions.toList.flatMap { partition =>
partition.spec.seq.map{case (column, value) => column + "=" + value}
}.toSet.toSeq
}
+
+ /**
+ * This is alternate way of getting partition information. It first
fetches all partitions from
+ * hive and then apply filter instead of querying hive along with
filters.
+ * @param partitionFilters
+ * @param sparkSession
+ * @param identifier
+ * @return
+ */
+ private def getPartitionsAlternate(partitionFilters: Seq[Expression],
--- End diff --
move `partitionFilters` to next line, please follow this in future
---
[GitHub] carbondata pull request #1735: [CARBONDATA-1947]fix select * issue after com...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1735#discussion_r158956869
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/DeleteCarbonTableTestCase.scala
---
@@ -165,6 +165,20 @@ class DeleteCarbonTableTestCase extends QueryTest with
BeforeAndAfterAll {
sql("drop table if exists preaggmain_preagg1")
}
+ test("test select query after compaction, delete and clean files") {
+sql("drop table if exists select_after_clean")
+sql("create table select_after_clean(id int, name string) stored by
'carbondata'")
+sql("insert into select_after_clean select 1,'abc'")
+sql("insert into select_after_clean select 2,'def'")
+sql("insert into select_after_clean select 3,'uhj'")
+sql("insert into select_after_clean select 4,'frg'")
+sql("alter table select_after_clean compact 'minor'")
+sql("clean files for table select_after_clean")
+sql("delete from select_after_clean where name='def'")
+sql("clean files for table select_after_clean")
+assert(sql("select * from select_after_clean").collect().length == 3)
--- End diff --
can you assert the result using checkAnswer
---
[GitHub] carbondata issue #1668: [CARBONDATA-1787] Updated data-management-on-carbond...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1668 Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1187/ ---
[GitHub] carbondata issue #1709: [CARBONDATA-1774] [PrestoIntegration] Not able to fe...
Github user anubhav100 commented on the issue: https://github.com/apache/carbondata/pull/1709 retest this please ---
[GitHub] carbondata issue #1711: [CARBONDATA-1754][BugFix] Fixed issue occuring on co...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/1711 Please describe what this bug is ---
[jira] [Created] (CARBONDATA-1948) Update help document for the change made for CARBONDATA-1929
Mohammad Shahid Khan created CARBONDATA-1948: Summary: Update help document for the change made for CARBONDATA-1929 Key: CARBONDATA-1948 URL: https://issues.apache.org/jira/browse/CARBONDATA-1948 Project: CarbonData Issue Type: Sub-task Reporter: Mohammad Shahid Khan Assignee: Gururaj Shetty Priority: Minor carbon.sort.file.write.buffer.size (minValue = 10 KB, maxValue=10MB, defaultValue =16 KB ) carbon.sort.intermediate.files.limit (minValue = 2, maxValue=50, defaultValue =20 ) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1694: [CARBONDATA-1925][Pre-Aggregate]Added code to suppor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1694 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2414/ ---
[GitHub] carbondata issue #1709: [CARBONDATA-1774] [PrestoIntegration] Not able to fe...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1709 Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1188/ ---
[GitHub] carbondata issue #1709: [CARBONDATA-1774] [PrestoIntegration] Not able to fe...
Github user anubhav100 commented on the issue: https://github.com/apache/carbondata/pull/1709 @chenliang613 I have rebased the code please review it seems like CI is unstable it keeps on failing but I have clean build on my machine ---
