[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1759 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2694/ ---
[GitHub] carbondata issue #1760: [CARBONDATA-1979] ][IMPLICIT COLUMN] Modified implic...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1760 Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1306/ ---
[GitHub] carbondata issue #1740: [CARBONDATA-1949, CARBONDATA-1950] Fixed bug related...
Github user geetikagupta16 commented on the issue: https://github.com/apache/carbondata/pull/1740 @manishgupta88 I have added the test case. Please check ---
[jira] [Created] (CARBONDATA-1982) Loading data into partition table with invalid partition column should throw proper exception
Geetika Gupta created CARBONDATA-1982:
-
Summary: Loading data into partition table with invalid partition
column should throw proper exception
Key: CARBONDATA-1982
URL: https://issues.apache.org/jira/browse/CARBONDATA-1982
Project: CarbonData
Issue Type: Bug
Components: data-query
Affects Versions: 1.3.0
Environment: spark2.1
Reporter: Geetika Gupta
Priority: Minor
Fix For: 1.3.0
Attachments: 2000_UniqData.csv
I created a partitioned table using:
CREATE TABLE uniqdata_int_dec(CUST_NAME String,ACTIVE_EMUI_VERSION string, DOB
timestamp,
DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,
DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,
INTEGER_COLUMN1 int) Partitioned by (cust_id int, decimal_column1
decimal(30,10)) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES
("TABLE_BLOCKSIZE"= "256 MB")
Load data command:
LOAD DATA INPATH 'hdfs://localhost:54311/2000_UniqData.csv' into table
uniqdata_int_dec partition(cust_id123='1', abc='12345678901.1234') OPTIONS
('FILEHEADER'='CUST_ID,CUST_NAME ,ACTIVE_EMUI_VERSION,DOB,DOJ,
BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,
Double_COLUMN2,INTEGER_COLUMN1','BAD_RECORDS_ACTION'='FORCE');
OUTPUT:
0: jdbc:hive2://localhost:1> LOAD DATA INPATH
'hdfs://localhost:54311/Files/2000_UniqData.csv' into table uniqdata_int_dec
partition(cust_id123='1', decimal_column1='12345678901.1234') OPTIONS
('FILEHEADER'='CUST_ID,CUST_NAME ,ACTIVE_EMUI_VERSION,DOB,DOJ,
BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,
Double_COLUMN2,INTEGER_COLUMN1','BAD_RECORDS_ACTION'='FORCE');
Error: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 (state=,code=0)
The above command throws java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
whereas it should throw a proper exception like invalid column expression for
partition load command.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] carbondata pull request #1752: [CARBONDATA-1972][PARTITION] Fix compaction a...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1752#discussion_r159594673
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForDeleteCommand.scala
---
@@ -77,7 +77,7 @@ private[sql] case class CarbonProjectForDeleteCommand(
dataRdd,
timestamp,
isUpdateOperation = false,
-executorErrors)) {
+executorErrors)._1) {
--- End diff --
It seems always true
---
[GitHub] carbondata pull request #1752: [CARBONDATA-1972][PARTITION] Fix compaction a...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1752#discussion_r159594689
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/datasources/CarbonFileFormat.scala
---
@@ -107,11 +107,19 @@ with Serializable {
CarbonTableOutputFormat.setOverwrite(conf,
options("overwrite").toBoolean)
// Set the update timestamp if user sets in case of update query. It
needs to be updated
// in load status update time
-val updateTimeStamp = options.getOrElse("updatetimestamp", null)
-if (updateTimeStamp != null) {
- conf.set(CarbonTableOutputFormat.UPADTE_TIMESTAMP, updateTimeStamp)
- model.setFactTimeStamp(updateTimeStamp.toLong)
+val updateTimeStamp = options.get("updatetimestamp")
+if (updateTimeStamp.isDefined) {
+ conf.set(CarbonTableOutputFormat.UPADTE_TIMESTAMP,
updateTimeStamp.get)
+ model.setFactTimeStamp(updateTimeStamp.get.toLong)
}
+// In case of update query there is chance to remove the older
segments, so here we can set
+// the to be deleted segments to mark as delete while updating
tablestatus
+val segemntsTobeDeleted = options.get("segmentsToBeDeleted")
+if (segemntsTobeDeleted.isDefined) {
+ conf.set(CarbonTableOutputFormat.SEGMENTS_TO_BE_DELETED,
segemntsTobeDeleted.get)
+}
+
+
--- End diff --
remove empty line
---
[GitHub] carbondata pull request #1752: [CARBONDATA-1972][PARTITION] Fix compaction a...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1752#discussion_r159594614
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/DeleteExecution.scala
---
@@ -58,14 +58,15 @@ object DeleteExecution {
dataRdd: RDD[Row],
timestamp: String,
isUpdateOperation: Boolean,
- executorErrors: ExecutionErrors): Boolean = {
+ executorErrors: ExecutionErrors): (Boolean, Seq[String]) = {
--- End diff --
It seems the first return value is always true
And it is hard to understand this function, can you add comment for it
---
[GitHub] carbondata issue #1755: [CARBONDATA-1976][PARTITION] Support combination of ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1755 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2535/ ---
[GitHub] carbondata pull request #1755: [CARBONDATA-1976][PARTITION] Support combinat...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1755#discussion_r159591690
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
---
@@ -611,7 +611,7 @@ case class CarbonLoadDataCommand(
val output = if (partition.nonEmpty) {
catalogTable.schema.map{ attr =>
attributes.find(_.name.equalsIgnoreCase(attr.name)).get
-}.filter(attr => partition.get(attr.name).isEmpty)
+}.filter(attr => partition.getOrElse(attr.name, None).isEmpty)
--- End diff --
It seems the same as before?
---
[GitHub] carbondata pull request #1755: [CARBONDATA-1976][PARTITION] Support combinat...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1755#discussion_r159592167
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
---
@@ -702,63 +702,71 @@ case class CarbonLoadDataCommand(
sparkSession: SparkSession,
table: CarbonTable,
logicalPlan: LogicalPlan): Unit = {
-sparkSession.sessionState.catalog.listPartitions(
+val existingPartitions =
sparkSession.sessionState.catalog.listPartitions(
TableIdentifier(table.getTableName, Some(table.getDatabaseName)),
- Some(partition.map(f => (f._1, f._2.get
-val partitionNames = partition.map(k => k._1 + "=" + k._2.get).toSet
+ Some(partition.filter(_._2.isDefined).map(f => (f._1, f._2.get
+val partitionNames = existingPartitions.toList.flatMap { partition =>
+ partition.spec.seq.map{case (column, value) => column + "=" + value}
+}.toSet
val uniqueId = System.currentTimeMillis().toString
val segments = new SegmentStatusManager(
table.getAbsoluteTableIdentifier).getValidAndInvalidSegments.getValidSegments
-try {
- // First drop the partitions from partition mapper files of each
segment
- new CarbonDropPartitionRDD(
-sparkSession.sparkContext,
-table.getTablePath,
-segments.asScala,
-partitionNames.toSeq,
-uniqueId).collect()
-} catch {
- case e: Exception =>
-// roll back the drop partitions from carbon store
-new CarbonDropPartitionCommitRDD(
+// If any existing partitions need to be overwritten then drop from
partitionmap
+if (partitionNames.nonEmpty) {
+ try {
+// First drop the partitions from partition mapper files of each
segment
+new CarbonDropPartitionRDD(
sparkSession.sparkContext,
table.getTablePath,
segments.asScala,
- false,
+ partitionNames.toSeq,
uniqueId).collect()
-throw e
-}
+ } catch {
+case e: Exception =>
+ // roll back the drop partitions from carbon store
+ new CarbonDropPartitionCommitRDD(
+sparkSession.sparkContext,
+table.getTablePath,
+segments.asScala,
+false,
+uniqueId).collect()
+ throw e
+ }
-try {
+ try {
+Dataset.ofRows(sparkSession, logicalPlan)
+ } catch {
+case e: Exception =>
+ // roll back the drop partitions from carbon store
+ new CarbonDropPartitionCommitRDD(
+sparkSession.sparkContext,
+table.getTablePath,
+segments.asScala,
+false,
+uniqueId).collect()
+ throw e
+ }
+ // Commit the removed partitions in carbon store.
+ new CarbonDropPartitionCommitRDD(
+sparkSession.sparkContext,
+table.getTablePath,
+segments.asScala,
+true,
+uniqueId).collect()
+ // Update the loadstatus with update time to clear cache from driver.
+ val segmentSet = new util.HashSet[String](new
SegmentStatusManager(table
+
.getAbsoluteTableIdentifier).getValidAndInvalidSegments.getValidSegments)
--- End diff --
make these two lines more readable
---
[GitHub] carbondata pull request #1755: [CARBONDATA-1976][PARTITION] Support combinat...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1755#discussion_r159591992
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
---
@@ -702,63 +702,71 @@ case class CarbonLoadDataCommand(
sparkSession: SparkSession,
table: CarbonTable,
logicalPlan: LogicalPlan): Unit = {
-sparkSession.sessionState.catalog.listPartitions(
+val existingPartitions =
sparkSession.sessionState.catalog.listPartitions(
TableIdentifier(table.getTableName, Some(table.getDatabaseName)),
- Some(partition.map(f => (f._1, f._2.get
-val partitionNames = partition.map(k => k._1 + "=" + k._2.get).toSet
+ Some(partition.filter(_._2.isDefined).map(f => (f._1, f._2.get
+val partitionNames = existingPartitions.toList.flatMap { partition =>
+ partition.spec.seq.map{case (column, value) => column + "=" + value}
+}.toSet
val uniqueId = System.currentTimeMillis().toString
val segments = new SegmentStatusManager(
table.getAbsoluteTableIdentifier).getValidAndInvalidSegments.getValidSegments
-try {
- // First drop the partitions from partition mapper files of each
segment
- new CarbonDropPartitionRDD(
-sparkSession.sparkContext,
-table.getTablePath,
-segments.asScala,
-partitionNames.toSeq,
-uniqueId).collect()
-} catch {
- case e: Exception =>
-// roll back the drop partitions from carbon store
-new CarbonDropPartitionCommitRDD(
+// If any existing partitions need to be overwritten then drop from
partitionmap
+if (partitionNames.nonEmpty) {
+ try {
+// First drop the partitions from partition mapper files of each
segment
+new CarbonDropPartitionRDD(
sparkSession.sparkContext,
table.getTablePath,
segments.asScala,
- false,
+ partitionNames.toSeq,
uniqueId).collect()
-throw e
-}
+ } catch {
+case e: Exception =>
+ // roll back the drop partitions from carbon store
+ new CarbonDropPartitionCommitRDD(
+sparkSession.sparkContext,
+table.getTablePath,
+segments.asScala,
+false,
+uniqueId).collect()
+ throw e
+ }
-try {
+ try {
+Dataset.ofRows(sparkSession, logicalPlan)
+ } catch {
+case e: Exception =>
+ // roll back the drop partitions from carbon store
+ new CarbonDropPartitionCommitRDD(
+sparkSession.sparkContext,
+table.getTablePath,
+segments.asScala,
+false,
+uniqueId).collect()
+ throw e
+ }
+ // Commit the removed partitions in carbon store.
+ new CarbonDropPartitionCommitRDD(
+sparkSession.sparkContext,
+table.getTablePath,
+segments.asScala,
+true,
--- End diff --
provide variable name to improve readability
---
[GitHub] carbondata pull request #1755: [CARBONDATA-1976][PARTITION] Support combinat...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1755#discussion_r159591980
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
---
@@ -702,63 +702,71 @@ case class CarbonLoadDataCommand(
sparkSession: SparkSession,
table: CarbonTable,
logicalPlan: LogicalPlan): Unit = {
-sparkSession.sessionState.catalog.listPartitions(
+val existingPartitions =
sparkSession.sessionState.catalog.listPartitions(
TableIdentifier(table.getTableName, Some(table.getDatabaseName)),
- Some(partition.map(f => (f._1, f._2.get
-val partitionNames = partition.map(k => k._1 + "=" + k._2.get).toSet
+ Some(partition.filter(_._2.isDefined).map(f => (f._1, f._2.get
+val partitionNames = existingPartitions.toList.flatMap { partition =>
+ partition.spec.seq.map{case (column, value) => column + "=" + value}
+}.toSet
val uniqueId = System.currentTimeMillis().toString
val segments = new SegmentStatusManager(
table.getAbsoluteTableIdentifier).getValidAndInvalidSegments.getValidSegments
-try {
- // First drop the partitions from partition mapper files of each
segment
- new CarbonDropPartitionRDD(
-sparkSession.sparkContext,
-table.getTablePath,
-segments.asScala,
-partitionNames.toSeq,
-uniqueId).collect()
-} catch {
- case e: Exception =>
-// roll back the drop partitions from carbon store
-new CarbonDropPartitionCommitRDD(
+// If any existing partitions need to be overwritten then drop from
partitionmap
+if (partitionNames.nonEmpty) {
+ try {
+// First drop the partitions from partition mapper files of each
segment
+new CarbonDropPartitionRDD(
sparkSession.sparkContext,
table.getTablePath,
segments.asScala,
- false,
+ partitionNames.toSeq,
uniqueId).collect()
-throw e
-}
+ } catch {
+case e: Exception =>
+ // roll back the drop partitions from carbon store
+ new CarbonDropPartitionCommitRDD(
+sparkSession.sparkContext,
+table.getTablePath,
+segments.asScala,
+false,
+uniqueId).collect()
+ throw e
+ }
-try {
+ try {
+Dataset.ofRows(sparkSession, logicalPlan)
+ } catch {
+case e: Exception =>
+ // roll back the drop partitions from carbon store
+ new CarbonDropPartitionCommitRDD(
+sparkSession.sparkContext,
+table.getTablePath,
+segments.asScala,
+false,
--- End diff --
provide variable name to improve readability
---
[GitHub] carbondata pull request #1755: [CARBONDATA-1976][PARTITION] Support combinat...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1755#discussion_r159591969
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
---
@@ -702,63 +702,71 @@ case class CarbonLoadDataCommand(
sparkSession: SparkSession,
table: CarbonTable,
logicalPlan: LogicalPlan): Unit = {
-sparkSession.sessionState.catalog.listPartitions(
+val existingPartitions =
sparkSession.sessionState.catalog.listPartitions(
TableIdentifier(table.getTableName, Some(table.getDatabaseName)),
- Some(partition.map(f => (f._1, f._2.get
-val partitionNames = partition.map(k => k._1 + "=" + k._2.get).toSet
+ Some(partition.filter(_._2.isDefined).map(f => (f._1, f._2.get
+val partitionNames = existingPartitions.toList.flatMap { partition =>
+ partition.spec.seq.map{case (column, value) => column + "=" + value}
+}.toSet
val uniqueId = System.currentTimeMillis().toString
val segments = new SegmentStatusManager(
table.getAbsoluteTableIdentifier).getValidAndInvalidSegments.getValidSegments
-try {
- // First drop the partitions from partition mapper files of each
segment
- new CarbonDropPartitionRDD(
-sparkSession.sparkContext,
-table.getTablePath,
-segments.asScala,
-partitionNames.toSeq,
-uniqueId).collect()
-} catch {
- case e: Exception =>
-// roll back the drop partitions from carbon store
-new CarbonDropPartitionCommitRDD(
+// If any existing partitions need to be overwritten then drop from
partitionmap
+if (partitionNames.nonEmpty) {
+ try {
+// First drop the partitions from partition mapper files of each
segment
+new CarbonDropPartitionRDD(
sparkSession.sparkContext,
table.getTablePath,
segments.asScala,
- false,
+ partitionNames.toSeq,
uniqueId).collect()
-throw e
-}
+ } catch {
+case e: Exception =>
+ // roll back the drop partitions from carbon store
+ new CarbonDropPartitionCommitRDD(
+sparkSession.sparkContext,
+table.getTablePath,
+segments.asScala,
+false,
--- End diff --
provide variable name to improve readability
---
[jira] [Commented] (CARBONDATA-1936) Bad Record logger is not working properly in Carbon Partition
[
https://issues.apache.org/jira/browse/CARBONDATA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310903#comment-16310903
]
Ravindra Pesala commented on CARBONDATA-1936:
-
: CREATE TABLE IF NOT EXISTS emp1 (emp_no int,ename string,job string,mgr_id
int,date_of_joining string,salary int,bonus int) partitioned by (dept_no int)
STORED BY 'org.apache.carbondata.format' ;
2: LOAD DATA INPATH 'hdfs://hacluster/user/test/emp.csv' overwrite INTO TABLE
emp1 OPTIONS('DELIMITER'=',', 'QUOTECHAR'= '\');
3: Observe that load is failing
4: LOAD DATA INPATH 'hdfs://hacluster/user/test/emp.csv' overwrite INTO TABLE
emp1 OPTIONS('DELIMITER'=',', 'QUOTECHAR'= '\','BAD_RECORDS_ACTION'='FORCE');
5: Observe that load is success when 'BAD_RECORDS_ACTION'='FORCE' is given in
load command
6: Also observe that same load is suucess in a table without partition.
> Bad Record logger is not working properly in Carbon Partition
> -
>
> Key: CARBONDATA-1936
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1936
> Project: CarbonData
> Issue Type: Bug
>Reporter: Ravindra Pesala
>Priority: Minor
> Attachments: test.csv.csv
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Bad records are not logging and the load is always success irrespective of
> bad records are present.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1936) Bad Record logger is not working properly in Carbon Partition
[
https://issues.apache.org/jira/browse/CARBONDATA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravindra Pesala updated CARBONDATA-1936:
Description:
Bad records are not logging and the load is always success irrespective of bad
records are present.
: CREATE TABLE IF NOT EXISTS emp1 (emp_no int,ename string,job string,mgr_id
int,date_of_joining string,salary int,bonus int) partitioned by (dept_no int)
STORED BY 'org.apache.carbondata.format' ;
2: LOAD DATA INPATH 'hdfs://hacluster/user/test/emp.csv' overwrite INTO TABLE
emp1 OPTIONS('DELIMITER'=',', 'QUOTECHAR'= '\');
3: Observe that load is failing
4: LOAD DATA INPATH 'hdfs://hacluster/user/test/emp.csv' overwrite INTO TABLE
emp1 OPTIONS('DELIMITER'=',', 'QUOTECHAR'= '\','BAD_RECORDS_ACTION'='FORCE');
5: Observe that load is success when 'BAD_RECORDS_ACTION'='FORCE' is given in
load command
6: Also observe that same load is suucess in a table without partition.
was:Bad records are not logging and the load is always success irrespective
of bad records are present.
> Bad Record logger is not working properly in Carbon Partition
> -
>
> Key: CARBONDATA-1936
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1936
> Project: CarbonData
> Issue Type: Bug
>Reporter: Ravindra Pesala
>Priority: Minor
> Attachments: test.csv.csv
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Bad records are not logging and the load is always success irrespective of
> bad records are present.
> : CREATE TABLE IF NOT EXISTS emp1 (emp_no int,ename string,job string,mgr_id
> int,date_of_joining string,salary int,bonus int) partitioned by (dept_no int)
> STORED BY 'org.apache.carbondata.format' ;
> 2: LOAD DATA INPATH 'hdfs://hacluster/user/test/emp.csv' overwrite INTO TABLE
> emp1 OPTIONS('DELIMITER'=',', 'QUOTECHAR'= '\');
> 3: Observe that load is failing
> 4: LOAD DATA INPATH 'hdfs://hacluster/user/test/emp.csv' overwrite INTO TABLE
> emp1 OPTIONS('DELIMITER'=',', 'QUOTECHAR'= '\','BAD_RECORDS_ACTION'='FORCE');
> 5: Observe that load is success when 'BAD_RECORDS_ACTION'='FORCE' is given in
> load command
> 6: Also observe that same load is suucess in a table without partition.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Issue Comment Deleted] (CARBONDATA-1936) Bad Record logger is not working properly in Carbon Partition
[
https://issues.apache.org/jira/browse/CARBONDATA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravindra Pesala updated CARBONDATA-1936:
Comment: was deleted
(was: : CREATE TABLE IF NOT EXISTS emp1 (emp_no int,ename string,job
string,mgr_id int,date_of_joining string,salary int,bonus int) partitioned by
(dept_no int) STORED BY 'org.apache.carbondata.format' ;
2: LOAD DATA INPATH 'hdfs://hacluster/user/test/emp.csv' overwrite INTO TABLE
emp1 OPTIONS('DELIMITER'=',', 'QUOTECHAR'= '\');
3: Observe that load is failing
4: LOAD DATA INPATH 'hdfs://hacluster/user/test/emp.csv' overwrite INTO TABLE
emp1 OPTIONS('DELIMITER'=',', 'QUOTECHAR'= '\','BAD_RECORDS_ACTION'='FORCE');
5: Observe that load is success when 'BAD_RECORDS_ACTION'='FORCE' is given in
load command
6: Also observe that same load is suucess in a table without partition.)
> Bad Record logger is not working properly in Carbon Partition
> -
>
> Key: CARBONDATA-1936
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1936
> Project: CarbonData
> Issue Type: Bug
>Reporter: Ravindra Pesala
>Priority: Minor
> Attachments: test.csv.csv
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Bad records are not logging and the load is always success irrespective of
> bad records are present.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1936) Bad Record logger is not working properly in Carbon Partition
[ https://issues.apache.org/jira/browse/CARBONDATA-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala updated CARBONDATA-1936: Attachment: test.csv.csv > Bad Record logger is not working properly in Carbon Partition > - > > Key: CARBONDATA-1936 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1936 > Project: CarbonData > Issue Type: Bug >Reporter: Ravindra Pesala >Priority: Minor > Attachments: test.csv.csv > > Time Spent: 4h > Remaining Estimate: 0h > > Bad records are not logging and the load is always success irrespective of > bad records are present. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata pull request #1755: [CARBONDATA-1976][PARTITION] Support combinat...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1755#discussion_r159591267
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/standardpartition/StandardPartitionTableLoadingTestCase.scala
---
@@ -28,7 +31,7 @@ import org.apache.carbondata.core.util.CarbonProperties
import org.apache.carbondata.core.util.path.CarbonTablePath
class StandardPartitionTableLoadingTestCase extends QueryTest with
BeforeAndAfterAll {
-
+ var executorService: ExecutorService = _
--- End diff --
move to within the testcase as only that testcase uses it
---
[GitHub] carbondata issue #1678: [CARBONDATA-1903] Fix code issues in carbondata
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1678 Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1305/ ---
[GitHub] carbondata issue #1752: [CARBONDATA-1972][PARTITION] Fix compaction after up...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1752 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2693/ ---
[GitHub] carbondata issue #1711: [CARBONDATA-1754][BugFix] Fixed issue occuring on co...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1711 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2534/ ---
[jira] [Closed] (CARBONDATA-1966) SET command for carbon.properties.filepath is not setting the property
[ https://issues.apache.org/jira/browse/CARBONDATA-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geetika Gupta closed CARBONDATA-1966. - Resolution: Fixed > SET command for carbon.properties.filepath is not setting the property > -- > > Key: CARBONDATA-1966 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1966 > Project: CarbonData > Issue Type: Bug > Components: data-query >Affects Versions: 1.3.0 > Environment: spark2.1 >Reporter: Geetika Gupta > Fix For: 1.3.0 > > > I executed the following command: > set carbon.properties.filepath=/home > It is executed successfully, but it's value is not updated on Spark UI in > spark properties. > It shows the following output: > jdbc:hive2://localhost:1> set carbon.properties.filepath; > +-++--+ > | key | value | > +-++--+ > | carbon.properties.filepath | /home | > +-++--+ > 1 row selected (0.029 seconds) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CARBONDATA-1966) SET command for carbon.properties.filepath is not setting the property
[ https://issues.apache.org/jira/browse/CARBONDATA-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310873#comment-16310873 ] Geetika Gupta commented on CARBONDATA-1966: --- This JIRA is invalid as carbon.properties.filepath is not a dynamically configurable property > SET command for carbon.properties.filepath is not setting the property > -- > > Key: CARBONDATA-1966 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1966 > Project: CarbonData > Issue Type: Bug > Components: data-query >Affects Versions: 1.3.0 > Environment: spark2.1 >Reporter: Geetika Gupta > Fix For: 1.3.0 > > > I executed the following command: > set carbon.properties.filepath=/home > It is executed successfully, but it's value is not updated on Spark UI in > spark properties. > It shows the following output: > jdbc:hive2://localhost:1> set carbon.properties.filepath; > +-++--+ > | key | value | > +-++--+ > | carbon.properties.filepath | /home | > +-++--+ > 1 row selected (0.029 seconds) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata pull request #1747: [Compatibility] Added changes for backward co...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1747#discussion_r159589331
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/path/CarbonTablePath.java ---
@@ -231,7 +231,22 @@ public String getSchemaFilePath() {
* @return schema file path
*/
public static String getSchemaFilePath(String tablePath) {
-return tablePath + File.separator + METADATA_DIR + File.separator +
SCHEMA_FILE;
+return getActualSchemaFilePath(tablePath);
+ }
+
+ private static String getActualSchemaFilePath(String tablePath) {
+String metaPath = tablePath + File.separator + METADATA_DIR;
--- End diff --
use CarbonCommonConstants.FILE_SEPARATOR instead of File.separator
---
[GitHub] carbondata pull request #1747: [Compatibility] Added changes for backward co...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1747#discussion_r159589266
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
---
@@ -291,4 +291,52 @@ object CarbonScalaUtil {
})
otherFields
}
+
+ /**
+ * If the table is from an old store then the table parameters are in
lowercase. In the current
+ * code we are reading the parameters as camel case.
+ * This method will convert all the schema parts to camel case
+ *
+ * @param parameters
+ * @return
+ */
+ def getDeserializedParameters(parameters: Map[String, String]):
Map[String, String] = {
--- End diff --
there is no calling point for this method
---
[GitHub] carbondata pull request #1747: [Compatibility] Added changes for backward co...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1747#discussion_r159589250
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
---
@@ -291,4 +291,52 @@ object CarbonScalaUtil {
})
otherFields
}
+
+ /**
+ * If the table is from an old store then the table parameters are in
lowercase. In the current
+ * code we are reading the parameters as camel case.
+ * This method will convert all the schema parts to camel case
+ *
+ * @param parameters
+ * @return
+ */
+ def getDeserializedParameters(parameters: Map[String, String]):
Map[String, String] = {
+val keyParts =
parameters.getOrElse("spark.sql.sources.options.keys.numparts", "0").toInt
+if (keyParts == 0) {
+ parameters
+} else {
+ var keyStr = ""
+ for(i <- 0 until keyParts) {
+keyStr += parameters(s"spark.sql.sources.options.keys.part.$i")
+ }
+ val finalProperties = scala.collection.mutable.Map.empty[String,
String]
+ keyStr.split(",") foreach {
--- End diff --
Please check the logic here. Keystr is not appended explicitly with , but
splitting is done based on comma
---
[GitHub] carbondata pull request #1732: [CARBONDATA-1946] Exception thrown after alte...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1732#discussion_r159589346 --- Diff: core/src/main/java/org/apache/carbondata/core/scan/result/vector/CarbonColumnVector.java --- @@ -67,6 +67,20 @@ DataType getType(); + /** + * Method to be used for getting the restructured data type. This method will used for + * retrieving the data after change in data type restructure operation + * + * @return + */ + DataType getRestructuredType(); --- End diff -- Better rename as BlockDataType to give more meaning ful ---
[GitHub] carbondata pull request #1759: [CARBONDATA-1980][PARTITION]Fix adding partit...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1759#discussion_r159589011
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/RefreshCarbonTableCommand.scala
---
@@ -91,6 +95,11 @@ case class RefreshCarbonTableCommand(
registerAggregates(databaseName, dataMapSchemaList)(sparkSession)
}
registerTableWithHive(databaseName, tableName,
tableInfo)(sparkSession)
+// Register partitions to hive metastore in case of hive
partitioning carbon table
+if (tableInfo.getFactTable.getPartitionInfo != null &&
+tableInfo.getFactTable.getPartitionInfo.getPartitionType ==
PartitionType.NATIVE_HIVE) {
+ registerAllPartitionsToHive(tablePath, databaseName, tableName,
sparkSession)
--- End diff --
I think passing AbsolutionTableIdentifier is better, it has all
information. It is created in line 65
---
[jira] [Updated] (CARBONDATA-1956) Select query with sum, count and avg throws exception for pre aggregate table
[
https://issues.apache.org/jira/browse/CARBONDATA-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geetika Gupta updated CARBONDATA-1956:
--
Description:
I create a datamap using the following command:
create datamap uniqdata_agg_d on table uniqdata_29 using 'preaggregate' as
select sum(decimal_column1), count(cust_id), avg(bigint_column1) from
uniqdata_29 group by cust_id;
The datamap creation was successfull, but when I tried the following query:
select sum(decimal_column1), count(cust_id), avg(bigint_column1) from
uniqdata_29 group by cust_id;
It throws the following exception:
Error: org.apache.spark.sql.AnalysisException: cannot resolve
'(sum(uniqdata_29_uniqdata_agg_d.`uniqdata_29_bigint_column1_sum`) /
sum(uniqdata_29_uniqdata_agg_d.`uniqdata_29_bigint_column1_count`))' due to
data type mismatch:
'(sum(uniqdata_29_uniqdata_agg_d.`uniqdata_29_bigint_column1_sum`) /
sum(uniqdata_29_uniqdata_agg_d.`uniqdata_29_bigint_column1_count`))' requires
(double or decimal) type, not bigint;;
'Aggregate [uniqdata_29_cust_id_count#244],
[sum(uniqdata_29_decimal_column1_sum#243) AS sum(decimal_column1)#274,
sum(cast(uniqdata_29_cust_id_count#244 as bigint)) AS count(cust_id)#276L,
(sum(uniqdata_29_bigint_column1_sum#245L) /
sum(uniqdata_29_bigint_column1_count#246L)) AS avg(bigint_column1)#279]
+-
Relation[uniqdata_29_decimal_column1_sum#243,uniqdata_29_cust_id_count#244,uniqdata_29_bigint_column1_sum#245L,uniqdata_29_bigint_column1_count#246L]
CarbonDatasourceHadoopRelation [ Database name :28dec, Table name
:uniqdata_29_uniqdata_agg_d, Schema
:Some(StructType(StructField(uniqdata_29_decimal_column1_sum,DecimalType(30,10),true),
StructField(uniqdata_29_cust_id_count,IntegerType,true),
StructField(uniqdata_29_bigint_column1_sum,LongType,true),
StructField(uniqdata_29_bigint_column1_count,LongType,true))) ] (state=,code=0)
Steps for creation of maintable:
CREATE TABLE uniqdata_29(CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION
string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2
bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2
decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1
int) STORED BY 'org.apache.carbondata.format';
Load command:
LOAD DATA INPATH 'hdfs://localhost:54311/Files/2000_UniqData.csv' into table
uniqdata_29 OPTIONS('DELIMITER'=',',
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
Datamap creation command:
create datamap uniqdata_agg_d on table uniqdata_29 using 'preaggregate' as
select sum(decimal_column1), count(cust_id), avg(bigint_column1) from
uniqdata_29 group by cust_id;
Note: select sum(decimal_column1), count(cust_id), avg(bigint_column1) from
uniqdata_29 group by cust_id; executed successfully on maintable
was:
I create a datamap using the following command:
create datamap uniqdata_agg_d on table uniqdata_29 using 'preaggregate' as
select sum(decimal_column1), count(cust_id), avg(bigint_column1) from
uniqdata_29 group by cust_id;
The datamap creation was successfull, but when I tried the following query:
select sum(decimal_column1), count(cust_id), avg(bigint_column1) from
uniqdata_29 group by cust_id;
It throws the following exception:
Error: org.apache.spark.sql.AnalysisException: cannot resolve
'(sum(uniqdata_29_uniqdata_agg_d.`uniqdata_29_bigint_column1_sum`) /
sum(uniqdata_29_uniqdata_agg_d.`uniqdata_29_bigint_column1_count`))' due to
data type mismatch:
'(sum(uniqdata_29_uniqdata_agg_d.`uniqdata_29_bigint_column1_sum`) /
sum(uniqdata_29_uniqdata_agg_d.`uniqdata_29_bigint_column1_count`))' requires
(double or decimal) type, not bigint;;
'Aggregate [uniqdata_29_cust_id_count#244],
[sum(uniqdata_29_decimal_column1_sum#243) AS sum(decimal_column1)#274,
sum(cast(uniqdata_29_cust_id_count#244 as bigint)) AS count(cust_id)#276L,
(sum(uniqdata_29_bigint_column1_sum#245L) /
sum(uniqdata_29_bigint_column1_count#246L)) AS avg(bigint_column1)#279]
+-
Relation[uniqdata_29_decimal_column1_sum#243,uniqdata_29_cust_id_count#244,uniqdata_29_bigint_column1_sum#245L,uniqdata_29_bigint_column1_count#246L]
CarbonDatasourceHadoopRelation [ Database name :28dec, Table name
:uniqdata_29_uniqdata_agg_d, Schema
:Some(StructType(StructField(uniqdata_29_decimal_column1_sum,DecimalType(30,10),true),
StructField(uniqdata_29_cust_id_count,IntegerType,true),
StructField(uniqdata_29_bigint_column1_sum,LongType,true),
StructField(uniqdata_29_bigint_column1_count,LongType,true))) ] (state=,code=0)
Steps for creation of maintable:
CREATE TABLE uniqdata_29(CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION
string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2
bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2
decimal(36,10),Double_COLUMN1 do
[GitHub] carbondata issue #1748: [CARBONDATA-1967][PARTITION] Fix autocompaction and ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1748 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2692/ ---
[jira] [Created] (CARBONDATA-1981) Error occurs while building project in windows environment
xuchuanyin created CARBONDATA-1981: -- Summary: Error occurs while building project in windows environment Key: CARBONDATA-1981 URL: https://issues.apache.org/jira/browse/CARBONDATA-1981 Project: CarbonData Issue Type: Bug Components: build, core Affects Versions: 1.3.0 Reporter: xuchuanyin Encounter error while building project in Windows environment, the error messages are shown as bellow: ``` [INFO] Apache CarbonData :: Parent SUCCESS [ 3.915 s] [INFO] Apache CarbonData :: Common SUCCESS [ 12.007 s] [INFO] Apache CarbonData :: Format SUCCESS [ 30.059 s] [INFO] Apache CarbonData :: Core .. FAILURE [ 3.604 s] [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 50.099 s [INFO] Finished at: 2018-01-04T11:19:28+08:00 [INFO] Final Memory: 62M/612M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (default) on project carbondata-core: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "bash" (in directory "D:\01_workspace\carbondata\core"): CreateProcess error=2, 系统找不到指定的文件。(cannot find the specified file) [ERROR] around Ant part .. @ 4:27 in D:\01_workspace\carbondata\core\target\antrun\build-main.xml [ERROR] -> [Help 1] ``` It is because in the pom, it calls bash script, which is not incompatible in windows. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1729: [CARBONDATA-1936][PARTITION] Corrected bad record an...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1729 Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1304/ ---
[GitHub] carbondata issue #1760: [CARBONDATA-1979] ][IMPLICIT COLUMN] Modified implic...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1760 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2533/ ---
[GitHub] carbondata pull request #1082: [CARBONDATA-1218] [GLOBAL SORT] In case of da...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1082#discussion_r159586543
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/util/CarbonDataProcessorUtil.java
---
@@ -145,6 +147,15 @@ public static void
renameBadRecordsFromInProgressToNormal(
}
}
+ public static void renameBadRecord(CarbonDataLoadConfiguration
configuration) {
+// rename the bad record in progress to normal
+CarbonTableIdentifier identifier =
+configuration.getTableIdentifier().getCarbonTableIdentifier();
+
CarbonDataProcessorUtil.renameBadRecordsFromInProgressToNormal(configuration,
--- End diff --
1. Avoid calling the method using classname. Method is present in the same
class.
2. I think we can introduce a new class called BadRecordsUtil which will
have all the functions related to bad records like
renameBadRecord, hasBadRecord, renameBadRecordsFromInProgressToNormal (make
it private in the new util class)
---
[GitHub] carbondata pull request #1082: [CARBONDATA-1218] [GLOBAL SORT] In case of da...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1082#discussion_r159584326
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/badrecordloger/BadRecordLoggerTest.scala
---
@@ -247,6 +248,33 @@ class BadRecordLoggerTest extends QueryTest with
BeforeAndAfterAll {
}
}
+ test(
+"test if first load failed due to bad record then second load should
not failed if there is " +
+"no bad record") {
+sql("drop table IF EXISTS loadIssue")
+sql("""CREATE TABLE IF NOT EXISTS loadIssue(ID BigInt, item String)
STORED BY 'carbondata'""")
+
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_BAD_RECORDS_ACTION,
+ "FAIL").addProperty(CarbonCommonConstants.LOAD_SORT_SCOPE,
"GLOBAL_SORT")
+try {
+ sql("insert into loadIssue select 'x','Book'");
+} catch {
+ case ex: Exception =>
+assert(true)
+}
+try {
+ sql("insert into loadIssue select 1,'Book'");
+} catch {
+ case ex: Exception =>
+assert(false)
+} finally {
+ CarbonProperties.getInstance()
+.addProperty(CarbonCommonConstants.CARBON_BAD_RECORDS_ACTION,
"FORCE")
+.addProperty(CarbonCommonConstants.LOAD_SORT_SCOPE, "LOCAL_SORT")
+}
+assert(true)
+sql("drop table IF EXISTS loadIssue")
+ }
+
--- End diff --
1. Use only one try and finally block inside test code..try should start
from first line and end at last and finally should only reset the carbon
property
2. Instead of try catch use intercept[Exception] only for the 1st case
where exception is expected. For 2nd case try catch is not required...if
exception comes anyways test case will fail
---
[GitHub] carbondata pull request #1732: [CARBONDATA-1946] Exception thrown after alte...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1732#discussion_r159585172
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java ---
@@ -775,4 +775,25 @@ public static DataType valueOf(String name) {
}
}
-}
\ No newline at end of file
+ /**
+ * Method to type case the data based on modified data type. This method
will used for
+ * retrieving the data after change in data type restructure operation
+ *
+ * @param data
+ * @param restructuredDataType
+ * @param currentDataOffset
+ * @param length
+ * @return
+ */
+ public static long getDataBasedOnRestructuredDataType(byte[] data,
DataType restructuredDataType,
--- End diff --
yes because alter datatype change is only for Int to BigInt and decimal
from lower precision to higher precision. So float to double case is not
required to be handled
---
[GitHub] carbondata pull request #1732: [CARBONDATA-1946] Exception thrown after alte...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1732#discussion_r159585515
--- Diff:
integration/spark2/src/test/scala/org/apache/spark/carbondata/restructure/vectorreader/ChangeDataTypeTestCases.scala
---
@@ -164,6 +164,21 @@ class ChangeDataTypeTestCases extends Spark2QueryTest
with BeforeAndAfterAll {
sql("drop table if exists PreAggMain_preagg1")
}
+ test("test data type change for dictionary exclude INT type column") {
--- End diff --
Alter table datatype change does not support short to int, short to long
and float to double. It only supports Int to BigInt and decimal from lower
precision to higher precision. If any other data type change request comes
validation is done and exception is thrown
---
[jira] [Resolved] (CARBONDATA-1975) Wrong input metrics displayed for carbon
[ https://issues.apache.org/jira/browse/CARBONDATA-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manish Gupta resolved CARBONDATA-1975. -- Resolution: Fixed Fix Version/s: 1.3.0 > Wrong input metrics displayed for carbon > > > Key: CARBONDATA-1975 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1975 > Project: CarbonData > Issue Type: Bug >Reporter: dhatchayani >Assignee: dhatchayani >Priority: Minor > Fix For: 1.3.0 > > Attachments: beeline.JPG > > Time Spent: 1h > Remaining Estimate: 0h > > Input metrics is updated twice. Record count is updated twice and it is > wrongly displayed in Spark UI -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1729: [CARBONDATA-1936][PARTITION] Corrected bad record an...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1729 retest this please ---
[GitHub] carbondata pull request #1754: [CARBONDATA-1975] Wrong input metrics display...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/1754 ---
[GitHub] carbondata issue #1754: [CARBONDATA-1975] Wrong input metrics displayed for ...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/1754 LGTM...please attach the test results snapshots when the jira and github issue is resolved ---
[GitHub] carbondata issue #1733: [CARBONDATA-1944][PARTITION]Special character like c...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1733 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2691/ ---
[jira] [Updated] (CARBONDATA-1975) Wrong input metrics displayed for carbon
[ https://issues.apache.org/jira/browse/CARBONDATA-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhatchayani updated CARBONDATA-1975: Attachment: beeline.JPG Description: Input metrics is updated twice. Record count is updated twice and it is wrongly displayed in Spark UI was:Input metrics is updated twice. Record count is updated twice and it is wrongly displayed in Spark UI > Wrong input metrics displayed for carbon > > > Key: CARBONDATA-1975 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1975 > Project: CarbonData > Issue Type: Bug >Reporter: dhatchayani >Assignee: dhatchayani >Priority: Minor > Attachments: beeline.JPG > > Time Spent: 40m > Remaining Estimate: 0h > > Input metrics is updated twice. Record count is updated twice and it is > wrongly displayed in Spark UI -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1718: [CARBONDATA-1929][Validation]carbon property configu...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1718 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2532/ ---
[GitHub] carbondata issue #1733: [CARBONDATA-1944][PARTITION]Special character like c...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1733 Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1303/ ---
[GitHub] carbondata issue #1729: [CARBONDATA-1936][PARTITION] Corrected bad record an...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1729 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2690/ ---
[GitHub] carbondata pull request #1760: [CARBONDATA-1979] ][IMPLICIT COLUMN] Modified...
GitHub user rahulforallp opened a pull request: https://github.com/apache/carbondata/pull/1760 [CARBONDATA-1979] ][IMPLICIT COLUMN] Modified implicit column filtering logic to directly validate the blo⦠â¦cklet ID as after driver pruning valid blocklets are identified and tasks are divided based on blocklets Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rahulforallp/incubator-carbondata implicit_column Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1760.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1760 commit 4df91bdf5608c280cd4c3f3320ddcb957aa496f2 Author: rahulforallp Date: 2018-01-03T09:51:37Z Modified implicit column filtering logic to directly validate the blocklet ID as after driver pruning valid blocklets are identified and tasks are divided based on blocklets ---
[GitHub] carbondata issue #1718: [CARBONDATA-1929][Validation]carbon property configu...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1718 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2689/ ---
[GitHub] carbondata issue #1753: [CARBONDATA-1970](Carbon1.3.0 - Spark 2.2) Use Spark...
Github user zzcclp commented on the issue: https://github.com/apache/carbondata/pull/1753 The discussion about this on mailing list : [Should-we-use-Spark-2-2-1-as-default-version-for-Spark-2-2-supported](http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Should-we-use-Spark-2-2-1-as-default-version-for-Spark-2-2-supported-td32713.html) ---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Github user sraghunandan commented on the issue: https://github.com/apache/carbondata/pull/1650 retest sdv please ---
[GitHub] carbondata issue #1753: [CARBONDATA-1970](Carbon1.3.0 - Spark 2.2) Use Spark...
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/1753 please add mailing list discussion link to description. ---
[GitHub] carbondata issue #1718: [CARBONDATA-1929][Validation]carbon property configu...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1718 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2688/ ---
[GitHub] carbondata issue #1678: [CARBONDATA-1903] Fix code issues in carbondata
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1678 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2531/ ---
[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1759 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2530/ ---
[GitHub] carbondata issue #1748: [CARBONDATA-1967][PARTITION] Fix autocompaction and ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1748 Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1302/ ---
[GitHub] carbondata issue #1758: [CARBONDATA-1978] Handled preaggregate issues with h...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1758 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2687/ ---
[GitHub] carbondata issue #1756: [HOTFIX] Remove unnecessary object creation in write...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1756 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2686/ ---
[jira] [Resolved] (CARBONDATA-1903) Fix some code issues in carbondata
[ https://issues.apache.org/jira/browse/CARBONDATA-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] QiangCai resolved CARBONDATA-1903. -- Resolution: Fixed Fix Version/s: 1.3.0 > Fix some code issues in carbondata > -- > > Key: CARBONDATA-1903 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1903 > Project: CarbonData > Issue Type: Improvement >Reporter: xuchuanyin >Assignee: xuchuanyin >Priority: Minor > Fix For: 1.3.0 > > Time Spent: 12h 10m > Remaining Estimate: 0h > > Remove useless code in carbondata -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata pull request #1678: [CARBONDATA-1903] Fix code issues in carbonda...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/1678 ---
[GitHub] carbondata issue #1718: [CARBONDATA-1929][Validation]carbon property configu...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1718 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2529/ ---
[GitHub] carbondata issue #1752: [CARBONDATA-1972][PARTITION] Fix compaction after up...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1752 Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1301/ ---
[GitHub] carbondata issue #1678: [CARBONDATA-1903] Fix code issues in carbondata
Github user QiangCai commented on the issue: https://github.com/apache/carbondata/pull/1678 LGTM ---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1757 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2685/ ---
[GitHub] carbondata issue #1678: [CARBONDATA-1903] Fix code issues in carbondata
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/1678 retest this please ---
[GitHub] carbondata issue #1729: [CARBONDATA-1936][PARTITION] Corrected bad record an...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1729 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2528/ ---
[GitHub] carbondata issue #1754: [CARBONDATA-1975] Wrong input metrics displayed for ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1754 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2684/ ---
[GitHub] carbondata issue #1759: [CARBONDATA-1980][PARTITION]Fix adding partition inf...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1759 Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1300/ ---
[GitHub] carbondata issue #1733: [CARBONDATA-1944][PARTITION]Special character like c...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1733 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2527/ ---
[GitHub] carbondata issue #1755: [CARBONDATA-1976][PARTITION] Support combination of ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1755 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2683/ ---
[GitHub] carbondata issue #1678: [CARBONDATA-1903] Fix code issues in carbondata
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1678 Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1299/ ---
[GitHub] carbondata issue #1748: [CARBONDATA-1967][PARTITION] Fix autocompaction and ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1748 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2526/ ---
[GitHub] carbondata issue #1713: [WIP] [CARBONDATA-1899] Optimize CarbonData concurre...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1713 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2682/ ---
[GitHub] carbondata issue #1713: [WIP] [CARBONDATA-1899] Optimize CarbonData concurre...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1713 Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1298/ ---
[GitHub] carbondata issue #1752: [CARBONDATA-1972][PARTITION] Fix compaction after up...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1752 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2525/ ---
[GitHub] carbondata issue #1743: [CARBONDATA-1954] [Pre-Aggregate] CarbonHiveMetastor...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1743 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2681/ ---
[GitHub] carbondata issue #1718: [CARBONDATA-1929][Validation]carbon property configu...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1718 Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1297/ ---
[GitHub] carbondata issue #1754: [CARBONDATA-1975] Wrong input metrics displayed for ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1754 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2524/ ---
[GitHub] carbondata issue #1747: [Compatibility] Added changes for backward compatibi...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1747 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2680/ ---
[GitHub] carbondata issue #1743: [CARBONDATA-1954] [Pre-Aggregate] CarbonHiveMetastor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1743 Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1296/ ---
[GitHub] carbondata issue #1755: [CARBONDATA-1976][PARTITION] Support combination of ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1755 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2523/ ---
[GitHub] carbondata issue #1697: [CARBONDATA-1719][Pre-Aggregate][Bug] Fixed bug to h...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1697 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2679/ ---
[GitHub] carbondata issue #1747: [Compatibility] Added changes for backward compatibi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1747 Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1295/ ---
[GitHub] carbondata issue #1756: [HOTFIX] Remove unnecessary object creation in write...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1756 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2522/ ---
[GitHub] carbondata issue #1702: [WIP][CARBONDATA-1896] Clean files operation improve...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1702 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2678/ ---
[GitHub] carbondata issue #1754: [CARBONDATA-1975] Wrong input metrics displayed for ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1754 Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1294/ ---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1757 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2521/ ---
[GitHub] carbondata issue #1758: [CARBONDATA-1978] Handled preaggregate issues with h...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1758 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2520/ ---
[GitHub] carbondata issue #1718: [CARBONDATA-1929][Validation]carbon property configu...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1718 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2677/ ---
[GitHub] carbondata issue #1755: [CARBONDATA-1976][PARTITION] Support combination of ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1755 Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1293/ ---
[GitHub] carbondata issue #1116: [CARBONDATA-1249] Wrong order of columns in redirect...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1116 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2519/ ---
[GitHub] carbondata pull request #1759: [CARBONDATA-1980][PARTITION]Fix adding partit...
GitHub user ravipesala opened a pull request: https://github.com/apache/carbondata/pull/1759 [CARBONDATA-1980][PARTITION]Fix adding partition information while doing refresh(restore) table.And fix the case sensitivity issue of partition column Problem While restoring the table the partitions are not added to the hive. And also if any uppercase letters are present in partition column then filter queries on that column gives error. Solution Read all partitionmap files of all segments to add back the partition information to the hive metatstore. And always update the partition filter columns to lower case before querying partition information. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [X] Any interfaces changed? NO - [X] Any backward compatibility impacted? NO - [X] Document update required? NO - [X] Testing done Tests added - [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ravipesala/incubator-carbondata restore-partition Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1759.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1759 commit 20e411487a11c2219e08e966ff83c6cc0e718678 Author: ravipesala Date: 2018-01-03T19:04:33Z Fix adding partition information while doing refresh(restore) table. And fix the case sensitivity issue of partition column ---
[GitHub] carbondata issue #1756: [HOTFIX] Remove unnecessary object creation in write...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1756 Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1292/ ---
[jira] [Created] (CARBONDATA-1980) Partition information is added while restore or refresh the table. And also query is not working if there is nay upper case letter in partition column.
Ravindra Pesala created CARBONDATA-1980: --- Summary: Partition information is added while restore or refresh the table. And also query is not working if there is nay upper case letter in partition column. Key: CARBONDATA-1980 URL: https://issues.apache.org/jira/browse/CARBONDATA-1980 Project: CarbonData Issue Type: Bug Reporter: Ravindra Pesala Partition information is added while restore or refresh the table. And also query is not working if there is nay upper case letter in partition column. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1678: [CARBONDATA-1903] Fix code issues in carbondata
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1678 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2518/ ---
[GitHub] carbondata issue #1697: [CARBONDATA-1719][Pre-Aggregate][Bug] Fixed bug to h...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1697 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2676/ ---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1757 Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1291/ ---
[GitHub] carbondata issue #1758: [CARBONDATA-1978] Handled preaggregate issues with h...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1758 Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1290/ ---
[GitHub] carbondata issue #1697: [CARBONDATA-1719][Pre-Aggregate][Bug] Fixed bug to h...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1697 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2517/ ---
[GitHub] carbondata issue #1697: [CARBONDATA-1719][Pre-Aggregate][Bug] Fixed bug to h...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1697 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2675/ ---
[GitHub] carbondata issue #1728: [CARBONDATA-1926][CARBONDATA-1927][Pre-Aggregate] Ex...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1728 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2674/ ---
