[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
CarbonDataQA1 commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-699588340 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4236/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
CarbonDataQA1 commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-699588047 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2492/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-699587843 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2493/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-699587732 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4237/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3960: [CARBONDATA-4014] Support Change Column Comment
CarbonDataQA1 commented on pull request #3960: URL: https://github.com/apache/carbondata/pull/3960#issuecomment-699585739 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2488/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
CarbonDataQA1 commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-699585741 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2489/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3960: [CARBONDATA-4014] Support Change Column Comment
CarbonDataQA1 commented on pull request #3960: URL: https://github.com/apache/carbondata/pull/3960#issuecomment-699585500 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4232/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
CarbonDataQA1 commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-699585268 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4233/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r495524053
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/DeleteCarbonTableTestCase.scala
##
@@ -26,15 +26,14 @@ import org.scalatest.BeforeAndAfterAll
import
org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException
import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.index.Segment
import org.apache.carbondata.core.datastore.filesystem.{CarbonFile,
CarbonFileFilter}
import org.apache.carbondata.core.datastore.impl.FileFactory
import org.apache.carbondata.core.mutate.CarbonUpdateUtil
-import org.apache.carbondata.core.util.CarbonProperties
-import org.apache.carbondata.core.util.CarbonUtil
+import org.apache.carbondata.core.util.{CarbonProperties, CarbonUtil}
import org.apache.carbondata.core.util.path.CarbonTablePath
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Klaus-xjp commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Klaus-xjp commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r495521329
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/DeleteCarbonTableTestCase.scala
##
@@ -26,15 +26,14 @@ import org.scalatest.BeforeAndAfterAll
import
org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException
import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.index.Segment
import org.apache.carbondata.core.datastore.filesystem.{CarbonFile,
CarbonFileFilter}
import org.apache.carbondata.core.datastore.impl.FileFactory
import org.apache.carbondata.core.mutate.CarbonUpdateUtil
-import org.apache.carbondata.core.util.CarbonProperties
-import org.apache.carbondata.core.util.CarbonUtil
+import org.apache.carbondata.core.util.{CarbonProperties, CarbonUtil}
import org.apache.carbondata.core.util.path.CarbonTablePath
Review comment:
too many blank line in there
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3960: [CARBONDATA-4014] Support Change Column Comment
CarbonDataQA1 commented on pull request #3960: URL: https://github.com/apache/carbondata/pull/3960#issuecomment-699580135 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2486/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3960: [CARBONDATA-4014] Support Change Column Comment
CarbonDataQA1 commented on pull request #3960: URL: https://github.com/apache/carbondata/pull/3960#issuecomment-699579525 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4230/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r495521844
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDropTableCommand.scala
##
@@ -82,8 +82,9 @@ case class CarbonDropTableCommand(
}
}
- if (SegmentStatusManager.isLoadInProgressInTable(carbonTable)) {
-throw new ConcurrentOperationException(carbonTable, "loading", "drop
table")
+ if (SegmentStatusManager.isInsertInProgress(carbonTable)) {
+throw new ConcurrentOperationException(carbonTable,
+ "insert or insert overwrite", "drop table")
Review comment:
done
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableRenameCommand.scala
##
@@ -96,8 +96,9 @@ private[sql] case class CarbonAlterTableRenameCommand(
.asInstanceOf[CarbonRelation].carbonTable
carbonTableLockFilePath = carbonTable.getTablePath
// if any load is in progress for table, do not allow rename table
- if (SegmentStatusManager.isLoadInProgressInTable(carbonTable)) {
-throw new ConcurrentOperationException(carbonTable, "loading", "alter
table rename")
+ if (SegmentStatusManager.isInsertInProgress(carbonTable)) {
+throw new ConcurrentOperationException(carbonTable,
+"insert or insert overwrite", "alter table rename")
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-699579025 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4234/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-699578946 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2490/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
marchpure commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r495521478
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDropTableCommand.scala
##
@@ -82,8 +82,9 @@ case class CarbonDropTableCommand(
}
}
- if (SegmentStatusManager.isLoadInProgressInTable(carbonTable)) {
-throw new ConcurrentOperationException(carbonTable, "loading", "drop
table")
+ if (SegmentStatusManager.isInsertInProgress(carbonTable)) {
+throw new ConcurrentOperationException(carbonTable,
+ "insert or insert overwrite", "drop table")
Review comment:
take care about the format
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableRenameCommand.scala
##
@@ -96,8 +96,9 @@ private[sql] case class CarbonAlterTableRenameCommand(
.asInstanceOf[CarbonRelation].carbonTable
carbonTableLockFilePath = carbonTable.getTablePath
// if any load is in progress for table, do not allow rename table
- if (SegmentStatusManager.isLoadInProgressInTable(carbonTable)) {
-throw new ConcurrentOperationException(carbonTable, "loading", "alter
table rename")
+ if (SegmentStatusManager.isInsertInProgress(carbonTable)) {
+throw new ConcurrentOperationException(carbonTable,
+"insert or insert overwrite", "alter table rename")
Review comment:
take care about the format
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-699578575 > Consider a scenario that the user want to insert data and then update the table, the dml commands are sent one by one in time series, but the 2nd(update) cmd is sent by another driver and start to execute while the 1st(insert) is still running, therefore the concurrency scenario described in this PR appears. > If we allow update table who has segment with status 'INSERT_INPROGRESS', the 2nd cmd(update) will execute successfully and the data inserted by the 1st cmd would not updated by the 2nd cmd. > In this case, the result may not be what the user expected. you are right This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r495520426
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/view/timeseries/TestCreateMVWithTimeSeries.scala
##
@@ -167,18 +182,24 @@ class TestCreateMVWithTimeSeries extends QueryTest with
BeforeAndAfterAll {
test("insert and create materialized view in progress") {
sql("drop materialized view if exists mv1")
-val query = s"LOAD DATA local inpath '$resourcesPath/data_big.csv' INTO
TABLE maintable " +
-s"OPTIONS('DELIMITER'= ',')"
-val executorService = Executors.newFixedThreadPool(4)
-executorService.submit(new QueryTask(query))
-intercept[UnsupportedOperationException] {
- sql(
-"create materialized view mv1 as " +
+
+val future = AsyncExecutorUtils.runSqlAsync("insert overwrite table
maintable select * from temp_maintable")
+val ex = intercept[UnsupportedOperationException] {
+ sql("create materialized view mv1 as " +
"select timeseries(projectjoindate,'year'), sum(projectcode) from
maintable group by timeseries(projectjoindate,'year')")
-}.getMessage
- .contains("Cannot create mv materialized view table when insert is in
progress on parent table: maintable")
-executorService.shutdown()
-executorService.awaitTermination(2, TimeUnit.HOURS)
+}
+assert(future.get.contains("PASS"))
+assert(ex.getMessage.contains("Cannot create mv when insert overwrite is
in progress on table default_maintable"))
+sql("drop materialized view if exists mv1")
+ }
+
+ test("create materialized view should success when parent table is insert in
progress") {
+sql("drop materialized view if exists mv1")
+
+val future = AsyncExecutorUtils.runSqlAsync("insert into table maintable
select * from temp_maintable")
+sql("create materialized view mv1 as " +
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r495520319
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/TestInsertAndOtherCommandConcurrent.scala
##
@@ -18,45 +18,31 @@
package org.apache.carbondata.spark.testsuite.iud
import java.text.SimpleDateFormat
-import java.util
-import java.util.concurrent.{Callable, ExecutorService, Executors, Future}
-import scala.collection.JavaConverters._
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.exception.ConcurrentOperationException
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.spark.util.{AsyncExecutorUtils, Global,
WaitingIndexFactory}
-import org.apache.hadoop.fs.Path
import org.apache.spark.sql.test.util.QueryTest
import org.apache.spark.sql.types.StructType
-import org.apache.spark.sql.{DataFrame, SaveMode}
+import org.apache.spark.sql.{DataFrame, Row, SaveMode}
import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.datastore.block.SegmentProperties
-import org.apache.carbondata.core.datastore.page.ColumnPage
-import org.apache.carbondata.core.exception.ConcurrentOperationException
-import org.apache.carbondata.core.features.TableOperation
-import org.apache.carbondata.core.index.dev.cgindex.{CoarseGrainIndex,
CoarseGrainIndexFactory}
-import org.apache.carbondata.core.index.dev.{IndexBuilder, IndexWriter}
-import org.apache.carbondata.core.index.{IndexInputSplit, IndexMeta, Segment}
-import org.apache.carbondata.core.indexstore.PartitionSpec
-import org.apache.carbondata.core.metadata.schema.table.{CarbonTable,
IndexSchema}
-import org.apache.carbondata.core.scan.filter.intf.ExpressionType
-import org.apache.carbondata.core.util.CarbonProperties
-import org.apache.carbondata.events.Event
-
// This testsuite test insert and insert overwrite with other commands
concurrently
class TestInsertAndOtherCommandConcurrent extends QueryTest with
BeforeAndAfterAll with BeforeAndAfterEach {
- private val executorService: ExecutorService =
Executors.newFixedThreadPool(10)
+
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-699575847 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4231/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
CarbonDataQA1 commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-699575770 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2487/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r495512523
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/view/timeseries/TestCreateMVWithTimeSeries.scala
##
@@ -165,20 +180,32 @@ class TestCreateMVWithTimeSeries extends QueryTest with
BeforeAndAfterAll {
sql("drop materialized view if exists mv2")
}
- test("insert and create materialized view in progress") {
+ test("reate materialized view should fail when parent table is insert
overwrite in progress") {
Review comment:
done
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/DeleteCarbonTableTestCase.scala
##
@@ -343,6 +342,31 @@ class DeleteCarbonTableTestCase extends QueryTest with
BeforeAndAfterAll {
sql("drop table if exists test_dm_index")
}
+ // block delete operation from table which has index. see PR2483
+ // DummyIndexFactory is used to pass the check "if(carbonTable.canAllow())"
+ // in class CarbonProjectForUpdateCommand
+ test("delete should fail if the table has index") {
+
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3960: [CARBONDATA-4014] Support Change Column Comment
CarbonDataQA1 commented on pull request #3960: URL: https://github.com/apache/carbondata/pull/3960#issuecomment-699539414 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2485/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3960: [CARBONDATA-4014] Support Change Column Comment
CarbonDataQA1 commented on pull request #3960: URL: https://github.com/apache/carbondata/pull/3960#issuecomment-699538898 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4229/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3960: [CARBONDATA-4014] Support Change Column Comment
CarbonDataQA1 commented on pull request #3960: URL: https://github.com/apache/carbondata/pull/3960#issuecomment-699526651 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2484/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3960: [CARBONDATA-4014] Support Change Column Comment
CarbonDataQA1 commented on pull request #3960: URL: https://github.com/apache/carbondata/pull/3960#issuecomment-699526051 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4228/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure opened a new pull request #3960: [CARBONDATA-4014] Support Change Column Comment
marchpure opened a new pull request #3960: URL: https://github.com/apache/carbondata/pull/3960 ### Why is this PR needed? Now, we support add column comment in "CREATE TABLE" and "ADD COLUMN". but do not support alter comment of the column, which shall be support in 'CHANGE COLUMN' ### What changes were proposed in this PR? Support "ALTER TABLE table_name CHANGE [COLUMN] col_name col_name data_type [COMMENT col_comment]" ### Does this PR introduce any user interface change? - Yes. add CHANGE COLUMN explaination in the ddl document. ### Is any new testcase added? - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (CARBONDATA-4014) Support Change Column Comment
Xingjun Hao created CARBONDATA-4014: --- Summary: Support Change Column Comment Key: CARBONDATA-4014 URL: https://issues.apache.org/jira/browse/CARBONDATA-4014 Project: CarbonData Issue Type: New Feature Components: sql Affects Versions: 2.0.1 Reporter: Xingjun Hao Fix For: 2.1.0 Now, we support add comment when CREATE TABLE and ADD COLUMN. but do not support alter comment of specified column. We shall support alter comment with hive syntax "ALTER TABLE table_name CHANGE [COLUMN] col_name col_name data_type [COMMENT col_comment]" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] Klaus-xjp commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Klaus-xjp commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r495454501
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/DeleteCarbonTableTestCase.scala
##
@@ -343,6 +342,31 @@ class DeleteCarbonTableTestCase extends QueryTest with
BeforeAndAfterAll {
sql("drop table if exists test_dm_index")
}
+ // block delete operation from table which has index. see PR2483
+ // DummyIndexFactory is used to pass the check "if(carbonTable.canAllow())"
+ // in class CarbonProjectForUpdateCommand
+ test("delete should fail if the table has index") {
+
Review comment:
delete redundant blank line
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
CarbonDataQA1 commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-699485934 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4227/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
CarbonDataQA1 commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-699485704 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2483/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
marchpure commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r495441113
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/events/DeleteFromTableEventListener.scala
##
@@ -53,6 +53,12 @@ class DeleteFromTableEventListener extends
OperationEventListener with Logging {
carbonTable
.getDatabaseName
}.${ carbonTable.getTableName }]")
+} else if (!carbonTable.getIndexesMap.isEmpty) {
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
marchpure commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r495440288
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/view/timeseries/TestCreateMVWithTimeSeries.scala
##
@@ -165,20 +180,32 @@ class TestCreateMVWithTimeSeries extends QueryTest with
BeforeAndAfterAll {
sql("drop materialized view if exists mv2")
}
- test("insert and create materialized view in progress") {
+ test("reate materialized view should fail when parent table is insert
overwrite in progress") {
Review comment:
reate -> create
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r495432417
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/DeleteCarbonTableTestCase.scala
##
@@ -322,7 +321,32 @@ class DeleteCarbonTableTestCase extends QueryTest with
BeforeAndAfterAll {
}
- test("block deleting records from table which has index") {
+ // block delete operation from table which has index. see PR2483
+ test("delete should fail if the table has index") {
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r495432401
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala
##
@@ -794,7 +795,31 @@ class UpdateCarbonTableTestCase extends QueryTest with
BeforeAndAfterAll {
sql("drop table if exists senten")
}
- test("block updating table which has index") {
+ // block update operation from table which has index. see PR2483
+ test("update should fail if the table has index") {
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r495432034
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/view/timeseries/TestCreateMVWithTimeSeries.scala
##
@@ -33,17 +34,30 @@ class TestCreateMVWithTimeSeries extends QueryTest with
BeforeAndAfterAll {
override def beforeAll(): Unit = {
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
"dd-MM-")
-drop()
+dropTable()
sql("CREATE TABLE maintable (empname String, designation String, doj
Timestamp, workgroupcategory int, workgroupcategoryname String, deptno int, " +
-"deptname String, projectcode int, projectjoindate Timestamp,
projectenddate Timestamp,attendance int, utilization int,salary int) STORED AS
carbondata")
+ "deptname String, projectcode int, projectjoindate Timestamp,
projectenddate Timestamp,attendance int, utilization int,salary int) STORED AS
carbondata")
+sql(s"""
+ | CREATE INDEX maintable_index_test
+ | ON TABLE maintable (designation)
+ | AS '${classOf[WaitingIndexFactory].getName}'
+""".stripMargin)
+
sql(s"""LOAD DATA local inpath '$resourcesPath/data_big.csv' INTO TABLE
maintable OPTIONS
|('DELIMITER'= ',', 'QUOTECHAR'= '"')""".stripMargin)
+
+sql("CREATE TABLE temp_maintable (empname String, designation String, doj
Timestamp, workgroupcategory int, workgroupcategoryname String, deptno int, " +
+ "deptname String, projectcode int, projectjoindate Timestamp,
projectenddate Timestamp,attendance int, utilization int,salary int) STORED AS
carbondata")
+
+sql(s"""LOAD DATA local inpath '$resourcesPath/data_big.csv' INTO TABLE
temp_maintable OPTIONS
+ |('DELIMITER'= ',', 'QUOTECHAR'= '"')""".stripMargin)
}
- def drop(): Unit = {
+ def dropTable(): Unit = {
sql("drop table if exists products")
sql("drop table IF EXISTS main_table")
sql("drop table IF EXISTS maintable")
+sql("drop table IF EXISTS temp_maintable")
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r495431801
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/view/timeseries/TestCreateMVWithTimeSeries.scala
##
@@ -33,17 +34,30 @@ class TestCreateMVWithTimeSeries extends QueryTest with
BeforeAndAfterAll {
override def beforeAll(): Unit = {
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
"dd-MM-")
-drop()
+dropTable()
sql("CREATE TABLE maintable (empname String, designation String, doj
Timestamp, workgroupcategory int, workgroupcategoryname String, deptno int, " +
-"deptname String, projectcode int, projectjoindate Timestamp,
projectenddate Timestamp,attendance int, utilization int,salary int) STORED AS
carbondata")
+ "deptname String, projectcode int, projectjoindate Timestamp,
projectenddate Timestamp,attendance int, utilization int,salary int) STORED AS
carbondata")
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r495431398
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/TestInsertAndOtherCommandConcurrent.scala
##
@@ -18,34 +18,18 @@
package org.apache.carbondata.spark.testsuite.iud
import java.text.SimpleDateFormat
-import java.util
-import java.util.concurrent.{Callable, ExecutorService, Executors, Future}
-
-import scala.collection.JavaConverters._
-
-import org.apache.hadoop.fs.Path
-import org.apache.spark.sql.test.util.QueryTest
-import org.apache.spark.sql.types.StructType
-import org.apache.spark.sql.{DataFrame, SaveMode}
-import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.datastore.block.SegmentProperties
-import org.apache.carbondata.core.datastore.page.ColumnPage
import org.apache.carbondata.core.exception.ConcurrentOperationException
-import org.apache.carbondata.core.features.TableOperation
-import org.apache.carbondata.core.index.dev.cgindex.{CoarseGrainIndex,
CoarseGrainIndexFactory}
-import org.apache.carbondata.core.index.dev.{IndexBuilder, IndexWriter}
-import org.apache.carbondata.core.index.{IndexInputSplit, IndexMeta, Segment}
-import org.apache.carbondata.core.indexstore.PartitionSpec
-import org.apache.carbondata.core.metadata.schema.table.{CarbonTable,
IndexSchema}
-import org.apache.carbondata.core.scan.filter.intf.ExpressionType
import org.apache.carbondata.core.util.CarbonProperties
-import org.apache.carbondata.events.Event
+import org.apache.spark.sql.test.util.QueryTest
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.{DataFrame, Row, SaveMode}
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
Review comment:
I moved all the classes realted to async functions to
AsyncExecutorutils. So relative classes are removed here. There is no need to
keep the order between spark and carbondata importing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on a change in pull request #3947:
URL: https://github.com/apache/carbondata/pull/3947#discussion_r495430881
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/TestInsertAndOtherCommandConcurrent.scala
##
@@ -18,45 +18,31 @@
package org.apache.carbondata.spark.testsuite.iud
import java.text.SimpleDateFormat
-import java.util
-import java.util.concurrent.{Callable, ExecutorService, Executors, Future}
-import scala.collection.JavaConverters._
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.exception.ConcurrentOperationException
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.spark.util.{AsyncExecutorUtils, Global,
WaitingIndexFactory}
-import org.apache.hadoop.fs.Path
import org.apache.spark.sql.test.util.QueryTest
import org.apache.spark.sql.types.StructType
-import org.apache.spark.sql.{DataFrame, SaveMode}
+import org.apache.spark.sql.{DataFrame, Row, SaveMode}
import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.datastore.block.SegmentProperties
-import org.apache.carbondata.core.datastore.page.ColumnPage
-import org.apache.carbondata.core.exception.ConcurrentOperationException
-import org.apache.carbondata.core.features.TableOperation
-import org.apache.carbondata.core.index.dev.cgindex.{CoarseGrainIndex,
CoarseGrainIndexFactory}
-import org.apache.carbondata.core.index.dev.{IndexBuilder, IndexWriter}
-import org.apache.carbondata.core.index.{IndexInputSplit, IndexMeta, Segment}
-import org.apache.carbondata.core.indexstore.PartitionSpec
-import org.apache.carbondata.core.metadata.schema.table.{CarbonTable,
IndexSchema}
-import org.apache.carbondata.core.scan.filter.intf.ExpressionType
-import org.apache.carbondata.core.util.CarbonProperties
-import org.apache.carbondata.events.Event
-
// This testsuite test insert and insert overwrite with other commands
concurrently
class TestInsertAndOtherCommandConcurrent extends QueryTest with
BeforeAndAfterAll with BeforeAndAfterEach {
- private val executorService: ExecutorService =
Executors.newFixedThreadPool(10)
+
var testData: DataFrame = _
override def beforeAll {
dropTable()
buildTestData()
-
+
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
