[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3965: [CARBONDATA-4016] NPE and FileNotFound in Show Segments and Insert Stage
CarbonDataQA1 commented on pull request #3965: URL: https://github.com/apache/carbondata/pull/3965#issuecomment-701160608 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4280/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3965: [CARBONDATA-4016] NPE and FileNotFound in Show Segments and Insert Stage
CarbonDataQA1 commented on pull request #3965: URL: https://github.com/apache/carbondata/pull/3965#issuecomment-701160434 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2535/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3965: [CARBONDATA-4016] NPE and FileNotFound in Show Segments and Insert Stage
CarbonDataQA1 commented on pull request #3965: URL: https://github.com/apache/carbondata/pull/3965#issuecomment-701129169 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2533/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3965: [CARBONDATA-4016] NPE and FileNotFound in Show Segments and Insert Stage
CarbonDataQA1 commented on pull request #3965: URL: https://github.com/apache/carbondata/pull/3965#issuecomment-701128059 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4278/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3945: [CARBONDATA-3991]Fix the set modified time function on S3 and Alluxio…
CarbonDataQA1 commented on pull request #3945: URL: https://github.com/apache/carbondata/pull/3945#issuecomment-701128054 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4277/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3945: [CARBONDATA-3991]Fix the set modified time function on S3 and Alluxio…
CarbonDataQA1 commented on pull request #3945: URL: https://github.com/apache/carbondata/pull/3945#issuecomment-701127913 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2532/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Zhangshunyu commented on pull request #3964: [CARBONDATA-4015] Remove hardcode of Lock configuration in Update and Delete
Zhangshunyu commented on pull request #3964: URL: https://github.com/apache/carbondata/pull/3964#issuecomment-701106628 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Klaus-xjp commented on pull request #3964: [CARBONDATA-4015] Remove hardcode of Lock configuration in Update and Delete
Klaus-xjp commented on pull request #3964: URL: https://github.com/apache/carbondata/pull/3964#issuecomment-701105544 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Klaus-xjp commented on pull request #3945: [CARBONDATA-3991]Fix the set modified time function on S3 and Alluxio…
Klaus-xjp commented on pull request #3945: URL: https://github.com/apache/carbondata/pull/3945#issuecomment-701097047 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3965: [CARBONDATA-4016] NPE and FileNotFound in Show Segments and Insert Stage
CarbonDataQA1 commented on pull request #3965: URL: https://github.com/apache/carbondata/pull/3965#issuecomment-700906032 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4276/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3945: [CARBONDATA-3991]Fix the set modified time function on S3 and Alluxio…
CarbonDataQA1 commented on pull request #3945: URL: https://github.com/apache/carbondata/pull/3945#issuecomment-700900738 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2530/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3965: [CARBONDATA-4016] NPE and FileNotFound in Show Segments and Insert Stage
CarbonDataQA1 commented on pull request #3965: URL: https://github.com/apache/carbondata/pull/3965#issuecomment-700899917 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2531/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3945: [CARBONDATA-3991]Fix the set modified time function on S3 and Alluxio…
CarbonDataQA1 commented on pull request #3945: URL: https://github.com/apache/carbondata/pull/3945#issuecomment-700894338 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4275/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3965: [CARBONDATA-4016] NPE and FileNotFound in Show Segments and Insert Stage
CarbonDataQA1 commented on pull request #3965: URL: https://github.com/apache/carbondata/pull/3965#issuecomment-700844396 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2529/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3965: [CARBONDATA-4016] NPE and FileNotFound in Show Segments and Insert Stage
CarbonDataQA1 commented on pull request #3965: URL: https://github.com/apache/carbondata/pull/3965#issuecomment-700843530 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4274/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Klaus-xjp commented on a change in pull request #3945: [CARBONDATA-3991]Fix the set modified time function on S3 and Alluxio…
Klaus-xjp commented on a change in pull request #3945:
URL: https://github.com/apache/carbondata/pull/3945#discussion_r496864594
##
File path:
core/src/main/java/org/apache/carbondata/core/datastore/filesystem/LocalCarbonFile.java
##
@@ -421,6 +421,12 @@ public boolean createNewFile(final FsPermission
permission) throws IOException {
return file.createNewFile();
}
+ @Override
+ public boolean createNewFile(Boolean overwrite, final FsPermission
permission)
+ throws IOException {
+return file.createNewFile();
Review comment:
i have changed this function
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Klaus-xjp commented on a change in pull request #3945: [CARBONDATA-3991]Fix the set modified time function on S3 and Alluxio…
Klaus-xjp commented on a change in pull request #3945:
URL: https://github.com/apache/carbondata/pull/3945#discussion_r496853936
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertFromStageCommand.scala
##
@@ -507,17 +507,17 @@ case class CarbonInsertFromStageCommand(
override def call(): (CarbonFile, CarbonFile, Boolean) = {
try {
// Get the loading files path
-val stageLoadingFile =
- FileFactory.getCarbonFile(stagePath +
-File.separator + files._1.getName +
CarbonTablePath.LOADING_FILE_SUFFIX);
+val stageLoadingFilePath = stagePath + File.separator +
files._1.getName +
+ CarbonTablePath.LOADING_FILE_SUFFIX
Review comment:
i have reformat this line
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3964: [CARBONDATA-4015] Remove hardcode of Lock configuration in Update and Delete
marchpure commented on pull request #3964: URL: https://github.com/apache/carbondata/pull/3964#issuecomment-700805019 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3964: [CARBONDATA-4015] Remove hardcode of Lock configuration in Update and Delete
CarbonDataQA1 commented on pull request #3964: URL: https://github.com/apache/carbondata/pull/3964#issuecomment-700804521 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4273/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3964: [CARBONDATA-4015] Remove hardcode of Lock configuration in Update and Delete
CarbonDataQA1 commented on pull request #3964: URL: https://github.com/apache/carbondata/pull/3964#issuecomment-700803544 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2528/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Klaus-xjp commented on a change in pull request #3945: [CARBONDATA-3991]Fix the set modified time function on S3 and Alluxio…
Klaus-xjp commented on a change in pull request #3945:
URL: https://github.com/apache/carbondata/pull/3945#discussion_r496844035
##
File path:
core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java
##
@@ -293,6 +293,32 @@ public static boolean createNewFile(
return getCarbonFile(filePath).createNewFile(permission);
}
+ public static boolean createNewFile(String filePath, Boolean overwrite,
+ final FsPermission permission) throws IOException {
+return getCarbonFile(filePath).createNewFile(overwrite, permission);
+ }
+
+ public static long setCurrentTimetoLastModifiedTime(String filePath) throws
IOException {
Review comment:
i have renamed this function
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
CarbonDataQA1 commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-700787176 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4271/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
CarbonDataQA1 commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-700785754 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2526/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure opened a new pull request #3965: [CARBONDATA-4016] NPE and FileNotFound in Show Segments and Insert Stage
marchpure opened a new pull request #3965: URL: https://github.com/apache/carbondata/pull/3965 ### Why is this PR needed? ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (CARBONDATA-4016) NPE and FileNotFound in Show Segments and Insert Stage
Xingjun Hao created CARBONDATA-4016: --- Summary: NPE and FileNotFound in Show Segments and Insert Stage Key: CARBONDATA-4016 URL: https://issues.apache.org/jira/browse/CARBONDATA-4016 Project: CarbonData Issue Type: Bug Components: flink-integration, spark-integration Affects Versions: 2.0.1 Reporter: Xingjun Hao Fix For: 2.1.0 # Insert Stage, While Spark read Stages which are writting by Flink in the meanwhile, JSONFORMAT EXCEPTION will be thrown. # Show Segments with STAGE, when read stages which are writting by Flink or deleting by spark. JSONFORMAT EXCEPTION will be thrown # Show Segment will load partition info for non-partition table, which shall be avoided. # In getLastModifiedTime of TableStatus, if the loadendtime is empty, getLastModifiedTime throw NPE. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] Kejian-Li commented on pull request #3964: [CARBONDATA-4015]Support Lock Configuration can be configured in Update and Delete
Kejian-Li commented on pull request #3964: URL: https://github.com/apache/carbondata/pull/3964#issuecomment-700722195 > Or change the pr title to "Remove hardcode of Lock configuration in Update and Delete" > Or change the pr title to "Remove hardcode of Lock configuration in Update and Delete" Ok This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li edited a comment on pull request #3964: [CARBONDATA-4015]Support Lock Configuration can be configured in Update and Delete
Kejian-Li edited a comment on pull request #3964: URL: https://github.com/apache/carbondata/pull/3964#issuecomment-700722195 > Or change the pr title to "Remove hardcode of Lock configuration in Update and Delete" Ok This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3964: [CARBONDATA-4015]Support Lock Configuration can be configured in Update and Delete
marchpure commented on pull request #3964: URL: https://github.com/apache/carbondata/pull/3964#issuecomment-700720893 Or change the pr title to "Remove hardcode of Lock configuration in Update and Delete" This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3964: [CARBONDATA-4015]Support Lock Configuration can be configured in Update and Delete
marchpure commented on pull request #3964: URL: https://github.com/apache/carbondata/pull/3964#issuecomment-700719957 In description: it is neccessary to be fixed in code => it is not neccessary to be fixed in code This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on pull request #3964: [CARBONDATA-4015]Allow retryCount and retryInterval of updateLock and compactLock to be configured by users or using default value.
Kejian-Li commented on pull request #3964: URL: https://github.com/apache/carbondata/pull/3964#issuecomment-700719022 > Change the PR Title to 'Support Lock Configuration can be configured in Update and Delete' Ok This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure edited a comment on pull request #3964: [CARBONDATA-4015]Allow retryCount and retryInterval of updateLock and compactLock to be configured by users or using default val
marchpure edited a comment on pull request #3964: URL: https://github.com/apache/carbondata/pull/3964#issuecomment-700718312 Change the PR Title to 'Support Lock configuration can be configured in Update and Delete' This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3964: [CARBONDATA-4015]Allow retryCount and retryInterval of updateLock and compactLock to be configured by users or using default value.
marchpure commented on pull request #3964: URL: https://github.com/apache/carbondata/pull/3964#issuecomment-700718312 Change the PR Title to 'Support Lock configuration can be configured in Update and Delete' This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure edited a comment on pull request #3964: [CARBONDATA-4015]Allow retryCount and retryInterval of updateLock and compactLock to be configured by users or using default val
marchpure edited a comment on pull request #3964: URL: https://github.com/apache/carbondata/pull/3964#issuecomment-700718312 Change the PR Title to 'Support Lock Configuration can be configured in Update and Delete' This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3964: [CARBONDATA-4015]Allow retryCount and retryInterval of updateLock and compactLock to be configured by users or using default value.
CarbonDataQA1 commented on pull request #3964: URL: https://github.com/apache/carbondata/pull/3964#issuecomment-700716620 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4272/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on pull request #3964: [CARBONDATA-4015]Allow retryCount and retryInterval of updateLock and compactLock to be configured by users or using default value.
Kejian-Li commented on pull request #3964: URL: https://github.com/apache/carbondata/pull/3964#issuecomment-700713068 @marchpure please help review this PR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li opened a new pull request #3964: [CARBONDATA-4015]Allow retryCount and retryInterval of updateLock and compactLock to be configured by users or using default value.
Kejian-Li opened a new pull request #3964: URL: https://github.com/apache/carbondata/pull/3964 ### Why is this PR needed? Retrycount and retryInterval of compactLock and updateLock is fixed as "3" when they try to get lcok, default values of these two parameters is exactly "3", therefore, it is neccessary to be fixed in code, it is also good to allow users to configure them in the carbon.properties by removing them. ### What changes were proposed in this PR? Removed the parameters of compactLock and updateLock when they try to get lock. Besides, I made some obvious but trivial changes and modified or added some testcases. ### Does this PR introduce any user interface change? - No ### Is any new testcase added? - No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li closed pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li closed pull request #3947: URL: https://github.com/apache/carbondata/pull/3947 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-700697188 > @Kejian-Li i can just see the test file changes and no source file changes for the issue which you described. So im confused what exactly or how exactly the problem is solved. can you please explain? My idea and implementation is not considerable, so I just removed the relative code concurrency > @Kejian-Li i can just see the test file changes and no source file changes for the issue which you described. So im confused what exactly or how exactly the problem is solved. can you please explain? My idea and implementation is not considerable, I will explain this later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Kejian-Li edited a comment on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
Kejian-Li edited a comment on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-700697188 > @Kejian-Li i can just see the test file changes and no source file changes for the issue which you described. So im confused what exactly or how exactly the problem is solved. can you please explain? My idea and implementation is not considerable, I will explain this later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on a change in pull request #3945: [CARBONDATA-3991]Fix the set modified time function on S3 and Alluxio…
marchpure commented on a change in pull request #3945:
URL: https://github.com/apache/carbondata/pull/3945#discussion_r496708385
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertFromStageCommand.scala
##
@@ -507,17 +507,17 @@ case class CarbonInsertFromStageCommand(
override def call(): (CarbonFile, CarbonFile, Boolean) = {
try {
// Get the loading files path
-val stageLoadingFile =
- FileFactory.getCarbonFile(stagePath +
-File.separator + files._1.getName +
CarbonTablePath.LOADING_FILE_SUFFIX);
+val stageLoadingFilePath = stagePath + File.separator +
files._1.getName +
+ CarbonTablePath.LOADING_FILE_SUFFIX
Review comment:
take care about format
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on a change in pull request #3945: [CARBONDATA-3991]Fix the set modified time function on S3 and Alluxio…
marchpure commented on a change in pull request #3945:
URL: https://github.com/apache/carbondata/pull/3945#discussion_r495646765
##
File path:
core/src/main/java/org/apache/carbondata/core/datastore/filesystem/LocalCarbonFile.java
##
@@ -421,6 +421,12 @@ public boolean createNewFile(final FsPermission
permission) throws IOException {
return file.createNewFile();
}
+ @Override
+ public boolean createNewFile(Boolean overwrite, final FsPermission
permission)
+ throws IOException {
+return file.createNewFile();
Review comment:
we can't discard parameters like this
##
File path:
core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java
##
@@ -293,6 +293,32 @@ public static boolean createNewFile(
return getCarbonFile(filePath).createNewFile(permission);
}
+ public static boolean createNewFile(String filePath, Boolean overwrite,
+ final FsPermission permission) throws IOException {
+return getCarbonFile(filePath).createNewFile(overwrite, permission);
+ }
+
+ public static long setCurrentTimetoLastModifiedTime(String filePath) throws
IOException {
Review comment:
setLastModifiedTimetoCurrentTimeto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (CARBONDATA-4015) RetryCount and retryInterval of updateLock and compactLock is fixed as 3 when they try to get lock
[ https://issues.apache.org/jira/browse/CARBONDATA-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kejian Li updated CARBONDATA-4015: -- Summary: RetryCount and retryInterval of updateLock and compactLock is fixed as 3 when they try to get lock (was: RetryCount and retryInterval of updateLock and compactLock when they try to get lock is fixed as 3) > RetryCount and retryInterval of updateLock and compactLock is fixed as 3 when > they try to get lock > --- > > Key: CARBONDATA-4015 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4015 > Project: CarbonData > Issue Type: Improvement > Components: spark-integration >Affects Versions: 2.0.1 >Reporter: Kejian Li >Priority: Trivial > Fix For: 2.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4015) RetryCount and retryInterval of updateLock and compactLock when they try to get lock is fixed as 3
[ https://issues.apache.org/jira/browse/CARBONDATA-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kejian Li updated CARBONDATA-4015: -- Summary: RetryCount and retryInterval of updateLock and compactLock when they try to get lock is fixed as 3 (was: Retry times and intervel of updateLock and compactLock when they try to get lock is fixed as 3) > RetryCount and retryInterval of updateLock and compactLock when they try to > get lock is fixed as 3 > -- > > Key: CARBONDATA-4015 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4015 > Project: CarbonData > Issue Type: Improvement > Components: spark-integration >Affects Versions: 2.0.1 >Reporter: Kejian Li >Priority: Trivial > Fix For: 2.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4015) Retry times and intervel of updateLock and compactLock when they try to get lock is fixed as 3
[ https://issues.apache.org/jira/browse/CARBONDATA-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kejian Li updated CARBONDATA-4015: -- Summary: Retry times and intervel of updateLock and compactLock when they try to get lock is fixed as 3 (was: Remove the parameters of updateLock and compactLock when they get lock) > Retry times and intervel of updateLock and compactLock when they try to get > lock is fixed as 3 > -- > > Key: CARBONDATA-4015 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4015 > Project: CarbonData > Issue Type: Improvement > Components: spark-integration >Affects Versions: 2.0.1 >Reporter: Kejian Li >Priority: Trivial > Fix For: 2.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4015) Remove the parameters of updateLock and compactLock when they get lock
Kejian Li created CARBONDATA-4015: - Summary: Remove the parameters of updateLock and compactLock when they get lock Key: CARBONDATA-4015 URL: https://issues.apache.org/jira/browse/CARBONDATA-4015 Project: CarbonData Issue Type: Improvement Components: spark-integration Affects Versions: 2.0.1 Reporter: Kejian Li Fix For: 2.1.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-4014) Support Change Column Comment
[ https://issues.apache.org/jira/browse/CARBONDATA-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-4014. -- Resolution: Fixed > Support Change Column Comment > - > > Key: CARBONDATA-4014 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4014 > Project: CarbonData > Issue Type: New Feature > Components: sql >Affects Versions: 2.0.1 >Reporter: Xingjun Hao >Priority: Minor > Fix For: 2.1.0 > > Time Spent: 6h 50m > Remaining Estimate: 0h > > Now, we support add comment when CREATE TABLE and ADD COLUMN. but do not > support alter comment of specified column. > We shall support alter comment with hive syntax > "ALTER TABLE table_name CHANGE [COLUMN] col_name col_name data_type [COMMENT > col_comment]" > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] asfgit closed pull request #3960: [CARBONDATA-4014] Support Change Column Comment
asfgit closed pull request #3960: URL: https://github.com/apache/carbondata/pull/3960 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] kunal642 commented on pull request #3960: [CARBONDATA-4014] Support Change Column Comment
kunal642 commented on pull request #3960: URL: https://github.com/apache/carbondata/pull/3960#issuecomment-700659418 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3960: [CARBONDATA-4014] Support Change Column Comment
CarbonDataQA1 commented on pull request #3960: URL: https://github.com/apache/carbondata/pull/3960#issuecomment-700650642 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2524/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3960: [CARBONDATA-4014] Support Change Column Comment
CarbonDataQA1 commented on pull request #3960: URL: https://github.com/apache/carbondata/pull/3960#issuecomment-700647666 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4269/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3960: [CARBONDATA-4014] Support Change Column Comment
CarbonDataQA1 commented on pull request #3960: URL: https://github.com/apache/carbondata/pull/3960#issuecomment-700553209 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2523/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3960: [CARBONDATA-4014] Support Change Column Comment
CarbonDataQA1 commented on pull request #3960: URL: https://github.com/apache/carbondata/pull/3960#issuecomment-700545126 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4268/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
akashrn5 commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-700534709 @Kejian-Li i can just see the test file changes and no source file changes for the issue which you described. So im confused what exactly or how exactly the problem is solved. can you please explain? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
akashrn5 commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-700503471 @Kejian-Li from the PR description, you have mentioned the problem is we cannot run the update to table when the insert is in progress, which is valid, that's because we want to maintain the data consistency. But you suggest we remove that check to allow the update. How do you plan to manage the data consistency when the insert in progress? You have not mentioned clearly in the description and its not clear from changes also. As @Zhangshunyu asked same doubt in above comments, there is no clear reply. Can you please explain the solution you have given here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3952: [CARBONDATA-4006] Fix for currentUser as NULL in getcount method during index server fallback mode
CarbonDataQA1 commented on pull request #3952: URL: https://github.com/apache/carbondata/pull/3952#issuecomment-700502890 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4267/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3952: [CARBONDATA-4006] Fix for currentUser as NULL in getcount method during index server fallback mode
CarbonDataQA1 commented on pull request #3952: URL: https://github.com/apache/carbondata/pull/3952#issuecomment-700501338 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2522/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] kunal642 commented on a change in pull request #3960: [CARBONDATA-4014] Support Change Column Comment
kunal642 commented on a change in pull request #3960:
URL: https://github.com/apache/carbondata/pull/3960#discussion_r496461049
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableColRenameDataTypeChangeCommand.scala
##
@@ -150,45 +141,50 @@ private[sql] case class
CarbonAlterTableColRenameDataTypeChangeCommand(
}
val newColumnPrecision =
alterTableColRenameAndDataTypeChangeModel.dataTypeInfo.precision
val newColumnScale =
alterTableColRenameAndDataTypeChangeModel.dataTypeInfo.scale
- if (alterTableColRenameAndDataTypeChangeModel.isColumnRename) {
-// validate the columns to be renamed
-validColumnsForRenaming(carbonColumns, oldCarbonColumn.head,
carbonTable)
-// if the datatype is source datatype, then it is just a column rename
operation, else set
-// the isDataTypeChange flag to true
-if (oldCarbonColumn.head.getDataType.getName
-
.equalsIgnoreCase(alterTableColRenameAndDataTypeChangeModel.dataTypeInfo.dataType))
{
- val newColumnPrecision =
-alterTableColRenameAndDataTypeChangeModel.dataTypeInfo.precision
- val newColumnScale =
alterTableColRenameAndDataTypeChangeModel.dataTypeInfo.scale
- // if the source datatype is decimal and there is change in
precision and scale, then
- // along with rename, datatype change is also required for the
command, so set the
- // isDataTypeChange flag to true in this case
- if
(oldCarbonColumn.head.getDataType.getName.equalsIgnoreCase("decimal") &&
-
(oldCarbonColumn.head.getDataType.asInstanceOf[DecimalType].getPrecision !=
- newColumnPrecision ||
-
oldCarbonColumn.head.getDataType.asInstanceOf[DecimalType].getScale !=
- newColumnScale)) {
-isDataTypeChange = true
- }
-} else {
+ // set isDataTypeChange flag
+ if (oldCarbonColumn.head.getDataType.getName
+
.equalsIgnoreCase(alterTableColRenameAndDataTypeChangeModel.dataTypeInfo.dataType))
{
+val newColumnPrecision =
+ alterTableColRenameAndDataTypeChangeModel.dataTypeInfo.precision
+val newColumnScale =
alterTableColRenameAndDataTypeChangeModel.dataTypeInfo.scale
+// if the source datatype is decimal and there is change in precision
and scale, then
+// along with rename, datatype change is also required for the
command, so set the
+// isDataTypeChange flag to true in this case
+if
(oldCarbonColumn.head.getDataType.getName.equalsIgnoreCase("decimal") &&
+
(oldCarbonColumn.head.getDataType.asInstanceOf[DecimalType].getPrecision !=
+ newColumnPrecision ||
+
oldCarbonColumn.head.getDataType.asInstanceOf[DecimalType].getScale !=
+ newColumnScale)) {
isDataTypeChange = true
}
} else {
isDataTypeChange = true
}
- if (isDataTypeChange) {
-// if column datatype change operation is on partition column, then
fail the datatype change
-// operation
-if (null != carbonTable.getPartitionInfo) {
- val partitionColumns =
carbonTable.getPartitionInfo.getColumnSchemaList
- partitionColumns.asScala.foreach {
-col =>
- if (col.getColumnName.equalsIgnoreCase(oldColumnName)) {
-throw new MalformedCarbonCommandException(
- s"Alter datatype of the partition column $newColumnName is
not allowed")
- }
- }
+ // If there is no columnrename and datatype change and comment change
+ // return directly without execution
+ if (!isColumnRename && !isDataTypeChange && !newColumnComment.isDefined)
{
+return Seq.empty
+ }
+ // if column datatype change operation is on partition column, then fail
the
+ // chang column operation
+ if (null != carbonTable.getPartitionInfo) {
+val partitionColumns = carbonTable.getPartitionInfo.getColumnSchemaList
+partitionColumns.asScala.foreach {
+ col =>
+if (col.getColumnName.equalsIgnoreCase(oldColumnName)) {
+ throw new MalformedCarbonCommandException(
+s"Change Column Operation failed. Renaming or ChangeType or " +
Review comment:
Please change this to
`throw new InvalidOperationException("Alter on partition columns is not
supported")`
And add a test case for this
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
marchpure commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-700490385 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #3947: [CARBONDATA-4003] Improve IUD Concurrency
akashrn5 commented on pull request #3947: URL: https://github.com/apache/carbondata/pull/3947#issuecomment-700489664 > please check mail list: > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-Parallel-compaction-and-update-td100338.html > > How do we process the parallel compaction(not only loading) and update? I involved in that solution derivation, But its recommended and community's suggestion to start working on it after segment refactoring. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Zhangshunyu commented on pull request #3960: [CARBONDATA-4014] Support Change Column Comment
Zhangshunyu commented on pull request #3960: URL: https://github.com/apache/carbondata/pull/3960#issuecomment-700485310 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (CARBONDATA-3830) Presto read support for complex columns
[ https://issues.apache.org/jira/browse/CARBONDATA-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3830: --- Description: This feature is to enable Presto to read complex columns from carbon file. Complex columns include - array, map and struct. This design document handles only for array type. Map and Struct types will be handled later. PR - https://github.com/apache/carbondata/pull/3887 was: This feature is to enable Presto to read complex columns from carbon file. Complex columns include - array, map and struct. This design document handles only for array type. Map and Struct types will be handled later. PR - [https://github.com/apache/carbondata/pull/3773] > Presto read support for complex columns > --- > > Key: CARBONDATA-3830 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3830 > Project: CarbonData > Issue Type: New Feature > Components: core, presto-integration >Reporter: Akshay >Assignee: Ajantha Bhat >Priority: Minor > Fix For: 2.1.0 > > Attachments: Presto Read Support.pdf > > Time Spent: 33h 40m > Remaining Estimate: 0h > > This feature is to enable Presto to read complex columns from carbon file. > Complex columns include - array, map and struct. > This design document handles only for array type. > Map and Struct types will be handled later. > > PR - https://github.com/apache/carbondata/pull/3887 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3830) Presto read support for complex columns
[ https://issues.apache.org/jira/browse/CARBONDATA-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated CARBONDATA-3830: --- Description: This feature is to enable Presto to read complex columns from carbon file. Complex columns include - array, map and struct. This design document handles only for array and struct type. Map type will be handled later. PR - [https://github.com/apache/carbondata/pull/3887] was: This feature is to enable Presto to read complex columns from carbon file. Complex columns include - array, map and struct. This design document handles only for array type. Map and Struct types will be handled later. PR - https://github.com/apache/carbondata/pull/3887 > Presto read support for complex columns > --- > > Key: CARBONDATA-3830 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3830 > Project: CarbonData > Issue Type: New Feature > Components: core, presto-integration >Reporter: Akshay >Assignee: Ajantha Bhat >Priority: Minor > Fix For: 2.1.0 > > Attachments: Presto Read Support.pdf > > Time Spent: 33h 40m > Remaining Estimate: 0h > > This feature is to enable Presto to read complex columns from carbon file. > Complex columns include - array, map and struct. > This design document handles only for array and struct type. > Map type will be handled later. > > PR - [https://github.com/apache/carbondata/pull/3887] -- This message was sent by Atlassian Jira (v8.3.4#803005)
