[GitHub] [carbondata] Indhumathi27 commented on issue #3690: [CARBONDATA-3762] Block creating Materialized view's with duplicate column
Indhumathi27 commented on issue #3690: [CARBONDATA-3762] Block creating Materialized view's with duplicate column URL: https://github.com/apache/carbondata/pull/3690#issuecomment-610174540 @akashrn5 Please review and merge This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3692: [WIP]Clean old mv code
CarbonDataQA1 commented on issue #3692: [WIP]Clean old mv code URL: https://github.com/apache/carbondata/pull/3692#issuecomment-610159172 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2657/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3692: [WIP]Clean old mv code
CarbonDataQA1 commented on issue #3692: [WIP]Clean old mv code URL: https://github.com/apache/carbondata/pull/3692#issuecomment-610158940 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/947/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3696: [HOTFIX] Fix Repeated access to getSegmentProperties
CarbonDataQA1 commented on issue #3696: [HOTFIX] Fix Repeated access to getSegmentProperties URL: https://github.com/apache/carbondata/pull/3696#issuecomment-609964286 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/946/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3696: [HOTFIX] Fix Repeated access to getSegmentProperties
CarbonDataQA1 commented on issue #3696: [HOTFIX] Fix Repeated access to getSegmentProperties URL: https://github.com/apache/carbondata/pull/3696#issuecomment-609958244 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2656/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3696: [HOTFIX] Fix Repeated access to getSegmentProperties
CarbonDataQA1 commented on issue #3696: [HOTFIX] Fix Repeated access to getSegmentProperties URL: https://github.com/apache/carbondata/pull/3696#issuecomment-609884426 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/945/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] marchpure commented on issue #3696: [HOTFIX] Fix Repeated access to getSegmentProperties
marchpure commented on issue #3696: [HOTFIX] Fix Repeated access to getSegmentProperties URL: https://github.com/apache/carbondata/pull/3696#issuecomment-609882006 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] marchpure closed pull request #3607: [CARBONDATA-3670] Support compress offheap data in columnpage directly, avoding a copy of data from offhead to heap before compressed.
marchpure closed pull request #3607: [CARBONDATA-3670] Support compress offheap data in columnpage directly, avoding a copy of data from offhead to heap before compressed. URL: https://github.com/apache/carbondata/pull/3607 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3301: [CARBONDATA-3446] Support read schema of complex data type from carbon file folder path
CarbonDataQA1 commented on issue #3301: [CARBONDATA-3446] Support read schema of complex data type from carbon file folder path URL: https://github.com/apache/carbondata/pull/3301#issuecomment-609788954 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2653/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3301: [CARBONDATA-3446] Support read schema of complex data type from carbon file folder path
CarbonDataQA1 commented on issue #3301: [CARBONDATA-3446] Support read schema of complex data type from carbon file folder path URL: https://github.com/apache/carbondata/pull/3301#issuecomment-609776983 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/943/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3680: [CARBONDATA-3766] Fixed desc formatted and show segment data size issues
CarbonDataQA1 commented on issue #3680: [CARBONDATA-3766] Fixed desc formatted and show segment data size issues URL: https://github.com/apache/carbondata/pull/3680#issuecomment-609766065 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2651/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3680: [CARBONDATA-3766] Fixed desc formatted and show segment data size issues
CarbonDataQA1 commented on issue #3680: [CARBONDATA-3766] Fixed desc formatted and show segment data size issues URL: https://github.com/apache/carbondata/pull/3680#issuecomment-609764288 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/941/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3697: [HOTFIX] Refactored hive related classes to use constants
CarbonDataQA1 commented on issue #3697: [HOTFIX] Refactored hive related classes to use constants URL: https://github.com/apache/carbondata/pull/3697#issuecomment-609763528 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/940/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3697: [HOTFIX] Refactored hive related classes to use constants
CarbonDataQA1 commented on issue #3697: [HOTFIX] Refactored hive related classes to use constants URL: https://github.com/apache/carbondata/pull/3697#issuecomment-609755799 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2650/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] xubo245 commented on a change in pull request #3301: [CARBONDATA-3446] Support read schema of complex data type from carbon file folder path
xubo245 commented on a change in pull request #3301: [CARBONDATA-3446] Support
read schema of complex data type from carbon file folder path
URL: https://github.com/apache/carbondata/pull/3301#discussion_r403982927
##
File path:
store/sdk/src/test/java/org/apache/carbondata/sdk/file/CSVCarbonWriterTest.java
##
@@ -580,6 +578,91 @@ public void testWritingAndReadingStructOfFloat() throws
IOException {
}
}
+ @Test
+ public void testWritingAndReadingArrayStruct() throws IOException {
+String path = "./testWriteFilesArrayStruct";
+FileUtils.deleteDirectory(new File(path));
+
+Field[] fields = new Field[4];
+fields[0] = new Field("id", DataTypes.STRING);
+fields[1] = new Field("source", DataTypes.STRING);
+fields[2] = new Field("usage", DataTypes.STRING);
+
+List structFieldsList = new ArrayList<>();
+structFieldsList.add(new StructField("name", DataTypes.STRING));
+structFieldsList.add(new StructField("type", DataTypes.STRING));
+structFieldsList.add(new StructField("creation-time", DataTypes.STRING));
+structFieldsList.add(new StructField("property", DataTypes.STRING));
+
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] xubo245 commented on a change in pull request #3301: [CARBONDATA-3446] Support read schema of complex data type from carbon file folder path
xubo245 commented on a change in pull request #3301: [CARBONDATA-3446] Support
read schema of complex data type from carbon file folder path
URL: https://github.com/apache/carbondata/pull/3301#discussion_r403982893
##
File path:
store/sdk/src/test/java/org/apache/carbondata/sdk/file/CSVCarbonWriterTest.java
##
@@ -580,6 +578,91 @@ public void testWritingAndReadingStructOfFloat() throws
IOException {
}
}
+ @Test
+ public void testWritingAndReadingArrayStruct() throws IOException {
+String path = "./testWriteFilesArrayStruct";
+FileUtils.deleteDirectory(new File(path));
+
+Field[] fields = new Field[4];
+fields[0] = new Field("id", DataTypes.STRING);
+fields[1] = new Field("source", DataTypes.STRING);
+fields[2] = new Field("usage", DataTypes.STRING);
+
+List structFieldsList = new ArrayList<>();
+structFieldsList.add(new StructField("name", DataTypes.STRING));
+structFieldsList.add(new StructField("type", DataTypes.STRING));
+structFieldsList.add(new StructField("creation-time", DataTypes.STRING));
+structFieldsList.add(new StructField("property", DataTypes.STRING));
+
+
+StructField structTypeByList = new StructField("annotation",
DataTypes.createStructType(structFieldsList));
+
+List list = new ArrayList<>();
+list.add(structTypeByList);
+
+Field arrayType = new Field("annotations", "array", list);
+fields[3] = arrayType;
+try {
+ CarbonWriterBuilder builder =
CarbonWriter.builder().taskNo(5).outputPath(path);
+ CarbonWriter writer = builder.withCsvInput(new
Schema(fields)).writtenBy("CSVCarbonWriterTest").build();
+ for (int i = 0; i < 15; i++) {
+String[] row = new String[]{
+"robot" + (i % 10),
+String.valueOf(i),
+String.valueOf(i + "." + i),
+"sunflowers" + (i % 10) + "\002" +
"modelarts/image_classification" + "\002" + "2019-03-30 17:22:31" + "\002" +
"{\"@modelarts:start_index\":0,\"@modelarts:end_index\":5}"
++ "\001" +
+"roses" + (i % 10) + "\002" + "modelarts/image_classification"
+ "\002" + "2019-03-30 17:22:32" + "\002" +
"{\"@modelarts:start_index\":0,\"@modelarts:end_index\":5}"};
+writer.write(row);
+ }
+ writer.close();
+
+} catch (Exception e) {
+ e.printStackTrace();
+ Assert.fail();
+}
+
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] xubo245 commented on a change in pull request #3301: [CARBONDATA-3446] Support read schema of complex data type from carbon file folder path
xubo245 commented on a change in pull request #3301: [CARBONDATA-3446] Support
read schema of complex data type from carbon file folder path
URL: https://github.com/apache/carbondata/pull/3301#discussion_r403980297
##
File path:
core/src/main/java/org/apache/carbondata/core/metadata/datatype/ArrayType.java
##
@@ -65,4 +73,11 @@ public DataType getElementType() {
return elementType;
}
+ public String getElementName() {
+return elementName;
+ }
+
+ public void setElementName(String elementName) {
Review comment:
should keep base method even though not used
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] xubo245 commented on a change in pull request #3301: [CARBONDATA-3446] Support read schema of complex data type from carbon file folder path
xubo245 commented on a change in pull request #3301: [CARBONDATA-3446] Support
read schema of complex data type from carbon file folder path
URL: https://github.com/apache/carbondata/pull/3301#discussion_r403980297
##
File path:
core/src/main/java/org/apache/carbondata/core/metadata/datatype/ArrayType.java
##
@@ -65,4 +73,11 @@ public DataType getElementType() {
return elementType;
}
+ public String getElementName() {
+return elementName;
+ }
+
+ public void setElementName(String elementName) {
Review comment:
should keep base method
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] xubo245 commented on a change in pull request #3301: [CARBONDATA-3446] Support read schema of complex data type from carbon file folder path
xubo245 commented on a change in pull request #3301: [CARBONDATA-3446] Support
read schema of complex data type from carbon file folder path
URL: https://github.com/apache/carbondata/pull/3301#discussion_r403980297
##
File path:
core/src/main/java/org/apache/carbondata/core/metadata/datatype/ArrayType.java
##
@@ -65,4 +73,11 @@ public DataType getElementType() {
return elementType;
}
+ public String getElementName() {
+return elementName;
+ }
+
+ public void setElementName(String elementName) {
Review comment:
should keep base method ecven though not used
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on issue #3680: [CARBONDATA-3766] Fixed desc formatted and show segment data size issues
akashrn5 commented on issue #3680: [CARBONDATA-3766] Fixed desc formatted and show segment data size issues URL: https://github.com/apache/carbondata/pull/3680#issuecomment-609694102 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3680: [CARBONDATA-3766] Fixed desc formatted and show segment data size issues
akashrn5 commented on a change in pull request #3680: [CARBONDATA-3766] Fixed
desc formatted and show segment data size issues
URL: https://github.com/apache/carbondata/pull/3680#discussion_r403966201
##
File path:
core/src/main/java/org/apache/carbondata/core/metadata/SegmentFileStore.java
##
@@ -527,10 +527,11 @@ public static boolean updateTableStatusFile(CarbonTable
carbonTable, String segm
for (LoadMetadataDetails detail : listOfLoadFolderDetailsArray) {
// if the segments is in the list of marked for delete then update
the status.
if (segmentId.equals(detail.getLoadName())) {
+detail.setLoadEndTime(System.currentTimeMillis());
Review comment:
why we need this change? is it sepicific to non carbon segment?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on issue #3669: [CARBONDATA-3743] Added pre-priming check in the Spark job description page
vikramahuja1001 commented on issue #3669: [CARBONDATA-3743] Added pre-priming check in the Spark job description page URL: https://github.com/apache/carbondata/pull/3669#issuecomment-609692072 @kunal642 , the build is completing. There seems to be some issue with jenkins. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3659: [CARBONDATA-3738] : Delete seg. by ID is displaying as failed with invalid ID upon deleting a added parquet segment.
vikramahuja1001 commented on a change in pull request #3659: [CARBONDATA-3738]
: Delete seg. by ID is displaying as failed with invalid ID upon deleting a
added parquet segment.
URL: https://github.com/apache/carbondata/pull/3659#discussion_r403964131
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonAddLoadCommand.scala
##
@@ -93,7 +93,14 @@ case class CarbonAddLoadCommand(
// If a path is already added then we should block the adding of the same
path again.
val allSegments =
SegmentStatusManager.readLoadMetadata(carbonTable.getMetadataPath)
-if (allSegments.exists(a => a.getPath != null &&
a.getPath.equalsIgnoreCase(inputPath))) {
+// If the segment has been already loaded from the same path and its
status is SUCCESS or
+// PARTIALLY_SUCCESS, throw an exception as we should block the adding of
the same path again.
+if (allSegments.exists(a => a.getPath != null &&
a.getPath.equalsIgnoreCase(inputPath) &&
+(a.getSegmentStatus.getMessage
+
.equalsIgnoreCase(SegmentStatus.SUCCESS.toString) ||
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3659: [CARBONDATA-3738] : Delete seg. by ID is displaying as failed with invalid ID upon deleting a added parquet segment.
vikramahuja1001 commented on a change in pull request #3659: [CARBONDATA-3738]
: Delete seg. by ID is displaying as failed with invalid ID upon deleting a
added parquet segment.
URL: https://github.com/apache/carbondata/pull/3659#discussion_r403964076
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/events/DeleteSegmentByIdListener.scala
##
@@ -49,8 +51,13 @@ class DeleteSegmentByIdListener extends
OperationEventListener with Logging {
val table = metastore
.lookupRelation(Some(carbonTable.getDatabaseName),
tableName)(sparkSession)
.asInstanceOf[CarbonRelation].carbonTable
- CarbonStore
-.deleteLoadById(loadIds, carbonTable.getDatabaseName,
table.getTableName, table)
+ val dataLoadLocation =
CarbonTablePath.getTableStatusFilePath(table.getTablePath)
+ // this check is added to check if the table status file exists or
not. Delete on index
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3697: [HOTFIX] Refactored hive related classes to use constants
akashrn5 commented on a change in pull request #3697: [HOTFIX] Refactored hive related classes to use constants URL: https://github.com/apache/carbondata/pull/3697#discussion_r403959294 ## File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java ## @@ -94,8 +95,9 @@ private static void populateCarbonTable(Configuration configuration, String path // persisted in the schema CarbonTable carbonTable; AbsoluteTableIdentifier absoluteTableIdentifier = AbsoluteTableIdentifier - .from(validInputPath, getDatabaseName(configuration), getTableName(configuration)); - String schemaPath = CarbonTablePath.getSchemaFilePath(validInputPath); + .from(configuration.get(hive_metastoreConstants.META_TABLE_LOCATION), + getDatabaseName(configuration), getTableName(configuration)); + String schemaPath = CarbonTablePath.getSchemaFilePath(absoluteTableIdentifier.getTablePath()); Review comment: pass the configuration also to `getSchemaFilePath` method This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3697: [HOTFIX] Refactored hive related classes to use constants
akashrn5 commented on a change in pull request #3697: [HOTFIX] Refactored hive
related classes to use constants
URL: https://github.com/apache/carbondata/pull/3697#discussion_r403961085
##
File path:
integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonOutputFormat.java
##
@@ -61,24 +68,25 @@ public void checkOutputSpecs(FileSystem fileSystem,
JobConf jobConf) throws IOEx
public FileSinkOperator.RecordWriter getHiveRecordWriter(JobConf jc, Path
finalOutPath,
Class valueClass, boolean isCompressed, Properties
tableProperties,
Progressable progress) throws IOException {
+ThreadLocalSessionInfo.setConfigurationToCurrentThread(jc);
CarbonLoadModel carbonLoadModel = null;
-String encodedString = jc.get(LOAD_MODEL);
+// Take carbonLoadModel from container environment if set.
Review comment:
i think here first we should try to get directly from conf, if we dont get
then we can try from systemEnv, it will serve all
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on issue #3669: [CARBONDATA-3743] Added pre-priming check in the Spark job description page
kunal642 commented on issue #3669: [CARBONDATA-3743] Added pre-priming check in the Spark job description page URL: https://github.com/apache/carbondata/pull/3669#issuecomment-609682404 @vikramahuja1001 Please fix the build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] kunal642 commented on issue #3680: [CARBONDATA-3766] Fixed desc formatted and show segment data size issues
kunal642 commented on issue #3680: [CARBONDATA-3766] Fixed desc formatted and show segment data size issues URL: https://github.com/apache/carbondata/pull/3680#issuecomment-609678363 @akashrn5 @jackylk @ajantha-bhat Please review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3669: [CARBONDATA-3743] Added pre-priming check in the Spark job description page
CarbonDataQA1 commented on issue #3669: [CARBONDATA-3743] Added pre-priming check in the Spark job description page URL: https://github.com/apache/carbondata/pull/3669#issuecomment-609677985 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2647/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3697: [HOTFIX] Refactored hive related classes to use constants
CarbonDataQA1 commented on issue #3697: [HOTFIX] Refactored hive related classes to use constants URL: https://github.com/apache/carbondata/pull/3697#issuecomment-609678229 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/939/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] kunal642 opened a new pull request #3697: [HOTFIX] Refactored hive related classes to use constants
kunal642 opened a new pull request #3697: [HOTFIX] Refactored hive related classes to use constants URL: https://github.com/apache/carbondata/pull/3697 ### Why is this PR needed? Use hiveConstants instead of harcoding the values. ### What changes were proposed in this PR? Refactor code for better understanding ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3687: [CARBONDATA-3761] Remove redundant conversion for complex type insert
CarbonDataQA1 commented on issue #3687: [CARBONDATA-3761] Remove redundant conversion for complex type insert URL: https://github.com/apache/carbondata/pull/3687#issuecomment-609671853 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/936/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3669: [CARBONDATA-3743] Added pre-priming check in the Spark job description page
CarbonDataQA1 commented on issue #3669: [CARBONDATA-3743] Added pre-priming check in the Spark job description page URL: https://github.com/apache/carbondata/pull/3669#issuecomment-609669854 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/937/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3687: [CARBONDATA-3761] Remove redundant conversion for complex type insert
CarbonDataQA1 commented on issue #3687: [CARBONDATA-3761] Remove redundant conversion for complex type insert URL: https://github.com/apache/carbondata/pull/3687#issuecomment-609664557 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2646/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] xubo245 commented on issue #3317: [CARBONDATA-3461] Carbon SDK support filter equal values set.
xubo245 commented on issue #3317: [CARBONDATA-3461] Carbon SDK support filter equal values set. URL: https://github.com/apache/carbondata/pull/3317#issuecomment-609629774 @jackylk please review it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3679: [CARBONDATA-3751]: Segments are not Marked for delete if everything is deleted in a segment with index server enabled
CarbonDataQA1 commented on issue #3679: [CARBONDATA-3751]: Segments are not Marked for delete if everything is deleted in a segment with index server enabled URL: https://github.com/apache/carbondata/pull/3679#issuecomment-609603497 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2645/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3679: [CARBONDATA-3751]: Segments are not Marked for delete if everything is deleted in a segment with index server enabled
CarbonDataQA1 commented on issue #3679: [CARBONDATA-3751]: Segments are not Marked for delete if everything is deleted in a segment with index server enabled URL: https://github.com/apache/carbondata/pull/3679#issuecomment-609602209 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/935/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on issue #3669: [CARBONDATA-3743] Added pre-priming check in the Spark job description page
vikramahuja1001 commented on issue #3669: [CARBONDATA-3743] Added pre-priming check in the Spark job description page URL: https://github.com/apache/carbondata/pull/3669#issuecomment-609597461 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3659: [CARBONDATA-3738] : Delete seg. by ID is displaying as failed with invalid ID upon deleting a added parquet segment.
CarbonDataQA1 commented on issue #3659: [CARBONDATA-3738] : Delete seg. by ID is displaying as failed with invalid ID upon deleting a added parquet segment. URL: https://github.com/apache/carbondata/pull/3659#issuecomment-609595946 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2643/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3659: [CARBONDATA-3738] : Delete seg. by ID is displaying as failed with invalid ID upon deleting a added parquet segment.
CarbonDataQA1 commented on issue #3659: [CARBONDATA-3738] : Delete seg. by ID is displaying as failed with invalid ID upon deleting a added parquet segment. URL: https://github.com/apache/carbondata/pull/3659#issuecomment-609595908 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/933/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3607: [CARBONDATA-3670] Support compress offheap data in columnpage directly, avoding a copy of data from offhead to heap before compressed.
ajantha-bhat commented on issue #3607: [CARBONDATA-3670] Support compress offheap data in columnpage directly, avoding a copy of data from offhead to heap before compressed. URL: https://github.com/apache/carbondata/pull/3607#issuecomment-609595628 I think this was already handled in @jackylk 's #3638 Please check and close the PR if handled. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3696: [HOTFIX] Fix Repeated access to getSegmentProperties
ajantha-bhat commented on a change in pull request #3696: [HOTFIX] Fix Repeated
access to getSegmentProperties
URL: https://github.com/apache/carbondata/pull/3696#discussion_r403857254
##
File path:
core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletIndexFactory.java
##
@@ -653,9 +653,13 @@ public SegmentProperties
getSegmentPropertiesFromDataMap(Index coarseGrainIndex)
throws IOException {
List blocklets = new ArrayList<>();
List dataMaps = getIndexes(segment, partitions);
+if (dataMaps.size() == 0 || dataMaps == null) {
Review comment:
dataMaps cannot be null from getIndexes, it can be empty.
**So, remove null check.**
Also If we checking null, it has to be the first condition of || else we can
get NPE for null scenarios
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3696: [HOTFIX] Fix Repeated access to getSegmentProperties
ajantha-bhat commented on a change in pull request #3696: [HOTFIX] Fix Repeated
access to getSegmentProperties
URL: https://github.com/apache/carbondata/pull/3696#discussion_r403857254
##
File path:
core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletIndexFactory.java
##
@@ -653,9 +653,13 @@ public SegmentProperties
getSegmentPropertiesFromDataMap(Index coarseGrainIndex)
throws IOException {
List blocklets = new ArrayList<>();
List dataMaps = getIndexes(segment, partitions);
+if (dataMaps.size() == 0 || dataMaps == null) {
Review comment:
dataMaps cannot be null from getIndexes, it can be empty.
**So, remove it.**
Also If we checking null, it has to be the first condition of || else we can
get NPE for null scenarios
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3669: [CARBONDATA-3743] Added pre-priming check in the Spark job description page
CarbonDataQA1 commented on issue #3669: [CARBONDATA-3743] Added pre-priming check in the Spark job description page URL: https://github.com/apache/carbondata/pull/3669#issuecomment-609591954 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2642/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3687: [CARBONDATA-3761] Remove redundant conversion for complex type insert
ajantha-bhat commented on a change in pull request #3687: [CARBONDATA-3761]
Remove redundant conversion for complex type insert
URL: https://github.com/apache/carbondata/pull/3687#discussion_r403855028
##
File path:
processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java
##
@@ -240,141 +238,137 @@ public boolean getIsColumnDictionary() {
@Override
public void writeByteArray(Object input, DataOutputStream dataOutputStream,
- BadRecordLogHolder logHolder) throws IOException {
-String parsedValue =
-input == null ? null : DataTypeUtil.parseValue(input.toString(),
carbonDimension);
-String message =
logHolder.getColumnMessageMap().get(carbonDimension.getColName());
-if (this.isDictionary) {
- Integer surrogateKey;
- if (null == parsedValue) {
-surrogateKey = CarbonCommonConstants.MEMBER_DEFAULT_VAL_SURROGATE_KEY;
-if (null == message) {
- message = CarbonDataProcessorUtil
- .prepareFailureReason(carbonDimension.getColName(),
carbonDimension.getDataType());
- logHolder.getColumnMessageMap().put(carbonDimension.getColName(),
message);
- logHolder.setReason(message);
-}
- } else {
-if (dictionaryGenerator instanceof DirectDictionary && input
instanceof Long) {
- surrogateKey = ((DirectDictionary)
dictionaryGenerator).generateKey((long) input);
-} else {
- surrogateKey = dictionaryGenerator.getOrGenerateKey(parsedValue);
-}
-if (surrogateKey == CarbonCommonConstants.INVALID_SURROGATE_KEY) {
- surrogateKey =
CarbonCommonConstants.MEMBER_DEFAULT_VAL_SURROGATE_KEY;
- message = CarbonDataProcessorUtil
- .prepareFailureReason(carbonDimension.getColName(),
carbonDimension.getDataType());
- logHolder.getColumnMessageMap().put(carbonDimension.getColName(),
message);
- logHolder.setReason(message);
-}
- }
- dataOutputStream.writeInt(surrogateKey);
-} else {
- // Transform into ByteArray for No Dictionary.
- // TODO have to refactor and place all the cases present in
NonDictionaryFieldConverterImpl
- if (null == parsedValue && this.carbonDimension.getDataType() !=
DataTypes.STRING) {
-updateNullValue(dataOutputStream, logHolder);
- } else if (null == parsedValue || parsedValue.equals(nullFormat)) {
+ BadRecordLogHolder logHolder, Boolean isWithoutConverter) throws
IOException {
+String parsedValue = null;
+if (null == input ||
+(this.carbonDimension.getDataType() == DataTypes.STRING &&
input.equals(nullFormat))) {
+ updateNullValue(dataOutputStream, logHolder);
+ return;
+}
+if (!isWithoutConverter) {
+ parsedValue = DataTypeUtil.parseValue(input.toString(), carbonDimension);
+ if (null == parsedValue || (this.carbonDimension.getDataType() ==
DataTypes.STRING
+ && parsedValue.equals(nullFormat))) {
updateNullValue(dataOutputStream, logHolder);
- } else {
-String dateFormat = null;
-if (this.carbonDimension.getDataType() == DataTypes.DATE) {
- dateFormat = carbonDimension.getDateFormat();
-} else if (this.carbonDimension.getDataType() == DataTypes.TIMESTAMP) {
- dateFormat = carbonDimension.getTimestampFormat();
-}
-try {
- if (!this.carbonDimension.getUseActualData()) {
-byte[] value = null;
-if (isDirectDictionary) {
- int surrogateKey;
- if (!(input instanceof Long) && !(input instanceof Integer)) {
-SimpleDateFormat parser = new
SimpleDateFormat(getDateFormat(carbonDimension));
-parser.parse(parsedValue);
- }
- // If the input is a long value then this means that logical
type was provided by
- // the user using AvroCarbonWriter. In this case directly
generate surrogate key
- // using dictionaryGenerator.
- if (dictionaryGenerator instanceof DirectDictionary && input
instanceof Long) {
-surrogateKey = ((DirectDictionary)
dictionaryGenerator).generateKey((long) input);
- } else if (dictionaryGenerator instanceof DirectDictionary
- && input instanceof Integer) {
-// In case of file format, for complex type date or time type,
input data comes as a
-// Integer object, so just assign the surrogate key with the
input object value
-surrogateKey = (int) input;
- } else {
-surrogateKey =
dictionaryGenerator.getOrGenerateKey(parsedValue);
- }
- if (surrogateKey == CarbonCommonConstants.INVALID_SURROGATE_KEY)
{
-value = new byte[0];
+return;
+ }
+}
+// Transform into ByteArray for No Dictionary.
+String dateFormat = null;
+
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3687: [CARBONDATA-3761] Remove redundant conversion for complex type insert
ajantha-bhat commented on a change in pull request #3687: [CARBONDATA-3761]
Remove redundant conversion for complex type insert
URL: https://github.com/apache/carbondata/pull/3687#discussion_r403847723
##
File path:
processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java
##
@@ -240,141 +238,137 @@ public boolean getIsColumnDictionary() {
@Override
public void writeByteArray(Object input, DataOutputStream dataOutputStream,
Review comment:
This part of code is messed up. Need to check and refactor `useActualData`
first (which needs some analysis). Without that these flows will be like this.
I will try to refactor to some extent now
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3669: [CARBONDATA-3743] Added pre-priming check in the Spark job description page
CarbonDataQA1 commented on issue #3669: [CARBONDATA-3743] Added pre-priming check in the Spark job description page URL: https://github.com/apache/carbondata/pull/3669#issuecomment-609585667 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/932/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3687: [CARBONDATA-3761] Remove redundant conversion for complex type insert
ajantha-bhat commented on a change in pull request #3687: [CARBONDATA-3761]
Remove redundant conversion for complex type insert
URL: https://github.com/apache/carbondata/pull/3687#discussion_r403847723
##
File path:
processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java
##
@@ -240,141 +238,137 @@ public boolean getIsColumnDictionary() {
@Override
public void writeByteArray(Object input, DataOutputStream dataOutputStream,
Review comment:
This part of code is messed up. Need to check and refactor `useActualData`
first (which needs some analysis). Without that these flows will be like this
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
