[GitHub] carbondata pull request #2582: [CARBONDATA-2801]Added documentation for flat...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2582#discussion_r206413320 --- Diff: docs/data-management-on-carbondata.md --- @@ -284,6 +286,20 @@ This tutorial is going to introduce all commands and data operations on CarbonDa ALTER TABLE employee SET TBLPROPERTIES (âCACHE_LEVELâ=âBlockletâ) ``` +- **Support Flat folder** + + This feature allows all carbondata and carbonindex files to keep directy under tablepath. Currently all carbondata/carbonindex files written under tablepath/Fact/Part0/Segment_NUM folder and it is not same as hive/parquet folder structure. This feature makes all files written will be directly under tablepath, it does not maintain any segment folder structure. --- End diff -- rephase to `This feature allows all data files and index files to keep directy under tablepath` ---
[GitHub] carbondata pull request #2588: [CARBONDATA-2807] Fixed data load performance...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2588#discussion_r206413037
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger/UnsafeIntermediateMerger.java
---
@@ -111,18 +108,15 @@ public void addFileToMerge(File sortTempFile) {
}
public void startFileMergingIfPossible() {
-File[] fileList = null;
-synchronized (lockObject) {
- if (procFiles.size() >=
parameters.getNumberOfIntermediateFileToBeMerged()) {
+File[] fileList;
+if (procFiles.size() >=
parameters.getNumberOfIntermediateFileToBeMerged()) {
+ synchronized (lockObject) {
fileList = procFiles.toArray(new File[procFiles.size()]);
this.procFiles = new ArrayList();
-if (LOGGER.isDebugEnabled()) {
- LOGGER
- .debug("Submitting request for intermediate merging no of
files: " + fileList.length);
-}
}
-}
-if (null != fileList) {
+ if (LOGGER.isDebugEnabled()) {
+LOGGER.debug("Sumitting request for intermediate merging no of
files: " + fileList.length);
+ }
startIntermediateMerging(fileList);
--- End diff --
no need to check not null as the old code?
---
[GitHub] carbondata pull request #2588: [CARBONDATA-2807] Fixed data load performance...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2588#discussion_r206412903
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger/UnsafeIntermediateMerger.java
---
@@ -111,18 +108,15 @@ public void addFileToMerge(File sortTempFile) {
}
public void startFileMergingIfPossible() {
-File[] fileList = null;
-synchronized (lockObject) {
- if (procFiles.size() >=
parameters.getNumberOfIntermediateFileToBeMerged()) {
+File[] fileList;
+if (procFiles.size() >=
parameters.getNumberOfIntermediateFileToBeMerged()) {
+ synchronized (lockObject) {
fileList = procFiles.toArray(new File[procFiles.size()]);
--- End diff --
better to use `procFiles.toArray(new File[0]);`
maybe better performance, see the JDK comment for toArray
---
[GitHub] carbondata issue #2583: [CARBONDATA-2803]fix wrong datasize calculation
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2583 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7655/ ---
[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2588 @kumarvishal09 Can you describe the solution in PR description? ---
[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2588 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6072/ ---
[GitHub] carbondata pull request #2587: [CARBONDATA-2806] Delete delete delta files u...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2587#discussion_r206411205
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/DeleteLoadFolders.java ---
@@ -75,21 +76,21 @@ public static void
physicalFactAndMeasureMetadataDeletion(
absoluteTableIdentifier,
currentDetails,
isForceDelete,
-specs);
+specs,
+currentDetails);
if (newAddedLoadHistoryList != null && newAddedLoadHistoryList.length
> 0) {
physicalFactAndMeasureMetadataDeletion(
absoluteTableIdentifier,
newAddedLoadHistoryList,
isForceDelete,
- specs);
+ specs,
+ currentDetails);
}
}
- public static void physicalFactAndMeasureMetadataDeletion(
- AbsoluteTableIdentifier absoluteTableIdentifier,
- LoadMetadataDetails[] loadDetails,
- boolean isForceDelete,
- List specs) {
+ private static void physicalFactAndMeasureMetadataDeletion(
+ AbsoluteTableIdentifier absoluteTableIdentifier,
LoadMetadataDetails[] loadDetails,
--- End diff --
Can you add comment to describe what is loadDetails and currLoadDetails?
---
[GitHub] carbondata pull request #2587: [CARBONDATA-2806] Delete delete delta files u...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2587#discussion_r206410979
--- Diff:
core/src/main/java/org/apache/carbondata/core/metadata/SegmentFileStore.java ---
@@ -793,25 +794,31 @@ public static void cleanSegments(CarbonTable table,
List partitio
/**
* Deletes the segment file and its physical files like partition
folders from disk
* @param tablePath
- * @param segmentFile
+ * @param segment
* @param partitionSpecs
* @throws IOException
*/
- public static void deleteSegment(String tablePath, String segmentFile,
- List partitionSpecs) throws IOException {
-SegmentFileStore fileStore = new SegmentFileStore(tablePath,
segmentFile);
+ public static void deleteSegment(String tablePath, Segment segment,
+ List partitionSpecs,
+ SegmentUpdateStatusManager updateStatusManager) throws Exception {
+SegmentFileStore fileStore = new SegmentFileStore(tablePath,
segment.getSegmentFileName());
List indexOrMergeFiles =
fileStore.readIndexFiles(SegmentStatus.SUCCESS, true);
Map> indexFilesMap = fileStore.getIndexFilesMap();
for (Map.Entry> entry : indexFilesMap.entrySet())
{
FileFactory.deleteFile(entry.getKey(),
FileFactory.getFileType(entry.getKey()));
for (String file : entry.getValue()) {
+String[] deltaFilePaths =
+updateStatusManager.getDeleteDeltaFilePath(file,
segment.getSegmentNo());
+for (String deltaFilePath : deltaFilePaths) {
+ FileFactory.deleteFile(deltaFilePath,
FileFactory.getFileType(deltaFilePath));
+}
FileFactory.deleteFile(file, FileFactory.getFileType(file));
}
}
deletePhysicalPartition(partitionSpecs, indexFilesMap,
indexOrMergeFiles, tablePath);
String segmentFilePath =
CarbonTablePath.getSegmentFilesLocation(tablePath) +
CarbonCommonConstants.FILE_SEPARATOR
--- End diff --
suggest to add a function in CarbonTablePath to get the segmentFilePath
instead of constructing it here
---
[GitHub] carbondata issue #2570: [CARBONDATA-2585]disable local dictionary by default
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2570 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6380/ ---
[GitHub] carbondata issue #2580: [CARBONDATA-2799][BloomDataMap] Fix bugs in querying...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2580 retest it please ---
[GitHub] carbondata issue #2587: [CARBONDATA-2806] Delete delete delta files upon cle...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2587 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7654/ ---
[GitHub] carbondata issue #2579: [HOTFIX][PR 2575] Fixed modular plan creation only i...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2579 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7653/ ---
[GitHub] carbondata issue #2571: [CARBONDATA-2792][schema restructure] Create externa...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2571 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6379/ ---
[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2588 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7652/ ---
[GitHub] carbondata issue #2579: [HOTFIX][PR 2575] Fixed modular plan creation only i...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2579 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6377/ ---
[GitHub] carbondata issue #2565: [HotFix][CARBONDATA-2788][BloomDataMap] Fix bugs in ...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/2565 LGTM ---
[GitHub] carbondata issue #2441: [CARBONDATA-2625] optimize CarbonReader performance
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2441 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7651/ ---
[GitHub] carbondata issue #2559: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2559 LGTM ---
[GitHub] carbondata issue #2583: [CARBONDATA-2803]fix wrong datasize calculation
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2583 LGTM ---
[GitHub] carbondata issue #2578: Fix Dictionary_Include for ComplexDataType
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2578 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6071/ ---
[GitHub] carbondata issue #2585: [CARBONDATA-2805]fix the ordering mismatch of segmen...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2585 retest this please ---
[GitHub] carbondata pull request #2578: Fix Dictionary_Include for ComplexDataType
Github user kunal642 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2578#discussion_r206396378 --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/model/LoadOption.java --- @@ -236,7 +236,7 @@ } } -if (carbonLoadModel.isCarbonTransactionalTable() && !CarbonDataProcessorUtil +if (hadoopConf != null && !CarbonDataProcessorUtil --- End diff -- why hadoopConf null check is needed? ---
[GitHub] carbondata issue #2559: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2559 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7650/ ---
[GitHub] carbondata issue #2578: Fix Dictionary_Include for ComplexDataType
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2578 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6378/ ---
[GitHub] carbondata issue #2581: [CARBONDATA-2800][Doc] Add useful tips about bloomfi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2581 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6375/ ---
[GitHub] carbondata issue #2583: [CARBONDATA-2803]fix wrong datasize calculation
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2583 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6374/ ---
[GitHub] carbondata issue #2585: [CARBONDATA-2805]fix the ordering mismatch of segmen...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2585 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6373/ ---
[GitHub] carbondata issue #2587: [CARBONDATA-2806] Delete delete delta files upon cle...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2587 retest this please ---
[GitHub] carbondata issue #2579: [HOTFIX][PR 2575] Fixed modular plan creation only i...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2579 retest this please ---
[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...
Github user brijoobopanna commented on the issue: https://github.com/apache/carbondata/pull/2588 retest this please ---
[GitHub] carbondata issue #2441: [CARBONDATA-2625] optimize CarbonReader performance
Github user brijoobopanna commented on the issue: https://github.com/apache/carbondata/pull/2441 retest this please ---
[GitHub] carbondata issue #2580: [CARBONDATA-2799][BloomDataMap] Fix bugs in querying...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2580 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7647/ ---
[GitHub] carbondata issue #2559: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user brijoobopanna commented on the issue: https://github.com/apache/carbondata/pull/2559 retest this please ---
[GitHub] carbondata issue #2570: [CARBONDATA-2585]disable local dictionary by default
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2570 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6070/ ---
[GitHub] carbondata issue #2587: [CARBONDATA-2806] Delete delete delta files upon cle...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2587 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6372/ ---
[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2588 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6371/ ---
[GitHub] carbondata issue #2441: [CARBONDATA-2625] optimize CarbonReader performance
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2441 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6370/ ---
[jira] [Commented] (CARBONDATA-2802) Creation of Bloomfilter Datamap is failing after UID,compaction,pre-aggregate datamap creation
[
https://issues.apache.org/jira/browse/CARBONDATA-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563072#comment-16563072
]
lianganping commented on CARBONDATA-2802:
-
i test it with github carbondata:master code, but i found if follow above test
steps, it maynot cause that error;
If follow these steps，it will certainly cause that error：
1.CREATE TABLE uniqdata(CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION
string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2
bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2
decimal(36,36),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1
int) STORED BY 'org.apache.carbondata.format';
2.LOAD DATA INPATH 'hdfs://hacluster/user/rahul/2000_UniqData.csv' into table
uniqdata OPTIONS('DELIMITER'=',' ,
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
3.create datamap uniqdata_agg on table uniqdata using 'preaggregate' as select
cust_name, avg(cust_id) from uniqdata group by cust_id, cust_name;
{color:#FF}4.select cust_name, avg(cust_id) from uniqdata group by cust_id,
cust_name;{color}
5.CREATE DATAMAP bloom_dob ON TABLE uniqdata USING 'bloomfilter' DMPROPERTIES
('INDEX_COLUMNS' = 'dob', 'BLOOM_SIZE'='64', 'BLOOM_FPP'='0.1');
> Creation of Bloomfilter Datamap is failing after UID,compaction,pre-aggregate
> datamap creation
> --
>
> Key: CARBONDATA-2802
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2802
> Project: CarbonData
> Issue Type: Bug
> Components: other
>Affects Versions: 1.4.1
> Environment: Spark 2.2
>Reporter: Rahul Singha
>Priority: Minor
> Labels: bloom-filter
>
> *Steps :*
> 1.CREATE TABLE uniqdata(CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION
> string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2
> bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2
> decimal(36,36),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1
> int) STORED BY 'org.apache.carbondata.format';
> 2.LOAD DATA INPATH 'hdfs://hacluster/user/rahul/2000_UniqData.csv' into table
> uniqdata OPTIONS('DELIMITER'=',' ,
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> 3.update uniqdata set (active_emui_version) = ('ACTIVE_EMUI_VERSION_1')
> where cust_id = 9000;
> 4.delete from uniqdata where cust_id = 9000;
> 5.insert into uniqdata select
> 9000,'CUST_NAME_0','ACTIVE_EMUI_VERSION_0','1970-01-01
> 01:00:03.0','1970-01-01
> 02:00:03.0',123372036854,-223372036854,12345678901.123400,22345678901.123400,1.12345674897976E10,
> -1.12345674897976E10,1;
> 6.alter table uniqdata compact 'major';
> 7.create datamap uniqdata_agg on table uniqdata using 'preaggregate' as
> select cust_name, avg(cust_id) from uniqdata group by cust_id, cust_name;
> 8.CREATE DATAMAP bloom_dob ON TABLE uniqdata USING 'bloomfilter' DMPROPERTIES
> ('INDEX_COLUMNS' = 'dob', 'BLOOM_SIZE'='64', 'BLOOM_FPP'='0.1');
> *Actual output :*
> 0: jdbc:hive2://ha-cluster/default> CREATE DATAMAP bloom_dob ON TABLE
> uniqdata USING 'bloomfilter' DMPROPERTIES ('INDEX_COLUMNS' = 'dob',
> 'BLOOM_SIZE'='64', 'BLOOM_FPP'='0.1');
> Error: org.apache.spark.SparkException: Job aborted due to stage failure:
> Task 1 in stage 199.0 failed 4 times, most recent failure: Lost task 1.3 in
> stage 199.0 (TID 484, BLR125336, executor 182):
> java.io.InvalidClassException:
> scala.collection.convert.Wrappers$MutableSetWrapper; no valid constructor
> at
> java.io.ObjectStreamClass$ExceptionInfo.newInvalidClassException(ObjectStreamClass.java:157)
> at java.io.ObjectStreamClass.checkDeserialize(ObjectStreamClass.java:862)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2041)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
> at java.util.ArrayList.readObject(ArrayList.java:797)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.l
[GitHub] carbondata issue #2544: [CARBONDATA-2776][CarbonStore] Support ingesting dat...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2544 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6368/ ---
[GitHub] carbondata issue #2524: [CARBONDATA-2532][Integration] Carbon to support spa...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2524 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6369/ ---
[jira] [Commented] (CARBONDATA-2306) Memory leak when using DataFrame.coalesce
[ https://issues.apache.org/jira/browse/CARBONDATA-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16562999#comment-16562999 ] anand commented on CARBONDATA-2306: --- Could you please share the logs for Memory Leak and Driver Heap Dump Logs. > Memory leak when using DataFrame.coalesce > - > > Key: CARBONDATA-2306 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2306 > Project: CarbonData > Issue Type: Bug > Components: spark-integration >Reporter: Jin Zhou >Priority: Major > > When using DataFrame.coalesce, taskContext object holds reader’s reference > until the task finished and > coalesce combines a lot of CarbonSparkPartition into one task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2587: [CARBONDATA-2806] Delete delete delta files upon cle...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2587 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6069/ ---
[GitHub] carbondata issue #2552: [CARBONDATA-2781] Added fix for Null Pointer Excpeti...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2552 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6367/ ---
[GitHub] carbondata issue #2567: [CARBONDATA-2790][BloomDataMap]Optimize default para...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2567 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7648/ ---
[GitHub] carbondata issue #2581: [CARBONDATA-2800][Doc] Add useful tips about bloomfi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2581 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7649/ ---
[GitHub] carbondata issue #2565: [HotFix][CARBONDATA-2788][BloomDataMap] Fix bugs in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2565 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6365/ ---
[GitHub] carbondata issue #2559: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2559 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6366/ ---
[GitHub] carbondata issue #2570: [CARBONDATA-2585]disable local dictionary by default
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2570 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6364/ ---
[GitHub] carbondata issue #2572: [CARBONDATA-2793][32k][Doc] Add 32k support in docum...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2572 only document updated, no code related ---
[GitHub] carbondata issue #2567: [CARBONDATA-2790][BloomDataMap]Optimize default para...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2567 retest this please ---
[GitHub] carbondata pull request #2565: [HotFix][CARBONDATA-2788][BloomDataMap] Fix b...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2565#discussion_r206367929
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMapSuite.scala
---
@@ -371,7 +371,7 @@ class LuceneFineGrainDataMapSuite extends QueryTest
with BeforeAndAfterAll {
"""
| CREATE TABLE datamap_test_table(id INT, name STRING, city
STRING, age INT)
| STORED BY 'carbondata'
-| TBLPROPERTIES('SORT_COLUMNS'='city,name',
'SORT_SCOPE'='GLOBAL_SORT')
+| TBLPROPERTIES('SORT_COLUMNS'='city,name',
'SORT_SCOPE'='GLOBAL_SORT', 'CACHE_LEVEL'='BLOCKLET')
--- End diff --
By default the cache_level is BLOCK which may affect the pruning info. In
some test cases in this file, they assert on the content of pruning info. So
here, I just change the cache_level to BLOCKLET, so that I do not to modify the
assertion.
---
[GitHub] carbondata pull request #2565: [HotFix][CARBONDATA-2788][BloomDataMap] Fix b...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2565#discussion_r206367642
--- Diff:
core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockDataMap.java
---
@@ -318,13 +318,22 @@ private DataMapRowImpl
loadBlockMetaInfo(CarbonRowSchema[] taskSummarySchema,
blockMinValues, blockMaxValues);
blockletCountInEachBlock.add(totalBlockletsInOneBlock);
}
-byte[] blockletCount = ArrayUtils
-.toPrimitive(blockletCountInEachBlock.toArray(new
Byte[blockletCountInEachBlock.size()]));
+byte[] blockletCount =
convertRowCountFromShortToByteArray(blockletCountInEachBlock);
// blocklet count index is the last index
summaryRow.setByteArray(blockletCount, taskSummarySchema.length - 1);
return summaryRow;
}
+ private byte[] convertRowCountFromShortToByteArray(List
blockletCountInEachBlock) {
--- End diff --
because we are using offheap store, which needs to store the bytes.
---
[GitHub] carbondata pull request #2565: [HotFix][CARBONDATA-2788][BloomDataMap] Fix b...
Github user xuchuanyin commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2565#discussion_r206367396 --- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMapFactory.java --- @@ -71,7 +71,7 @@ /** * variable for cache level BLOCKLET */ - private static final String CACHE_LEVEL_BLOCKLET = "BLOCKLET"; + public static final String CACHE_LEVEL_BLOCKLET = "BLOCKLET"; --- End diff -- Because this member needs to be accessed outside this class. Currently in `CarbonInputFormat` we need to use this variable to know the current cache level. ---
[GitHub] carbondata issue #2580: [CARBONDATA-2799][BloomDataMap] Fix bugs in querying...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2580 retest this please ---
[GitHub] carbondata issue #2586: [wip]Ui kill
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2586 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6068/ ---
[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2588 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7645/ ---
[GitHub] carbondata issue #2587: [CARBONDATA-2806] Delete delete delta files upon cle...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2587 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7646/ ---
[GitHub] carbondata issue #2441: [CARBONDATA-2625] optimize CarbonReader performance
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2441 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7644/ ---
[GitHub] carbondata issue #2571: [CARBONDATA-2792][schema restructure] Create externa...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2571 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6363/ ---
[GitHub] carbondata issue #2575: [WIP] fixed for ModularPlan exception during update ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2575 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6361/ ---
[GitHub] carbondata issue #2572: [CARBONDATA-2793][32k][Doc] Add 32k support in docum...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2572 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6362/ ---
[GitHub] carbondata issue #2585: [CARBONDATA-2805]fix the ordering mismatch of segmen...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2585 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6067/ ---
[GitHub] carbondata issue #2552: [CARBONDATA-2781] Added fix for Null Pointer Excpeti...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2552 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7643/ ---
[GitHub] carbondata issue #2559: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2559 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7642/ ---
[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2564 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7641/ ---
[GitHub] carbondata issue #2580: [CARBONDATA-2799][BloomDataMap] Fix bugs in querying...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2580 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6356/ ---
[GitHub] carbondata issue #2576: [CARBONDATA-2795] Add documentation for S3
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2576 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6360/ ---
[GitHub] carbondata issue #2577: [CARBONDATA-2796][32K]Fix data loading problem when ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2577 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6359/ ---
[GitHub] carbondata issue #2565: [HotFix][CARBONDATA-2788][BloomDataMap] Fix bugs in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2565 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7640/ ---
[GitHub] carbondata issue #2580: [CARBONDATA-2799][BloomDataMap] Fix bugs in querying...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2580 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7635/ ---
[GitHub] carbondata issue #2570: [CARBONDATA-2585]disable local dictionary by default
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2570 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7639/ ---
[GitHub] carbondata issue #2583: [CARBONDATA-2803]fix wrong datasize calculation
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2583 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6066/ ---
[GitHub] carbondata issue #2578: Fix Dictionary_Include for ComplexDataType
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2578 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6358/ ---
[GitHub] carbondata issue #2579: [HOTFIX][PR 2575] Fixed modular plan creation only i...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2579 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6357/ ---
[GitHub] carbondata issue #2578: Fix Dictionary_Include for ComplexDataType
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2578 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7638/ ---
[GitHub] carbondata issue #2571: [CARBONDATA-2792][schema restructure] Create externa...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2571 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7637/ ---
[GitHub] carbondata issue #2581: [CARBONDATA-2800][Doc] Add useful tips about bloomfi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2581 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6355/ ---
[GitHub] carbondata issue #2582: [CARBONDATA-2801]Added documentation for flat folder
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2582 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6354/ ---
[GitHub] carbondata issue #2582: [CARBONDATA-2801]Added documentation for flat folder
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2582 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6065/ ---
[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2588 retest this please ---
[GitHub] carbondata issue #2587: [CARBONDATA-2806] Delete delete delta files upon cle...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2587 retest this please ---
[GitHub] carbondata issue #2583: [CARBONDATA-2803]fix wrong datasize calculation
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2583 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7634/ ---
[GitHub] carbondata issue #2583: [CARBONDATA-2803]fix wrong datasize calculation
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2583 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6353/ ---
[GitHub] carbondata issue #2585: [CARBONDATA-2805]fix the ordering mismatch of segmen...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2585 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6352/ ---
[GitHub] carbondata issue #2585: [CARBONDATA-2805]fix the ordering mismatch of segmen...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2585 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7633/ ---
[GitHub] carbondata issue #2586: [wip]Ui kill
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2586 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6351/ ---
[GitHub] carbondata issue #2587: [CARBONDATA-2806] Delete delete delta files upon cle...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2587 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7632/ ---
[GitHub] carbondata issue #2578: Fix Dictionary_Include for ComplexDataType
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2578 Retest this please ---
[GitHub] carbondata issue #2582: [CARBONDATA-2801]Added documentation for flat folder
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2582 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6064/ ---
[GitHub] carbondata issue #2579: [HOTFIX][PR 2575] Fixed modular plan creation only i...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2579 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7631/ ---
[GitHub] carbondata issue #2587: [CARBONDATA-2806] Delete delete delta files upon cle...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2587 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6349/ ---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] add ParallelScanStore interface
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2589 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6347/ ---
[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2588 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7630/ ---
[GitHub] carbondata issue #2584: Fixed Spelling.
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2584 Can one of the admins verify this patch? ---
[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2588 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6348/ ---
[GitHub] carbondata issue #2559: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user brijoobopanna commented on the issue: https://github.com/apache/carbondata/pull/2559 retest sdv please ---
[GitHub] carbondata issue #2441: [CARBONDATA-2625] optimize CarbonReader performance
Github user brijoobopanna commented on the issue: https://github.com/apache/carbondata/pull/2441 retest sdv please ---
[GitHub] carbondata issue #2441: [CARBONDATA-2625] optimize CarbonReader performance
Github user brijoobopanna commented on the issue: https://github.com/apache/carbondata/pull/2441 retest this please ---
[GitHub] carbondata issue #2578: Fix Dictionary_Include for ComplexDataType
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2578 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7628/ ---
[GitHub] carbondata issue #2552: [CARBONDATA-2781] Added fix for Null Pointer Excpeti...
Github user brijoobopanna commented on the issue: https://github.com/apache/carbondata/pull/2552 retest this please ---
