[GitHub] carbondata issue #2453: [CARBONDATA-2528][MV] Fixed order by in mv and aggre...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2453 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5875/ ---
[GitHub] carbondata issue #2423: [CARBONDATA-2530][MV] Fix wrong data displayed when ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2423 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7104/ ---
[GitHub] carbondata issue #2431: [MINOR] Adding a testcase for stream-table join in S...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2431 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7103/ ---
[GitHub] carbondata issue #2489: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user kunal642 commented on the issue: https://github.com/apache/carbondata/pull/2489 LGTM ---
[GitHub] carbondata issue #2455: [CARBONDATA-2512][32k] Support writing longstring th...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2455 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5874/ ---
[jira] [Created] (CARBONDATA-2743) [MV] After MV creation limit queries Throws Exceptions for table which does not have mv data map
Babulal created CARBONDATA-2743:
---
Summary: [MV] After MV creation limit queries Throws Exceptions
for table which does not have mv data map
Key: CARBONDATA-2743
URL: https://issues.apache.org/jira/browse/CARBONDATA-2743
Project: CarbonData
Issue Type: Bug
Reporter: Babulal
0: jdbc:hive2://10.18.16.173:23040/default> create table mytest_50_s13 (name
string,rownumber string, m1 float) stored by 'carbondata'
TBLPROPERTIES('sort_scope'='global_sort');
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (12.209 seconds)
0: jdbc:hive2://10.18.16.173:23040/default> load data inpath
'hdfs://hacluster/tmp/data/cbo_1.csv' into table mytest_50_s13
options('FILEHEADER'='name,rownumber,m1');
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (79.901 seconds)
0: jdbc:hive2://10.18.16.173:23040/default> create datamap map10 using 'mv' as
select sum(m1),rownumber from mytest_50_s13 group by rownumber;
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (17.05 seconds)
0: jdbc:hive2://10.18.16.173:23040/default> show datamap on table mytest_50_s13;
+--++---+-+--+
| DataMapName | ClassName | Associated Table | DataMap Properties |
+--++---+-+--+
| map10 | mv | babu.map10_table | |
+--++---+-+--+
1 row selected (0.815 seconds)
Now create one more table without mv
: jdbc:hive2://10.18.16.173:23040/default> create table mytest_50_s14 (name
string,rownumber string, m1 float) stored by 'carbondata'
TBLPROPERTIES('sort_scope'='global_sort');
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (12.209 seconds)
0: jdbc:hive2://10.18.16.173:23040/default> load data inpath
'hdfs://hacluster/tmp/data/cbo_1.csv'' into table mytest_50_s14
options('FILEHEADER'='name,rownumber,m1');
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (79.901 seconds)
0: jdbc:hive2://10.18.16.173:23040/default> select * from mytest_50_s14 limit
10;
Error: java.lang.UnsupportedOperationException: unsupported operation: Modular
plan not supported (e.g. has subquery expression) for
GlobalLimit 10
+- LocalLimit 10
+- Relation[name#1026,rownumber#1027,m1#1028] CarbonDatasourceHadoopRelation
[ Database name :babu, Table name :mytest_50_s14, Schema
:Some(StructType(StructField(name,StringType,true),
StructField(rownumber,StringType,true), StructField(m1,DoubleType,true))) ]
(state=,code=0)
2018-07-13 00:42:51,540 | INFO | [pool-25-thread-32] |
OperationId=b5c2c8b2-1ef4-4894-a709-2a738bd81f76 Result=FAIL |
org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:280)
2018-07-13 00:42:51,540 | ERROR | [pool-25-thread-32] | Error executing query,
currentState RUNNING, |
org.apache.spark.internal.Logging$class.logError(Logging.scala:91)
java.lang.UnsupportedOperationException: unsupported operation: Modular plan
not supported (e.g. has subquery expression) for
GlobalLimit 10
+- LocalLimit 10
+- Relation[name#1026,rownumber#1027,m1#1028] CarbonDatasourceHadoopRelation
[ Database name :babu, Table name :mytest_50_s14, Schema
:Some(StructType(StructField(name,StringType,true),
StructField(rownumber,StringType,true), StructField(m1,DoubleType,true))) ]
at org.apache.carbondata.mv.plans.package$.supports(package.scala:52)
at
org.apache.carbondata.mv.plans.modular.Modularizer.org$apache$carbondata$mv$plans$modular$Modularizer$$modularizeCore(Modularizer.scala:102)
at
org.apache.carbondata.mv.plans.modular.Modularizer.modularize(Modularizer.scala:65)
at
org.apache.carbondata.mv.rewrite.QueryRewrite.modularPlan$lzycompute(QueryRewrite.scala:50)
at
org.apache.carbondata.mv.rewrite.QueryRewrite.modularPlan(QueryRewrite.scala:49)
at
org.apache.carbondata.mv.rewrite.QueryRewrite.withSummaryData$lzycompute(QueryRewrite.scala:53)
at
org.apache.carbondata.mv.rewrite.QueryRewrite.withSummaryData(QueryRewrite.scala:52)
at
org.apache.carbondata.mv.rewrite.QueryRewrite.withMVTable$lzycompute(QueryRewrite.scala:55)
at
org.apache.carbondata.mv.rewrite.QueryRewrite.withMVTable(QueryRewrite.scala:55)
at
org.apache.carbondata.mv.datamap.MVAnalyzerRule.apply(MVAnalyzerRule.scala:68)
at
org.apache.carbondata.mv.datamap.MVAnalyzerRule.apply(MVAnalyzerRule.scala:38)
at
org.apache.spark.sql.hive.CarbonAnalyzer.execute(CarbonAnalyzer.scala:46)
at
org.apache.spark.sql.hive.CarbonAnalyzer.execute(CarbonAnalyzer.scala:27)
at
org.apache.spark.sql.execution.Que
[jira] [Created] (CARBONDATA-2742) [MV] Wrong data displayed after MV creation.
Babulal created CARBONDATA-2742: --- Summary: [MV] Wrong data displayed after MV creation. Key: CARBONDATA-2742 URL: https://issues.apache.org/jira/browse/CARBONDATA-2742 Project: CarbonData Issue Type: Bug Reporter: Babulal 0: jdbc:hive2://10.18.16.173:23040/default> create table mytest_48 (rownumber int,name string, m1 int) stored by 'carbondata'; +-+--+ | Result | +-+--+ +-+--+ No rows selected (1.267 seconds) 0: jdbc:hive2://10.18.16.173:23040/default> load data inpath 'hdfs://hacluster/tmp/babu/testdata_1.csv' into table mytest_48 ; +-+--+ | Result | +-+--+ +-+–+ 0: jdbc:hive2://10.18.16.173:23040/default> show datamap on table mytest_48; +--++---+-+--+ | DataMapName | ClassName | Associated Table | DataMap Properties | +--++---+-+--+ +--++---+-+--+ No rows selected (0.162 seconds) 0: jdbc:hive2://10.18.16.173:23040/default> 0: jdbc:hive2://10.18.16.173:23040/default> select * from mytest_48; ++---+--+--+ | rownumber | name | m1 | ++---+--+--+ | 1 | aaa | 1000 | | 2 | aaa | 65000 | | 3 | aaa | 100 | | 1 | ddd | 1000 | | 2 | ddd | 65000 | | 3 | ddd | 100 | ++---+--+--+ 6 rows selected (1.266 seconds) 0: jdbc:hive2://10.18.16.173:23040/default> create datamap map9 using 'mv' as select sum(m1),name from mytest_48 group by name; +-+--+ | Result | +-+--+ +-+--+ No rows selected (0.82 seconds) 0: jdbc:hive2://10.18.16.173:23040/default> select sum(m1),name from mytest_48 group by name; +--+---+--+ | sum(m1) | name | +--+---+--+ +--+---+--+ No rows selected (2.615 seconds) 0: jdbc:hive2://10.18.16.173:23040/default> explain select sum(m1),name from mytest_48 group by name; +-+--+ | plan | +-+--+ | == CarbonData Profiler == | | == Physical Plan == *HashAggregate(keys=[mytest_48_name#297], functions=[sum(sum_m1#296L)]) +- Exchange hashpartitioning(mytest_48_name#297, 200) +- *HashAggregate(keys=[mytest_48_name#297], functions=[partial_sum(sum_m1#296L)]) +- *BatchedScan CarbonDatasourceHadoopRelation [ Database name :babu, Table name :map9_table, Schema :Some(StructType(StructField(sum_m1,LongType,true), StructField(mytest_48_name,StringType,true))) ] babu.map9_table[sum_m1#296L,mytest_48_name#297] | +
[GitHub] carbondata issue #2504: [CARBONDATA-2740]flat folder handling for implicit c...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2504 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5826/ ---
[jira] [Created] (CARBONDATA-2741) Exception occurs after alter add few columns and selecting in random order
Jatin created CARBONDATA-2741:
-
Summary: Exception occurs after alter add few columns and
selecting in random order
Key: CARBONDATA-2741
URL: https://issues.apache.org/jira/browse/CARBONDATA-2741
Project: CarbonData
Issue Type: Bug
Components: spark-integration
Affects Versions: 1.5.0
Environment: 3 node cluster with spark2.2
Reporter: Jatin
Assignee: Jatin
Fix For: 1.5.0
create table tb1 (imei string,AMSize string,channelsId string,ActiveCountry
string, Activecity string,gamePointId double,deviceInformationId
double,productionDate Timestamp,deliveryDate timestamp,deliverycharge double)
STORED BY 'org.apache.carbondata.format'
TBLPROPERTIES('table_blocksize'='1','COLUMN_META_CACHE'='AMSize');
LOAD DATA INPATH 'hdfs://hacluster/csv/vardhandaterestruct.csv' INTO TABLE tb1
OPTIONS('DELIMITER'=',', 'QUOTECHAR'=
'"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'=
'imei,deviceInformationId,AMSize,channelsId,ActiveCountry,Activecity,gamePointId,productionDate,deliveryDate,deliverycharge');
alter table tb1 add columns(age int, name string);
select * from tb1 where name is NULL or channelsId =4;
Exception occurs :
*Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task
0 in stage 6508.0 failed 4 times, most recent failure: Lost task 0.3 in stage
6508.0 (TID 140476, linux-49, executor 3): java.lang.RuntimeException: internal
error:*
org.apache.carbondata.core.datastore.page.encoding.adaptive.AdaptiveFloatingCodec[src
type: DOUBLE, target type: INT, stats(min: 1.0, max: 100.0, decimal: 1 )]
at
org.apache.carbondata.core.datastore.page.encoding.adaptive.AdaptiveFloatingCodec$3.decodeLong(AdaptiveFloatingCodec.java:185)
at
org.apache.carbondata.core.datastore.page.LazyColumnPage.getLong(LazyColumnPage.java:64)
at
org.apache.carbondata.core.scan.result.vector.MeasureDataVectorProcessor$IntegralMeasureVectorFiller.fillMeasureVector(MeasureDataVectorProcessor.java:73)
at
org.apache.carbondata.core.scan.result.impl.FilterQueryScannedResult.fillColumnarMeasureBatch(FilterQueryScannedResult.java:129)
at
org.apache.carbondata.core.scan.collector.impl.DictionaryBasedVectorResultCollector.fillResultToColumnarBatch(DictionaryBasedVectorResultCollector.java:167)
at
org.apache.carbondata.core.scan.collector.impl.RestructureBasedVectorResultCollector.collectResultInColumnarBatch(RestructureBasedVectorResultCollector.java:127)
at
org.apache.carbondata.core.scan.processor.DataBlockIterator.processNextBatch(DataBlockIterator.java:245)
at
org.apache.carbondata.core.scan.result.iterator.VectorDetailQueryResultIterator.processNextBatch(VectorDetailQueryResultIterator.java:48)
at
org.apache.carbondata.spark.vectorreader.VectorizedCarbonRecordReader.nextBatch(VectorizedCarbonRecordReader.java:290)
at
org.apache.carbondata.spark.vectorreader.VectorizedCarbonRecordReader.nextKeyValue(VectorizedCarbonRecordReader.java:180)
at
org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.hasNext(CarbonScanRDD.scala:497)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.scan_nextBatch$(Unknown
Source)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:381)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:381)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:231)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:225)
at
org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:828)
at
org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:828)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Driver stacktrace: (state=,code=0)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata issue #2441: [CARBONDATA-2625] optimize CarbonReader performance
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2441 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7102/ ---
[GitHub] carbondata issue #2448: [HotFix] Getting carbon table identifier to datamap ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2448 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7101/ ---
[jira] [Resolved] (CARBONDATA-2721) [SDK] [JsonWriter] NPE when schema and data are not of same length or Data is null.
[ https://issues.apache.org/jira/browse/CARBONDATA-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2721. - Resolution: Fixed Fix Version/s: 1.4.1 > [SDK] [JsonWriter] NPE when schema and data are not of same length or Data is > null. > --- > > Key: CARBONDATA-2721 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2721 > Project: CarbonData > Issue Type: Bug >Reporter: Ajantha Bhat >Assignee: Ajantha Bhat >Priority: Major > Fix For: 1.4.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2722) [SDK] [JsonWriter] Json writer is writing only first element of an array and discarding the rest of the elements
[ https://issues.apache.org/jira/browse/CARBONDATA-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2722. - Resolution: Fixed Fix Version/s: 1.4.1 > [SDK] [JsonWriter] Json writer is writing only first element of an array and > discarding the rest of the elements > > > Key: CARBONDATA-2722 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2722 > Project: CarbonData > Issue Type: Bug >Reporter: Ajantha Bhat >Assignee: Ajantha Bhat >Priority: Major > Fix For: 1.4.1 > > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2456: [CARBONDATA-2694][32k] Show longstring table propert...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2456 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5873/ ---
[GitHub] carbondata pull request #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWrite...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2485 ---
[GitHub] carbondata issue #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2485 LGTM ---
[GitHub] carbondata issue #2504: [CARBONDATA-2740]flat folder handling for implicit c...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2504 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5825/ ---
[GitHub] carbondata issue #2501: [CARBONDATA-2738]Block Preaggregate, Dictionary Excl...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2501 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5824/ ---
[GitHub] carbondata issue #2462: [CARBONDATA-2704] Index file size in describe format...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2462 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5872/ ---
[GitHub] carbondata issue #2452: [CARBONDATA-2693][BloomDataMap]Fix bug for alter ren...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2452 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7100/ ---
[GitHub] carbondata issue #2506: [CARBONDATA-2682][32K] fix create table with long_st...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2506 retest this please ---
[GitHub] carbondata issue #2505: [CARBONDATA-2698][CARBONDATA-2700][CARBONDATA-2732][...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2505 retest this please ---
[GitHub] carbondata issue #2453: [CARBONDATA-2528][MV] Fixed order by in mv and aggre...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2453 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7099/ ---
[GitHub] carbondata issue #2463: [CARBONDATA-2708][BloomDataMap] clear index file in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2463 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5871/ ---
[GitHub] carbondata issue #2482: [CARBONDATA-2714] Support merge index files for the ...
Github user dhatchayani commented on the issue: https://github.com/apache/carbondata/pull/2482 retest this please ---
[GitHub] carbondata issue #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2485 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5823/ ---
[GitHub] carbondata issue #2455: [CARBONDATA-2512][32k] Support writing longstring th...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2455 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7098/ ---
[GitHub] carbondata issue #2464: [CARBONDATA-2618][32K] Split to multiple pages if va...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2464 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5870/ ---
[GitHub] carbondata issue #2456: [CARBONDATA-2694][32k] Show longstring table propert...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2456 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7097/ ---
[GitHub] carbondata issue #2465: [WIP] Refactored CarbonFile interface
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2465 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5869/ ---
[GitHub] carbondata issue #2503: [CARBONDATA-2734] Update is not working on the table...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2503 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5822/ ---
[GitHub] carbondata issue #2462: [CARBONDATA-2704] Index file size in describe format...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2462 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7096/ ---
[GitHub] carbondata pull request #2504: [CARBONDATA-2740]flat folder handling for imp...
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2504#discussion_r202329272
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonRelation.scala
---
@@ -173,15 +172,38 @@ case class CarbonRelation(
.getValidAndInvalidSegments.getValidSegments.asScala
var size = 0L
// for each segment calculate the size
- segments.foreach {validSeg =>
-// for older store
-if (null != validSeg.getLoadMetadataDetails.getDataSize &&
-null != validSeg.getLoadMetadataDetails.getIndexSize) {
- size = size +
validSeg.getLoadMetadataDetails.getDataSize.toLong +
- validSeg.getLoadMetadataDetails.getIndexSize.toLong
-} else {
- size = size + FileFactory.getDirectorySize(
-CarbonTablePath.getSegmentPath(tablePath,
validSeg.getSegmentNo))
+ if
(carbonTable.getTableInfo.getFactTable.getTableProperties.asScala
+.get(CarbonCommonConstants.FLAT_FOLDER).isDefined &&
+
carbonTable.getTableInfo.getFactTable.getTableProperties.asScala
+ (CarbonCommonConstants.FLAT_FOLDER).toBoolean) {
+val tableDirectorySize =
FileFactory.getDirectorySize(carbonTable.getTablePath)
+val metaDirectorySize =
FileFactory.getDirectorySize(carbonTable.getMetadataPath)
+val factDirectorySize = FileFactory
+
.getDirectorySize(CarbonTablePath.getFactDir(carbonTable.getTablePath))
+val lockDirSize = FileFactory
+
.getDirectorySize(CarbonTablePath.getLockFilesDirPath(carbonTable.getTablePath))
+val datamaps =
DataMapStoreManager.getInstance().getAllDataMap(carbonTable)
+var datamapsDirectorySize = 0L
+if (datamaps.size() > 0) {
+ datamaps.asScala.foreach { datamap =>
+datamapsDirectorySize = datamapsDirectorySize + FileFactory
+ .getDirectorySize(
+carbonTable.getTablePath +
datamap.getDataMapSchema.getDataMapName)
--- End diff --
Please use a utility method to make to dm path like
org.apache.carbondata.core.util.path.CarbonTablePath#getDataMapStorePath.
---
[GitHub] carbondata issue #2452: [CARBONDATA-2693][BloomDataMap]Fix bug for alter ren...
Github user ndwangsen commented on the issue: https://github.com/apache/carbondata/pull/2452 retest this please ---
[GitHub] carbondata issue #2463: [CARBONDATA-2708][BloomDataMap] clear index file in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2463 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7095/ ---
[GitHub] carbondata issue #2472: [CARBONDATA-2717] fixed table id empty problem while...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2472 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5867/ ---
[GitHub] carbondata issue #2466: [CARBONDATA-2710][Spark Integration] Refactor Carbon...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2466 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5868/ ---
[jira] [Updated] (CARBONDATA-2717) Table id is empty when taking drop lock which causes failure
[ https://issues.apache.org/jira/browse/CARBONDATA-2717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G updated CARBONDATA-2717: - Issue Type: Bug (was: Improvement) > Table id is empty when taking drop lock which causes failure > > > Key: CARBONDATA-2717 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2717 > Project: CarbonData > Issue Type: Bug >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor >Priority: Minor > Fix For: 1.4.1 > > Time Spent: 3h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2488: [CARBONDATA-2724][DataMap]Unsupported create datamap...
Github user ndwangsen commented on the issue: https://github.com/apache/carbondata/pull/2488 retest sdv please ---
[jira] [Resolved] (CARBONDATA-2717) Table id is empty when taking drop lock which causes failure
[ https://issues.apache.org/jira/browse/CARBONDATA-2717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Ramana G resolved CARBONDATA-2717. -- Resolution: Fixed Fix Version/s: 1.4.1 > Table id is empty when taking drop lock which causes failure > > > Key: CARBONDATA-2717 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2717 > Project: CarbonData > Issue Type: Bug >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor >Priority: Minor > Fix For: 1.4.1 > > Time Spent: 3h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2472: [CARBONDATA-2717] fixed table id empty proble...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2472 ---
[GitHub] carbondata issue #2489: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2489 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5821/ ---
[GitHub] carbondata issue #2472: [CARBONDATA-2717] fixed table id empty problem while...
Github user gvramana commented on the issue: https://github.com/apache/carbondata/pull/2472 LGTM ---
[GitHub] carbondata pull request #2487: [CARBONDATA-2735]Fixed Performance issue for ...
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2487#discussion_r202321022
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/page/VarLengthColumnPageBase.java
---
@@ -201,31 +209,32 @@ private static ColumnPage
getComplexLVBytesColumnPage(TableSpec.ColumnSpec colum
throws MemoryException {
// extract length and data, set them to rowOffset and unsafe memory
correspondingly
int rowId = 0;
-List rowOffset = new ArrayList<>();
-List rowLength = new ArrayList<>();
+TableSpec.ColumnSpec spec = TableSpec.ColumnSpec
+.newInstance(columnSpec.getFieldName(), DataTypes.INT,
ColumnType.MEASURE);
+ColumnPage rowOffset = ColumnPage.newPage(spec, DataTypes.INT, 1024);
--- End diff --
This default can be equal NUMBER_OF_ROWS_PER_BLOCKLET_COLUMN_PAGE_DEFAULT
---
[GitHub] carbondata pull request #2487: [CARBONDATA-2735]Fixed Performance issue for ...
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2487#discussion_r202320940
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/page/VarLengthColumnPageBase.java
---
@@ -175,24 +182,25 @@ private static ColumnPage
getLVBytesColumnPage(TableSpec.ColumnSpec columnSpec,
throws MemoryException {
// extract length and data, set them to rowOffset and unsafe memory
correspondingly
int rowId = 0;
-List rowOffset = new ArrayList<>();
-List rowLength = new ArrayList<>();
+TableSpec.ColumnSpec spec = TableSpec.ColumnSpec
+.newInstance(columnSpec.getFieldName(), DataTypes.INT,
ColumnType.MEASURE);
+ColumnPage rowOffset = ColumnPage.newPage(spec, DataTypes.INT, 1024);
--- End diff --
This default can be equal NUMBER_OF_ROWS_PER_BLOCKLET_COLUMN_PAGE_DEFAULT
---
[GitHub] carbondata pull request #2487: [CARBONDATA-2735]Fixed Performance issue for ...
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2487#discussion_r202319613
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/page/SafeVarLengthColumnPage.java
---
@@ -40,6 +40,7 @@
@Override
public void freeMemory() {
byteArrayData = null;
+rowOffset.freeMemory();
--- End diff --
rowOffset.freeMemory also required in SafeDecimalColumnPage.
It will we good practice to call rowOffset.freeMemory in same class it is
allocated, call abstract freememory class from derived to ensure any clean up
from abstract class
---
[GitHub] carbondata issue #2464: [CARBONDATA-2618][32K] Split to multiple pages if va...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2464 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7094/ ---
[GitHub] carbondata issue #2465: [WIP] Refactored CarbonFile interface
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2465 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7093/ ---
[GitHub] carbondata issue #2307: [CARBONDATA-2482] Pass uuid while writing segment fi...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2307 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5820/ ---
[GitHub] carbondata issue #2482: [CARBONDATA-2714] Support merge index files for the ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2482 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5866/ ---
[GitHub] carbondata pull request #2307: [CARBONDATA-2482] Pass uuid while writing seg...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2307#discussion_r202310789 --- Diff: core/src/main/java/org/apache/carbondata/core/writer/CarbonIndexFileMergeWriter.java --- @@ -145,9 +145,10 @@ private String writeMergeIndexFileBasedOnSegmentFile( } } -String uniqueId = String.valueOf(System.currentTimeMillis()); -String newSegmentFileName = -SegmentFileStore.genSegmentFileName(segmentId, String.valueOf(uniqueId)) +String uniqueId = (uuid.endsWith(CarbonTablePath.SEGMENT_EXT)) ? --- End diff -- Don't pass segment file name, just pass only uuid . ---
[GitHub] carbondata issue #2484: [WIP] added hadoop conf to thread local
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2484 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5865/ ---
[jira] [Resolved] (CARBONDATA-2723) Failed to recreate the table which has bloomfilter on it with same table name but different bloom index
[
https://issues.apache.org/jira/browse/CARBONDATA-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravindra Pesala resolved CARBONDATA-2723.
-
Resolution: Fixed
Fix Version/s: 1.4.1
> Failed to recreate the table which has bloomfilter on it with same table name
> but different bloom index
> ---
>
> Key: CARBONDATA-2723
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2723
> Project: CarbonData
> Issue Type: Sub-task
>Reporter: xuchuanyin
>Assignee: xuchuanyin
>Priority: Major
> Fix For: 1.4.1
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Steps to reproduce:
> ```
> create table xu_t11(id int,name string,sal float) stored by 'carbondata'
> tblproperties('sort_columns'='id')
> create datamap xu_dm_t1_new1 on table xu_t11 using 'bloomfilter'
> dmproperties('index_columns'='id')
> insert into xu_t11 select 1,'m',122.33
> // id.bloomindex is generated
> drop table if exists xu_t11
> // recreate table with same datamap but with different index column
> create table xu_t11(id int,name string,sal float) stored by 'carbondata'
> tblproperties('sort_columns'='id')
> create datamap xu_dm_t1_new1 on table xu_t11 using 'bloomfilter'
> dmproperties('index_columns'='sal')
> insert into xu_t11 select 1,'m',122.33
> // Expecte sal.bloomindex to be generated, but actually id.bloomindex is
> generated.
> ```
> This will not be reproduced in testcase, can only be generated in cluster
> (with multiple executors).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata issue #2476: [CARBONDATA-2534][MV] Fix substring expression not w...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2476 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7092/ ---
[GitHub] carbondata pull request #2486: [CARBONDATA-2723][DataMap] Fix bugs in recrea...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2486 ---
[GitHub] carbondata issue #2486: [CARBONDATA-2723][DataMap] Fix bugs in recreate data...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2486 LGTM ---
[GitHub] carbondata issue #2486: [CARBONDATA-2723][DataMap] Fix bugs in recreate data...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2486 The error in SDV is not related to this PRï¼ ``` org.apache.carbondata.cluster.sdv.generated.QueriesIncludeDictionaryTestCase.TABLE_DICTIONARY_INCLUDE_PushUP_044 Results do not match for query: == Results == !== Correct Answer - 1 == == Spark Answer - 1 == ![7.452597748961225E39] [7.452597748961223E39] at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:495) at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1555) at org.scalatest.Assertions$class.fail(Assertions.scala:1328) at org.scalatest.FunSuite.fail(FunSuite.scala:1555) at org.apache.spark.sql.common.util.QueryTest.checkAnswer(QueryTest.scala:75) at org.apache.spark.sql.common.util.QueryTest.writeAndCheckAnswer(QueryTest.scala:113) ``` ---
[GitHub] carbondata pull request #2482: [CARBONDATA-2714] Support merge index files f...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2482#discussion_r202301535 --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/SegmentFileStore.java --- @@ -488,6 +485,49 @@ private void readIndexFiles(SegmentStatus status, boolean ignoreStatus) throws I } } + /** + * Reads all merge index / index files as per the status of the file. + * In case of @ignoreStatus is true it just reads all merge index / index files + * + * @param status + * @param ignoreStatus + * @return + * @throws IOException + */ + private List readIndexOrMergeFiles(SegmentStatus status, boolean ignoreStatus) --- End diff -- why do you need this method, Already merge files are available in `SegmentIndexFileStore.getCarbonMergeFileToIndexFilesMap` ---
[GitHub] carbondata issue #2479: [CARBONDATA-2542][MV] Fix the mv query from table wi...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2479 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7091/ ---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Github user kevinjmh commented on the issue: https://github.com/apache/carbondata/pull/2490 retest this please ---
[GitHub] carbondata issue #2486: [CARBONDATA-2723][DataMap] Fix bugs in recreate data...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2486 retest sdv please ---
[GitHub] carbondata issue #2486: [CARBONDATA-2723][DataMap] Fix bugs in recreate data...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2486 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5819/ ---
[GitHub] carbondata issue #2307: [CARBONDATA-2482] Pass uuid while writing segment fi...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/2307 LGTM...can be merged once build passes ---
[GitHub] carbondata pull request #2482: [CARBONDATA-2714] Support merge index files f...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2482#discussion_r202298843
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/events/AlterTableMergeIndexEventListener.scala
---
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.events
+
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.util.CarbonException
+
+import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
+import org.apache.carbondata.core.datamap.Segment
+import org.apache.carbondata.core.locks.{CarbonLockFactory, LockUsage}
+import org.apache.carbondata.core.statusmanager.SegmentStatusManager
+import org.apache.carbondata.events.{AlterTableMergeIndexEvent, Event,
OperationContext, OperationEventListener}
+import org.apache.carbondata.processing.merger.CarbonDataMergerUtil
+import org.apache.carbondata.spark.util.CommonUtil
+
+class AlterTableMergeIndexEventListener extends OperationEventListener
with Logging {
--- End diff --
This listener also merge with `MergeIndexEventListener` as another `case `
---
[GitHub] carbondata issue #2505: [CARBONDATA-2698][CARBONDATA-2700][CARBONDATA-2732][...
Github user Sssan520 commented on the issue: https://github.com/apache/carbondata/pull/2505 retest this please. ---
[GitHub] carbondata issue #2506: [CARBONDATA-2682][32K] fix create table with long_st...
Github user Sssan520 commented on the issue: https://github.com/apache/carbondata/pull/2506 retest this please. ---
[GitHub] carbondata issue #2506: [CARBONDATA-2682][32K] fix create table with long_st...
Github user Sssan520 commented on the issue: https://github.com/apache/carbondata/pull/2506 There are some problems in the CI env, so I closed #2436 and opened this PR. ---
[GitHub] carbondata issue #2505: [CARBONDATA-2698][CARBONDATA-2700][CARBONDATA-2732][...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2505 Can one of the admins verify this patch? ---
[GitHub] carbondata issue #2506: [CARBONDATA-2682][32K] fix create table with long_st...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2506 Can one of the admins verify this patch? ---
[GitHub] carbondata pull request #2506: [CARBONDATA-2682][32K] fix create table with ...
GitHub user Sssan520 opened a pull request: https://github.com/apache/carbondata/pull/2506 [CARBONDATA-2682][32K] fix create table with long_string_columns properties bugs Changes This PR fixes create table with long_string_columns bugs which are: 1.create table with columns both in long_string_columns and partition or no_inverted_index property should be blocked. 2.create table with duplicate columns in long_string_column property should be blocked. And already add new test cases to take care of testing this. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Sssan520/carbondata dts_longstringcols Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2506.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2506 commit 8f5d6a0ded3080f6c6d7021bf6618e4f823d34cd Author: Sssan520 Date: 2018-07-12T11:58:23Z fix create table with long_string_columns properties bugs ---
[GitHub] carbondata issue #2505: [CARBONDATA-2698][CARBONDATA-2700][CARBONDATA-2732][...
Github user Sssan520 commented on the issue: https://github.com/apache/carbondata/pull/2505 There are some problems in the CI env, so I closed #2491 and opened this PR ---
[GitHub] carbondata pull request #2505: [CARBONDATA-2698][CARBONDATA-2700][CARBONDATA...
GitHub user Sssan520 opened a pull request: https://github.com/apache/carbondata/pull/2505 [CARBONDATA-2698][CARBONDATA-2700][CARBONDATA-2732][BloomDataMap] block some oprerations of bloomfilter datamap 1.Block create bloomfilter datamap index on column which its datatype is complex type; 2.Block change datatype for bloomfilter index datamap; 3.Block dropping index columns for bloomfilter index datamap Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Sssan520/carbondata bloomfilterdatamap Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2505.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2505 commit 8e40f663070022b4b5d919fcc8dc5503fd37fa21 Author: Sssan520 Date: 2018-07-11T12:17:09Z Block some operations of bloomfilter datamap ---
[GitHub] carbondata pull request #2436: [CARBONDATA-2682][32K] fix create table with ...
Github user Sssan520 closed the pull request at: https://github.com/apache/carbondata/pull/2436 ---
[GitHub] carbondata pull request #2504: [CARBONDATA-2740]flat folder handling for imp...
GitHub user akashrn5 opened a pull request: https://github.com/apache/carbondata/pull/2504 [CARBONDATA-2740]flat folder handling for implicit column and other bug fix ### Problem 1) When flat folder is enabled for table, pruning is not happening for implicit column. 2) error message is wrong when complex column which does not have any string datatype column is given in local dictionary and exclude column. 3) when data load is failed, corresponding segment file is not getting deleted. ### Solution: 1) when flat folder is enabled, based on the segment number present in the carbondata file name, pruning is taken care 2) error message is corrected 3) when data load is failed, segment file will be deleted You can merge this pull request into a Git repository by running: $ git pull https://github.com/akashrn5/incubator-carbondata implicit Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2504.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2504 commit 037769a4ccedd9a5a76cf1c9992a36b99cb4926d Author: akashrn5 Date: 2018-07-13T09:04:42Z flat folder handling for implicit column and other bug fix ---
[GitHub] carbondata issue #2480: [CARBONDATA-2550][CARBONDATA-2576][MV] Fix limit and...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2480 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7090/ ---
[GitHub] carbondata issue #2453: [CARBONDATA-2528][MV] Fixed order by in mv and aggre...
Github user brijoobopanna commented on the issue: https://github.com/apache/carbondata/pull/2453 retest this please ---
[GitHub] carbondata issue #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2485 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5864/ ---
[jira] [Created] (CARBONDATA-2740) flat folder structure is not handled for implicit column and segment file is not getting deleted after load is failed
Akash R Nilugal created CARBONDATA-2740: --- Summary: flat folder structure is not handled for implicit column and segment file is not getting deleted after load is failed Key: CARBONDATA-2740 URL: https://issues.apache.org/jira/browse/CARBONDATA-2740 Project: CarbonData Issue Type: Bug Reporter: Akash R Nilugal Assignee: Akash R Nilugal flat folder structure is not handled for implicit column and segment file is not getting deleted after load is failed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2491: [CARBONDATA-2698][CARBONDATA-2700][CARBONDATA...
Github user Sssan520 closed the pull request at: https://github.com/apache/carbondata/pull/2491 ---
[GitHub] carbondata pull request #2501: [CARBONDATA-2738]Block Preaggregate, Dictiona...
Github user akashrn5 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2501#discussion_r202285088
--- Diff:
integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala
---
@@ -720,18 +720,18 @@ abstract class CarbonDDLSqlParser extends
AbstractCarbonSparkSQLParser {
tableProperties.get(CarbonCommonConstants.DICTIONARY_EXCLUDE).get.split(',').map(_.trim)
dictExcludeCols
.foreach { dictExcludeCol =>
+ if (dictExcludeCol.contains(".") ||
dictExcludeCol.contains("[")) {
--- End diff --
do not check, for contains ".", get from field whether it is child column,
then you can check whether that is given in include or exclude and throw
exception
---
[GitHub] carbondata pull request #2501: [CARBONDATA-2738]Block Preaggregate, Dictiona...
Github user akashrn5 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2501#discussion_r202285188
--- Diff:
integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala
---
@@ -748,6 +748,10 @@ abstract class CarbonDDLSqlParser extends
AbstractCarbonSparkSQLParser {
dictIncludeCols =
tableProperties(CarbonCommonConstants.DICTIONARY_INCLUDE).split(",").map(_.trim)
dictIncludeCols.foreach { distIncludeCol =>
+if (distIncludeCol.contains(".") || distIncludeCol.contains("[")) {
--- End diff --
same as above
---
[GitHub] carbondata pull request #2501: [CARBONDATA-2738]Block Preaggregate, Dictiona...
Github user akashrn5 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2501#discussion_r202286122
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForUpdateCommand.scala
---
@@ -59,6 +60,13 @@ private[sql] case class CarbonProjectForUpdateCommand(
return Seq.empty
}
val carbonTable = CarbonEnv.getCarbonTable(databaseNameOp,
tableName)(sparkSession)
+columns.foreach { col =>
+ var dataType = carbonTable.getColumnByName(tableName,
col).getColumnSchema.getDataType
+ if(dataType.getName.equals("STRUCT") ||
dataType.getName.equals("ARRAY")) {
--- End diff --
do not check for hard coded value, get the datatype and dicrectly check
`isComplexType` and throw error
---
[GitHub] carbondata issue #2486: [CARBONDATA-2723][DataMap] Fix bugs in recreate data...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2486 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5863/ ---
[GitHub] carbondata issue #2489: [CARBONDATA-2606][Complex DataType Enhancements]Fix ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2489 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5818/ ---
[GitHub] carbondata issue #2482: [CARBONDATA-2714] Support merge index files for the ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2482 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7089/ ---
[GitHub] carbondata pull request #2491: [CARBONDATA-2698][CARBONDATA-2700][CARBONDATA...
Github user Sssan520 closed the pull request at: https://github.com/apache/carbondata/pull/2491 ---
[GitHub] carbondata pull request #2491: [CARBONDATA-2698][CARBONDATA-2700][CARBONDATA...
GitHub user Sssan520 reopened a pull request: https://github.com/apache/carbondata/pull/2491 [CARBONDATA-2698][CARBONDATA-2700][CARBONDATA-2732][BloomDataMap] block some oprerations of bloomfilter datamap 1.Block create bloomfilter datamap index on column which its datatype is complex type; 2.Block change datatype for bloomfilter index datamap; 3.Block dropping index columns for bloomfilter index datamap Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? add a parameter "targets" for method "canAllow" of "CarbonTable" class - [ ] Any backward compatibility impacted? No - [ ] Document update required? No - [ ] Testing done NA Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Sssan520/carbondata bloomfilter Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2491.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2491 commit 815c497380fa52b98aa270d35ce67f2cb9c67191 Author: lianganping 00251374 Date: 2018-07-11T12:17:09Z 1.Block create bloomfilter datamap index on column which its datatype is complex;2.Block create bloomfilter datamap index on local_dictionary column;3.Block change datatype for bloomfilter index datamap;4.Block dropping index columns for bloomfilter index datamap ---
[GitHub] carbondata pull request #2491: [CARBONDATA-2698][CARBONDATA-2700][CARBONDATA...
Github user Sssan520 closed the pull request at: https://github.com/apache/carbondata/pull/2491 ---
[GitHub] carbondata pull request #2491: [CARBONDATA-2698][CARBONDATA-2700][CARBONDATA...
GitHub user Sssan520 reopened a pull request: https://github.com/apache/carbondata/pull/2491 [CARBONDATA-2698][CARBONDATA-2700][CARBONDATA-2730][CARBONDATA-2732][BloomDataMap] block some oprerations of bloomfilter datamap 1.Block create bloomfilter datamap index on column which its datatype is complex type; 2.Block change datatype for bloomfilter index datamap; 3.Block dropping index columns for bloomfilter index datamap Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? add a parameter "targets" for method "canAllow" of "CarbonTable" class - [ ] Any backward compatibility impacted? No - [ ] Document update required? No - [ ] Testing done NA Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Sssan520/carbondata bloomfilter Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2491.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2491 commit 815c497380fa52b98aa270d35ce67f2cb9c67191 Author: lianganping 00251374 Date: 2018-07-11T12:17:09Z 1.Block create bloomfilter datamap index on column which its datatype is complex;2.Block create bloomfilter datamap index on local_dictionary column;3.Block change datatype for bloomfilter index datamap;4.Block dropping index columns for bloomfilter index datamap ---
[jira] [Closed] (CARBONDATA-2730) Block create bloomfilter datamap index on local_dictionary column
[ https://issues.apache.org/jira/browse/CARBONDATA-2730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lianganping closed CARBONDATA-2730. --- Resolution: Not A Problem > Block create bloomfilter datamap index on local_dictionary column > - > > Key: CARBONDATA-2730 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2730 > Project: CarbonData > Issue Type: Sub-task >Reporter: lianganping >Assignee: lianganping >Priority: Major > > Block create bloomfilter datamap index on local_dictionary column; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2491: [CARBONDATA-2698][CARBONDATA-2700][CARBONDATA-2730][...
Github user Sssan520 commented on the issue: https://github.com/apache/carbondata/pull/2491 retest this please ---
[GitHub] carbondata issue #2491: [CARBONDATA-2698][CARBONDATA-2700][CARBONDATA-2730][...
Github user Sssan520 commented on the issue: https://github.com/apache/carbondata/pull/2491 all code review comments has been handled. ---
[GitHub] carbondata issue #2436: [CARBONDATA-2682][32K] fix create table with long_st...
Github user Sssan520 commented on the issue: https://github.com/apache/carbondata/pull/2436 retest this please ---
[GitHub] carbondata pull request #2499: [CARBONDATA-2648] Fixed NPE issue with legacy...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2499 ---
[GitHub] carbondata issue #2499: [CARBONDATA-2648] Fixed NPE issue with legacy store ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2499 LGTM ---
[GitHub] carbondata pull request #2499: [CARBONDATA-2648] Fixed NPE issue with legacy...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2499#discussion_r202278145
--- Diff:
core/src/main/java/org/apache/carbondata/core/indexstore/TableBlockIndexUniqueIdentifierWrapper.java
---
@@ -35,18 +35,34 @@
// holds the reference to CarbonTable
private CarbonTable carbonTable;
+ /**
+ * flag to specify whether to load table block metadata in unsafe or
safe. Default value is true
+ */
+ private boolean addTableBlockToUnsafe = true;
public TableBlockIndexUniqueIdentifierWrapper(
TableBlockIndexUniqueIdentifier tableBlockIndexUniqueIdentifier,
CarbonTable carbonTable) {
this.tableBlockIndexUniqueIdentifier = tableBlockIndexUniqueIdentifier;
this.carbonTable = carbonTable;
}
+ public TableBlockIndexUniqueIdentifierWrapper(
--- End diff --
As discussed I have added a Note
---
[GitHub] carbondata pull request #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWrite...
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2485#discussion_r202277326
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/loading/parser/impl/JsonRowParser.java
---
@@ -56,6 +56,9 @@ public JsonRowParser(DataField[] dataFields) {
Map jsonNodeMap =
objectMapper.readValue(jsonString, new TypeReference>() {
});
+ if (jsonNodeMap == null) {
--- End diff --
yeah, not required. removed it.
---
[GitHub] carbondata issue #2485: [CARBONDATA-2722] [CARBONDATA-2721] JsonWriter issue...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2485 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7087/ ---
[GitHub] carbondata issue #2487: [CARBONDATA-2735]Fixed Performance issue for complex...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2487 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5862/ ---
[GitHub] carbondata pull request #2503: [CARBONDATA-2734] Update is not working on th...
GitHub user ravipesala opened a pull request: https://github.com/apache/carbondata/pull/2503 [CARBONDATA-2734] Update is not working on the table which has segmentfile present It reverts to the PR https://github.com/apache/carbondata/pull/2385 It fixes the IUD on the flat folder. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ravipesala/incubator-carbondata flat-folder-update-issue Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2503.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2503 commit caf2304736741748e6ba896d5ba29333ee95defc Author: ravipesala Date: 2018-07-13T07:45:15Z Update is not working on the table which has segmentfile present ---
