[GitHub] carbondata issue #2623: [HOTFIX] add sk ak to file factory on creation of ca...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2623 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6574/ ---
[GitHub] carbondata issue #2623: [HOTFIX] add sk ak to file factory on creation of ca...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2623 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7849/ ---
[GitHub] carbondata issue #2623: [HOTFIX] add sk ak to file factory on creation of ca...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2623 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6220/ ---
[GitHub] carbondata pull request #2623: [HOTFIX] add sk ak to file factory on creatio...
GitHub user kunal642 opened a pull request: https://github.com/apache/carbondata/pull/2623 [HOTFIX] add sk ak to file factory on creation of carbon env add SK AK to file factory on creation of carbon env Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kunal642/carbondata sk_ak_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2623.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2623 commit 3a69dc5818ee9d31f195a5737bffd1b80298ed3f Author: kunal642 Date: 2018-08-08T16:20:44Z add sk ak to file factory on creation of carbon env ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2619 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6573/ ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2619 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7848/ ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2619 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6219/ ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user Sssan520 commented on the issue: https://github.com/apache/carbondata/pull/2619 retest this please ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2619 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6572/ ---
[GitHub] carbondata issue #2620: [CARBONDATA-2839] Add custom compaction example
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2620 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6571/ ---
[GitHub] carbondata issue #2620: [CARBONDATA-2839] Add custom compaction example
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2620 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7846/ ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2619 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6218/ ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user Sssan520 commented on the issue: https://github.com/apache/carbondata/pull/2619 retest this please ---
[GitHub] carbondata pull request #2620: [CARBONDATA-2839] Add custom compaction examp...
Github user chenliang613 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2620#discussion_r208780697
--- Diff:
examples/spark2/src/main/scala/org/apache/carbondata/examples/CustomCompactionExample.scala
---
@@ -0,0 +1,69 @@
+package org.apache.carbondata.examples
+
+import java.io.File
+
+import org.apache.spark.sql.SparkSession
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.examples.util.ExampleUtils
+
+
+object CustomCompactionExample {
+
+ def main(args: Array[String]): Unit = {
+val spark = ExampleUtils.createCarbonSession("CustomCompactionExample")
+exampleBody(spark)
+spark.close()
+ }
+
+ def exampleBody(spark : SparkSession): Unit = {
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT, "/MM/dd")
+
+spark.sql("DROP TABLE IF EXISTS custom_compaction_table")
+
+spark.sql(
+ s"""
+ | CREATE TABLE IF NOT EXISTS custom_compaction_table(
+ | ID Int,
+ | date Date,
+ | country String,
+ | name String,
+ | phonetype String,
+ | serialname String,
+ | salary Int,
+ | floatField float
+ | ) STORED BY 'carbondata'
+ """.stripMargin)
+
+val rootPath = new File(this.getClass.getResource("/").getPath
+ + "../../../..").getCanonicalPath
+val path =
s"$rootPath/examples/spark2/src/main/resources/dataSample.csv"
+
+// load 4 segments
+// scalastyle:off
+(1 to 4).foreach(_ => spark.sql(
+ s"""
+ | LOAD DATA LOCAL INPATH '$path'
+ | INTO TABLE custom_compaction_table
+ | OPTIONS('HEADER'='true')
+ """.stripMargin))
+// scalastyle:on
+
+// show all segments: 0,1,2,3
+spark.sql("SHOW SEGMENTS FOR TABLE custom_compaction_table").show()
+
+// do custom compaction, segments specified will be merged
+spark.sql("ALTER TABLE custom_compaction_table COMPACT 'CUSTOM' WHERE
SEGMENT.ID IN (1,2)")
+spark.sql("SHOW SEGMENTS FOR TABLE custom_compaction_table").show()
+
+CarbonProperties.getInstance().addProperty(
+ CarbonCommonConstants.CARBON_DATE_FORMAT,
+ CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT)
+
--- End diff --
After custom compaction, please query table data once to check the data if
it is correct?
---
[GitHub] carbondata pull request #2620: [CARBONDATA-2839] Add custom compaction examp...
Github user chenliang613 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2620#discussion_r208780252 --- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/CustomCompactionExample.scala --- @@ -0,0 +1,69 @@ +package org.apache.carbondata.examples + +import java.io.File + +import org.apache.spark.sql.SparkSession + +import org.apache.carbondata.core.constants.CarbonCommonConstants +import org.apache.carbondata.core.util.CarbonProperties +import org.apache.carbondata.examples.util.ExampleUtils + + --- End diff -- please add the description for explaining the example. ---
[GitHub] carbondata pull request #2620: [CARBONDATA-2839] Add custom compaction examp...
Github user chenliang613 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2620#discussion_r208780123 --- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/CustomCompactionExample.scala --- @@ -0,0 +1,69 @@ +package org.apache.carbondata.examples --- End diff -- please add the apache license header ---
[GitHub] carbondata issue #2620: [CARBONDATA-2839] Add custom compaction example
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/2620 retest this please ---
[jira] [Updated] (CARBONDATA-2332) Potentially overflowing expression in UnsafeFixLengthColumnPage
[
https://issues.apache.org/jira/browse/CARBONDATA-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated CARBONDATA-2332:
---
Description:
Here is one example from getFloatPage :
{code}
for (int i = 0; i < data.length; i++) {
long offset = i << floatBits;
{code}
The shift expression with type "int" (32 bits, signed) is evaluated using
32-bit arithmetic.
But the variable offset is of type long.
There are a few other shift expressions of this nature.
was:
Here is one example from getFloatPage :
{code}
for (int i = 0; i < data.length; i++) {
long offset = i << floatBits;
{code}
The shift expression with type "int" (32 bits, signed) is evaluated using
32-bit arithmetic.
But the variable offset is of type long.
There are a few other shift expressions of this nature.
> Potentially overflowing expression in UnsafeFixLengthColumnPage
> ---
>
> Key: CARBONDATA-2332
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2332
> Project: CarbonData
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> Here is one example from getFloatPage :
> {code}
> for (int i = 0; i < data.length; i++) {
> long offset = i << floatBits;
> {code}
> The shift expression with type "int" (32 bits, signed) is evaluated using
> 32-bit arithmetic.
> But the variable offset is of type long.
> There are a few other shift expressions of this nature.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata issue #2622: [HOTFIX] Fixed Query performance issue
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2622 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7845/ ---
[GitHub] carbondata issue #2622: [HOTFIX] Fixed Query performance issue
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2622 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6217/ ---
[GitHub] carbondata pull request #2622: [HOTFIX] Fixed Query performance issue
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2622#discussion_r208686198
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/CarbonScanRDD.scala
---
@@ -463,9 +463,12 @@ class CarbonScanRDD[T: ClassTag](
model.setStatisticsRecorder(recorder)
// TODO: rewrite this logic to call free memory in FailureListener
on failures. On success,
- // no memory leak should be there, resources should be freed on
success completion.
- val listeners =
CarbonReflectionUtils.getField("onCompleteCallbacks", context)
+ // TODO: no memory leak should be there, resources should be freed
on success completion.
+ val onCompleteCallbacksField =
context.getClass.getDeclaredField("onCompleteCallbacks")
+ onCompleteCallbacksField.setAccessible(true)
--- End diff --
Can you move this code to getField method of CarbonReflectionUtil, other
callers gets advantage
---
[GitHub] carbondata issue #2622: [HOTFIX] Fixed Query performance issue
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2622 LGTM ---
[GitHub] carbondata issue #2622: [HOTFIX] Fixed Query performance issue
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2622 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6570/ ---
[GitHub] carbondata issue #2622: [HOTFIX] Fixed Query performance issue
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2622 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6216/ ---
[GitHub] carbondata issue #2622: [HOTFIX] Fixed Query performance issue
Github user kumarvishal09 commented on the issue: https://github.com/apache/carbondata/pull/2622 @ravipesala Please review ---
[GitHub] carbondata pull request #2622: [HOTFIX] Fixed Query performance issue
GitHub user kumarvishal09 opened a pull request: https://github.com/apache/carbondata/pull/2622 [HOTFIX] Fixed Query performance issue Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kumarvishal09/incubator-carbondata queryperf Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2622.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2622 commit 551fe868f941f1dffa50e6fe296cf2c491910b1a Author: kumarvishal09 Date: 2018-08-08T17:55:41Z Fixed Query performance issue ---
[jira] [Created] (CARBONDATA-2842) Global sort local/preaggregate with Dictionary not working for S3
Babulal created CARBONDATA-2842:
---
Summary: Global sort local/preaggregate with Dictionary not
working for S3
Key: CARBONDATA-2842
URL: https://issues.apache.org/jira/browse/CARBONDATA-2842
Project: CarbonData
Issue Type: Bug
Reporter: Babulal
Configure AK/SK and Endpoint for S3
Now run Global sort
reate table c2 (rownumber string,name string,m1 int) stored by 'carbondata'
tblproperties('DICTIONARY_INCLUDE'='name','sort_scope'='global_sort');
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (18.047 seconds)
0: jdbc:hive2://ha-cluster/default> create datamap ag1 on table c2 using
'preaggregate' as select sum(m1),name from c2 group by name;
Error: java.lang.Exception: DataLoad failure: Dictionary file does not exist:
s3a://carbon/req/carbon_store_mrs_118_new/x1/c2/Metadata/a4d2bec1-0aa0-4908-82fa-c31863af9280.dict
(state=,code=0)
0: jdbc:hive2://ha-cluster/default>
0: jdbc:hive2://ha-cluster/default>
0: jdbc:hive2://ha-cluster/default>
same Error when runs global dictionary loading also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CARBONDATA-2843) S3 Load is not working when load initiated on starup of JDBCServer.
Babulal created CARBONDATA-2843:
---
Summary: S3 Load is not working when load initiated on starup of
JDBCServer.
Key: CARBONDATA-2843
URL: https://issues.apache.org/jira/browse/CARBONDATA-2843
Project: CarbonData
Issue Type: Bug
Reporter: Babulal
Configure AK/SK and endpoint in spark-default.conf
create table in S3 and then restart thrift server
once JDBCserver is up start data loading
0: jdbc:hive2://ha-cluster/default> load data inpath 's3a://carbon/req/data/1'
into table c2 options('fileheader'='rownumber,name,m1');
Error: java.io.InterruptedIOException: doesBucketExist on carbon:
com.amazonaws.AmazonClientException: No AWS Credentials provided by
BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider
SharedInstanceProfileCredentialsProvider : com.amazonaws.AmazonClientException:
Unable to load credentials from Amazon EC2 metadata service (state=,code=0)
0: jdbc:hive2://ha-cluster/default>
0: jdbc:hive2://ha-cluster/default> !q
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CARBONDATA-2841) S3 Not working for Update/Delete Command
Babulal created CARBONDATA-2841: --- Summary: S3 Not working for Update/Delete Command Key: CARBONDATA-2841 URL: https://issues.apache.org/jira/browse/CARBONDATA-2841 Project: CarbonData Issue Type: Bug Reporter: Babulal configure in spark defauult.conf spark.hadoop.fs.s3a.endpoint spark.hadoop.fs.s3a.secret.key spark.hadoop.fs.s3a.access.key Run Update /Delete Command Caused by: java.io.InterruptedIOException: doesBucketExist on carbon: com.amazonaws.AmazonClientException: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider SharedInstanceProfileCredentialsProvider : com.amazonaws.AmazonClientException: Unable to load credentials from Amazon EC2 metadata service at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:125) at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:288) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:236) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2838) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2875) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2857) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356) at org.apache.carbondata.core.datastore.filesystem.AbstractDFSCarbonFile.getDataOutputStream(AbstractDFSCarbonFile.java:383) at org.apache.carbondata.core.datastore.impl.FileFactory.getDataOutputStream(FileFactory.java:150) at org.apache.carbondata.core.util.CarbonUtil.copyLocalFileToCarbonStore(CarbonUtil.java:2821) at org.apache.carbondata.core.util.CarbonUtil.copyCarbonDataFileToCarbonStorePath(CarbonUtil.java:2790) ... 13 more Caused by: com.amazonaws.AmazonClientException: No AWS Credentials provided by BasicAWSCredentialsProvider Environ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2616 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6568/ ---
[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2616 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7843/ ---
[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2616 retest this please ---
[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2616 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6564/ ---
[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2616 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7840/ ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2619 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6567/ ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2619 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7842/ ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2619 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6215/ ---
[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...
Github user kumarvishal09 commented on the issue: https://github.com/apache/carbondata/pull/2616 LGTM ---
[GitHub] carbondata pull request #2616: [HOTFIX]Fixed int overflow and comparison gon...
Github user kumarvishal09 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2616#discussion_r208569907 --- Diff: core/src/main/java/org/apache/carbondata/core/util/CarbonMetadataUtil.java --- @@ -368,7 +368,13 @@ private static int compareMeasureData(byte[] first, byte[] second, DataType data secondBuffer.put(second); firstBuffer.flip(); secondBuffer.flip(); - return (int) (firstBuffer.getDouble() - secondBuffer.getDouble()); + long compare = firstBuffer.getLong() - secondBuffer.getLong(); --- End diff -- :) ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user Sssan520 commented on the issue: https://github.com/apache/carbondata/pull/2619 retest this please ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2619 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6565/ ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2619 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6214/ ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2619 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7841/ ---
[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2616 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6213/ ---
[jira] [Resolved] (CARBONDATA-2817) Thread Leak in Update and in No sort flow
[ https://issues.apache.org/jira/browse/CARBONDATA-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kumar vishal resolved CARBONDATA-2817. -- Resolution: Fixed Assignee: Babulal > Thread Leak in Update and in No sort flow > - > > Key: CARBONDATA-2817 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2817 > Project: CarbonData > Issue Type: Bug >Reporter: Babulal >Assignee: Babulal >Priority: Major > Time Spent: 4.5h > Remaining Estimate: 0h > > After update finished , loading threads (input process,convet,sort..etc) are > still alive. > > "Thread-36" #172 daemon prio=5 os_prio=0 tid=0x7fd021eba000 nid=0x17136 > waiting on condition [0x7fd01000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > org.apache.carbondata.processing.loading.AbstractDataLoadProcessorStep$1.run(AbstractDataLoadProcessorStep.java:81) > "Thread-35" #171 daemon prio=5 os_prio=0 tid=0x01a7f000 nid=0x17135 > waiting on condition [0x7fd01101] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > org.apache.carbondata.processing.loading.AbstractDataLoadProcessorStep$1.run(AbstractDataLoadProcessorStep.java:81) > "Thread-34" #170 daemon prio=5 os_prio=0 tid=0x01e4 nid=0x17134 > waiting on condition [0x7fd019aa9000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > org.apache.carbondata.processing.loading.AbstractDataLoadProcessorStep$1.run(AbstractDataLoadProcessorStep.java:81) > > "NoSortDataWriterPool:tbl_data_event_41_carbon_nosort" #96 prio=5 > os_prio=0 tid=0x00f2e800 nid=0x129a2 waiting on condition > [0x7fd0197a6000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x0006a35989a0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2606: [CARBONDATA-2817]Thread Leak in Update and in...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2606 ---
[GitHub] carbondata pull request #2616: [HOTFIX]Fixed int overflow and comparison gon...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2616#discussion_r208556581 --- Diff: core/src/main/java/org/apache/carbondata/core/util/CarbonMetadataUtil.java --- @@ -368,7 +368,13 @@ private static int compareMeasureData(byte[] first, byte[] second, DataType data secondBuffer.put(second); firstBuffer.flip(); secondBuffer.flip(); - return (int) (firstBuffer.getDouble() - secondBuffer.getDouble()); + long compare = firstBuffer.getLong() - secondBuffer.getLong(); --- End diff -- Yes , its a copy paste mistake :) ---
[GitHub] carbondata pull request #2616: [HOTFIX]Fixed int overflow and comparison gon...
Github user kumarvishal09 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2616#discussion_r208556245 --- Diff: core/src/main/java/org/apache/carbondata/core/util/CarbonMetadataUtil.java --- @@ -368,7 +368,13 @@ private static int compareMeasureData(byte[] first, byte[] second, DataType data secondBuffer.put(second); firstBuffer.flip(); secondBuffer.flip(); - return (int) (firstBuffer.getDouble() - secondBuffer.getDouble()); + long compare = firstBuffer.getLong() - secondBuffer.getLong(); --- End diff -- For double type we should call getDouble ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2619 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6563/ ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2619 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7839/ ---
[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...
Github user kumarvishal09 commented on the issue: https://github.com/apache/carbondata/pull/2606 LGTM ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user Sssan520 commented on the issue: https://github.com/apache/carbondata/pull/2619 retest this please ---
[GitHub] carbondata issue #2621: [CARBONDATA-2840] Added SDV testcases for Complex Da...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2621 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6562/ ---
[GitHub] carbondata issue #2621: [CARBONDATA-2840] Added SDV testcases for Complex Da...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2621 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7838/ ---
[GitHub] carbondata issue #2620: [CARBONDATA-2839] Add custom compaction example
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2620 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6561/ ---
[GitHub] carbondata issue #2620: [CARBONDATA-2839] Add custom compaction example
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2620 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7836/ ---
[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2606 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6560/ ---
[GitHub] carbondata issue #2621: [CARBONDATA-2840] Added SDV testcases for Complex Da...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2621 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6212/ ---
[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2606 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7835/ ---
[GitHub] carbondata issue #2621: [CARBONDATA-2840] Added SDV testcases for Complex Da...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2621 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6211/ ---
[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2617 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6559/ ---
[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2617 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7834/ ---
[GitHub] carbondata pull request #2621: [CARBONDATA-2840] Added SDV testcases for Com...
Github user praveenmeenakshi56 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2621#discussion_r208528095
--- Diff:
integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/ComplexDataTypeTestCase.scala
---
@@ -0,0 +1,407 @@
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.cluster.sdv.generated
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream,
DataInputStream, File, InputStream}
+import java.sql.Timestamp
+
+import scala.collection.mutable
+
+import org.apache.avro.file.DataFileWriter
+import org.apache.avro.generic.{GenericDatumReader, GenericDatumWriter,
GenericRecord}
+import org.apache.avro.io.{DecoderFactory, Encoder}
+import org.apache.commons.io.FileUtils
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.common.util.QueryTest
+import org.apache.spark.sql.test.TestQueryExecutor
+import org.scalatest.BeforeAndAfterAll
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.sdk.file.CarbonWriter
+
+/**
+ * Test Class for ComplexDataTypeTestCase to verify all scenerios
+ */
+
+class ComplexDataTypeTestCase extends QueryTest with BeforeAndAfterAll {
+
+ val filePath = TestQueryExecutor.integrationPath +
"/spark-common-test/src/test/resources"
+ val writerPath =
+s"${ resourcesPath }" + "/SparkCarbonFileFormat/WriterOutputComplex/"
+
+ override def beforeAll(): Unit = {
+FileUtils.deleteDirectory(new File(writerPath))
+sql("DROP TABLE IF EXISTS complexcarbontable")
+sql("DROP TABLE IF EXISTS test")
+sql("DROP TABLE IF EXISTS sdkOutputTable")
+ }
+
+ override def afterAll(): Unit = {
+FileUtils.deleteDirectory(new File(writerPath))
+sql("DROP TABLE IF EXISTS complexcarbontable")
+sql("DROP TABLE IF EXISTS test")
+sql("DROP TABLE IF EXISTS sdkOutputTable")
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT)
+ .addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT,
+CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT)
+ }
+
+ // check create table with complex data type
+ test("test Complex_DataType-001") {
+sql("DROP TABLE IF EXISTS test")
+sql(
+ "create table test(person
struct>) stored by 'carbondata'")
+sql("DROP TABLE IF EXISTS test")
+sql(
+ "create table test(p1 array,p2 array,p3
array,p4 array,p5 " +
+ "array,p6 array) stored by 'carbondata'")
+ }
--- End diff --
remove this test case. It is not making use of the cluster. If it is in UT,
it is enough
---
[GitHub] carbondata pull request #2621: [CARBONDATA-2840] Added SDV testcases for Com...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2621 [CARBONDATA-2840] Added SDV testcases for Complex DataType Support **Added SDV testcases for Complex DataType Support** - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata complexsdv Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2621.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2621 commit f9bd3e83720b6657654d38d96f1b3d07f9172192 Author: Indhumathi27 Date: 2018-08-08T10:03:36Z [CARBONDATA-2840] Added SDV testcases for Complex DataType Support ---
[jira] [Created] (CARBONDATA-2840) Add SDV testcases for Complex DataType Support
Indhumathi Muthumurugesh created CARBONDATA-2840: Summary: Add SDV testcases for Complex DataType Support Key: CARBONDATA-2840 URL: https://issues.apache.org/jira/browse/CARBONDATA-2840 Project: CarbonData Issue Type: Test Reporter: Indhumathi Muthumurugesh Assignee: Indhumathi Muthumurugesh -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2620: [CARBONDATA-2839] Add custom compaction example
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2620 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6210/ ---
[GitHub] carbondata pull request #2620: [CARBONDATA-2839] Add custom compaction examp...
Github user zzcclp commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2620#discussion_r208523007
--- Diff:
examples/spark2/src/main/scala/org/apache/carbondata/examples/CustomCompactionExample.scala
---
@@ -0,0 +1,69 @@
+package org.apache.carbondata.examples
+
+import java.io.File
+
+import org.apache.spark.sql.SparkSession
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.examples.util.ExampleUtils
+
+
+object CustomCompactionExample {
+
+ def main(args: Array[String]): Unit = {
+val spark = ExampleUtils.createCarbonSession("CustomCompactionExample")
+exampleBody(spark)
+spark.close()
+ }
+
+ def exampleBody(spark : SparkSession): Unit = {
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT, "/MM/dd")
+
+spark.sql("DROP TABLE IF EXISTS custom_compaction_table")
+
+spark.sql(
+ s"""
+ | CREATE TABLE IF NOT EXISTS custom_compaction_table(
+ | ID Int,
+ | date Date,
+ | country String,
+ | name String,
+ | phonetype String,
+ | serialname String,
+ | salary Int,
+ | floatField float
+ | ) STORED BY 'carbondata'
+ """.stripMargin)
+
+val rootPath = new File(this.getClass.getResource("/").getPath
+ + "../../../..").getCanonicalPath
+val path =
s"$rootPath/examples/spark2/src/main/resources/dataSample.csv"
+
+// load 4 segments
+// scalastyle:off
+(1 to 4).foreach(_ => spark.sql(
+ s"""
+ | LOAD DATA LOCAL INPATH '$path'
+ | INTO TABLE custom_compaction_table
+ | OPTIONS('HEADER'='true')
+ """.stripMargin))
+// scalastyle:on
+
+// show all segments: 0,1,2,3
+spark.sql("SHOW SEGMENTS FOR TABLE custom_compaction_table").show()
+
+// do custom compaction, segments specified will be merged
+spark.sql("ALTER TABLE custom_compaction_table COMPACT 'CUSTOM' WHERE
SEGMENT.ID IN (1,2)")
+spark.sql("SHOW SEGMENTS FOR TABLE custom_compaction_table").show()
+
+CarbonProperties.getInstance().addProperty(
--- End diff --
why set this property here?
---
[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2606 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6209/ ---
[GitHub] carbondata pull request #2620: [CARBONDATA-2839] Add custom compaction examp...
GitHub user Xaprice opened a pull request: https://github.com/apache/carbondata/pull/2620 [CARBONDATA-2839] Add custom compaction example Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ x ] Any interfaces changed? no - [ x ] Any backward compatibility impacted? no - [ x ] Document update required? no - [ x ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ x ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. small change You can merge this pull request into a Git repository by running: $ git pull https://github.com/Xaprice/carbondata custom_compaction_example Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2620.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2620 commit 0402a5f1f66027e5b7d72b514eb80b09c2d7222e Author: Jin Zhou Date: 2018-08-08T09:01:43Z [CARBONDATA-2839] Add custom compaction example ---
[GitHub] carbondata pull request #2606: [CARBONDATA-2817]Thread Leak in Update and in...
Github user BJangir commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2606#discussion_r208515484
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/loading/steps/CarbonRowDataWriterProcessorStepImpl.java
---
@@ -169,24 +171,36 @@ private void doExecute(Iterator
iterator, int iteratorIndex) thr
if (rowsNotExist) {
rowsNotExist = false;
dataHandler =
CarbonFactHandlerFactory.createCarbonFactHandler(model);
+this.carbonFactHandlers.add(dataHandler);
dataHandler.initialise();
}
processBatch(iterator.next(), dataHandler, iteratorIndex);
}
-if (!rowsNotExist) {
- finish(dataHandler, iteratorIndex);
+try {
+ if (!rowsNotExist) {
+finish(dataHandler, iteratorIndex);
+ }
+} finally {
+ carbonFactHandlers.remove(dataHandler);
}
+
+
}
@Override protected String getStepName() {
return "Data Writer";
}
private void finish(CarbonFactHandler dataHandler, int iteratorIndex) {
+CarbonDataWriterException exception = null;
--- End diff --
Fixed
---
[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2617 retest this please ---
[jira] [Created] (CARBONDATA-2839) Add custom compaction example
Jin Zhou created CARBONDATA-2839: Summary: Add custom compaction example Key: CARBONDATA-2839 URL: https://issues.apache.org/jira/browse/CARBONDATA-2839 Project: CarbonData Issue Type: Task Components: examples Reporter: Jin Zhou Assignee: Jin Zhou Add custom compaction example -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2617 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6557/ ---
[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2617 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7832/ ---
[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...
Github user QiangCai commented on the issue: https://github.com/apache/carbondata/pull/2616 LGTM ---
[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2617 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6208/ ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2619 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6207/ ---
[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2617 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6206/ ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2619 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6558/ ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2619 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7833/ ---
[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2616 LGTM ---
[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2617 retest SDV please ---
[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2617 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6205/ ---
[GitHub] carbondata pull request #2619: [CARBONDATA-2819] Fixed cannot drop preagg da...
GitHub user Sssan520 opened a pull request:
https://github.com/apache/carbondata/pull/2619
[CARBONDATA-2819] Fixed cannot drop preagg datamap on table if the tabâ¦
1.Use "SET carbon.datamap.visible.{dbName}.{mainTable}.{datamapName} =
{true or false}",if datamap provider is 'preagg',block this configuration.
2.When create preagg datamap, now also create its datamap schema file in
system folder.
3.Use command "show datamap on table {tableName}" will get its all datamaps
from datamap schema files
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?No
- [ ] Any backward compatibility impacted? No
- [ ] Document update required? No
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance
test report.
- Any additional information to help reviewers in testing this
change.
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Sssan520/carbondata droppreagg
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2619.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2619
commit 7884ba86f13b89fb23895400bbac12eaa4e62546
Author: Sssan520
Date: 2018-08-08T06:27:41Z
[CARBONDATA-2819]Fixed cannot drop preagg datamap on table if the table has
other index datamaps
---
[GitHub] carbondata pull request #2618: [CARBONDATA-2819] Fixed cannot drop preagg da...
GitHub user lianganping opened a pull request:
https://github.com/apache/carbondata/pull/2618
[CARBONDATA-2819] Fixed cannot drop preagg datamap on table if the tabâ¦
1.Use "SET carbon.datamap.visible.{dbName}.{mainTable}.{datamapName} =
{true or false}",if datamap provider is 'preagg',block this configuration.
2.When create preagg datamap, now also create its datamap schema file in
system folder.
3.Use command "show datamap on table {tableName}" will get its all datamaps
from datamap schema files
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed? No
- [ ] Any backward compatibility impacted? No
- [ ] Document update required? No
- [ ] Testing done NA
Please provide details on
- Whether new unit test cases have been added or why no new tests
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance
test report.
- Any additional information to help reviewers in testing this
change.
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Sssan520/carbondata droppreagg
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2618.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2618
commit 7884ba86f13b89fb23895400bbac12eaa4e62546
Author: Sssan520
Date: 2018-08-08T06:27:41Z
[CARBONDATA-2819]Fixed cannot drop preagg datamap on table if the table has
other index datamaps
---
[GitHub] carbondata issue #2618: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2618 Can one of the admins verify this patch? ---
[GitHub] carbondata pull request #2618: [CARBONDATA-2819] Fixed cannot drop preagg da...
Github user lianganping closed the pull request at: https://github.com/apache/carbondata/pull/2618 ---
[jira] [Resolved] (CARBONDATA-2659) Support partitioned carbon table by DataFrame.write
[ https://issues.apache.org/jira/browse/CARBONDATA-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2659. - Resolution: Fixed Fix Version/s: (was: 1.5.0) > Support partitioned carbon table by DataFrame.write > --- > > Key: CARBONDATA-2659 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2659 > Project: CarbonData > Issue Type: Improvement >Reporter: Jacky Li >Priority: Major > Fix For: 1.4.1 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > Currently only partition table is only supported by SQL, it should be > supported by Spark DataFrame API also. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2415: [CARBONDATA-2659] Support partition table by ...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2415 ---
[GitHub] carbondata issue #2415: [CARBONDATA-2659] Support partition table by DataFra...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2415 LGTM ---
[GitHub] carbondata pull request #2613: [HOTFIX] Modified code to fix the degrade in ...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2613 ---
[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2617 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6204/ ---
[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2613 LGTM ---
[jira] [Resolved] (CARBONDATA-2834) Refactor code to remove nested for loop to extract invalidTimestampRange.
[ https://issues.apache.org/jira/browse/CARBONDATA-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2834. - Resolution: Fixed Fix Version/s: 1.4.1 > Refactor code to remove nested for loop to extract invalidTimestampRange. > - > > Key: CARBONDATA-2834 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2834 > Project: CarbonData > Issue Type: Bug >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor >Priority: Major > Fix For: 1.4.1 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Reactor getInvalidTimestampRange method in SegmentUpdateStatusManager because > it has an unnecessary nested loop to get timestamp from invalid segments. > This will cause query performance degradation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2612: [CARBONDATA-2834] Remove unnecessary nested l...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2612 ---
[GitHub] carbondata issue #2612: [CARBONDATA-2834] Remove unnecessary nested looping ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2612 LGTM ---
[GitHub] carbondata pull request #2617: [CARBONDATA-2838] Added SDV test cases for Lo...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2617 [CARBONDATA-2838] Added SDV test cases for Local Dictionary Support Added SDV test cases for Local Dictionary Support - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? NA - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. Test cases are added - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata local_dict_sdv Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2617.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2617 commit d05ba074b239e3d52672e1638eece39367fddd40 Author: praveenmeenakshi56 Date: 2018-08-08T06:15:49Z Added SDV test cases for Local Dictionary Support ---
[jira] [Created] (CARBONDATA-2838) Add SDV test cases for Local Dictionary Support
Praveen M P created CARBONDATA-2838: --- Summary: Add SDV test cases for Local Dictionary Support Key: CARBONDATA-2838 URL: https://issues.apache.org/jira/browse/CARBONDATA-2838 Project: CarbonData Issue Type: Sub-task Reporter: Praveen M P Assignee: Praveen M P -- This message was sent by Atlassian JIRA (v7.6.3#76005)
