[GitHub] carbondata issue #2187: [CARBONDATA-2362] Changing the Cacheable object from...

[dhatchayani]

Github user dhatchayani commented on the issue:

https://github.com/apache/carbondata/pull/2187
  
retest this please


---

[GitHub] carbondata issue #2199: [CARBONDATA-2370] Added document for presto multinod...

[geetikagupta16]

Github user geetikagupta16 commented on the issue:

https://github.com/apache/carbondata/pull/2199
  
@sgururajshetty I have made the required changes. Please review


---

[GitHub] carbondata issue #2173: [CARBONDATA-2379] Support Search mode run in the clu...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2173
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4137/



---

[GitHub] carbondata issue #2206: [CARBONDATA-2376] Improve Lucene datamap performance...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2206
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5315/



---

[GitHub] carbondata issue #2197: [CARBONDATA-2371] Add Profiler output in EXPLAIN com...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2197
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4136/



---

[jira] [Resolved] (CARBONDATA-2359) Support applicable load options and table properties for Non Transactional table

[Venkata Ramana G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Venkata Ramana G resolved CARBONDATA-2359.
--
   Resolution: Fixed
 Assignee: Ajantha Bhat
Fix Version/s: 1.4.0

> Support applicable load options and table properties for Non Transactional 
> table
> 
>
> Key: CARBONDATA-2359
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2359
> Project: CarbonData
>  Issue Type: Sub-task
>Reporter: Ajantha Bhat
>Assignee: Ajantha Bhat
>Priority: Major
> Fix For: 1.4.0
>
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #1668: [CARBONDATA-1787] Updated data-management-on-carbond...

[vandana7]

Github user vandana7 commented on the issue:

https://github.com/apache/carbondata/pull/1668
  
please retest this


---

[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/2190


---

[GitHub] carbondata issue #2182: [CARBONDATA-2354] fixed streaming example

[vandana7]

Github user vandana7 commented on the issue:

https://github.com/apache/carbondata/pull/2182
  
@jackylk please review 


---

[GitHub] carbondata issue #2190: [CARBONDATA-2359] Support applicable load options an...

[gvramana]

Github user gvramana commented on the issue:

https://github.com/apache/carbondata/pull/2190
  
LGTM


---

[GitHub] carbondata pull request #2209: [WIP][Non Transactional Table]Avro Record Com...

[sounakr]

GitHub user sounakr opened a pull request:

https://github.com/apache/carbondata/pull/2209

[WIP][Non Transactional Table]Avro Record Complex Type Implementation

Avro Complex DataType Support.

 - [ ] Any interfaces changed?
 
 - [ ] Any backward compatibility impacted?
 
 - [ ] Document update required?

 - [ ] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sounakr/incubator-carbondata avro-datatype

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2209.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2209


commit 464ade68ca1d45a4fcac15c070b7edae8838bff7
Author: sounakr 
Date:   2018-04-23T04:48:10Z

Avro Record Complex Type Implementation




---

[GitHub] carbondata issue #2206: [CARBONDATA-2376] Improve Lucene datamap performance...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2206
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4135/



---

[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[ajantha-bhat]

Github user ajantha-bhat commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2190#discussion_r183274384
  
--- Diff: 
store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java 
---
@@ -123,6 +125,43 @@ public CarbonWriterBuilder uniqueIdentifier(long UUID) 
{
 return this;
   }
 
+  /**
+   * To support the load options for sdk writer
+   * @param options key,value pair of load options.
+   *supported keys values are
+   *a. bad_records_logger_enable -- true, false
--- End diff --

ok


---

[GitHub] carbondata issue #2205: [CARBONDATA-2377][CarbonSearch] Support message thro...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2205
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4134/



---

[jira] [Updated] (CARBONDATA-2379) Support Search mode run in the cluster and fix some error

[xubo245 (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

xubo245 updated CARBONDATA-2379:

Description: 
##  Error 1
 JVM crashed

{code:java}
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7fa3ae68c0c7, pid=28382, tid=0x7fa281f8e700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_161-b12) (build 
1.8.0_161-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode linux-amd64 
compressed oops)
# Problematic frame:
# V  [libjvm.so+0xa9b0c7]  Unsafe_GetInt+0x47
#
# Failed to write core dump. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /huawei/xubo/git/carbondata2/hs_err_pid28382.log
Compiled method (nm)  106412 8135 n 0   sun.misc.Unsafe::getInt (native)
 total in heap  [0x7fa399a84b10,0x7fa399a84e70] = 864
 relocation [0x7fa399a84c38,0x7fa399a84c80] = 72
 main code  [0x7fa399a84c80,0x7fa399a84e70] = 496
Compiled method (nm)  106412 8135 n 0   sun.misc.Unsafe::getInt (native)
 total in heap  [0x7fa399a84b10,0x7fa399a84e70] = 864
 relocation [0x7fa399a84c38,0x7fa399a84c80] = 72
 main code  [0x7fa399a84c80,0x7fa399a84e70] = 496
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
{code}

## Error 2
 It will error that concurrentQueryBenchmark read data from HDFS in clster

  was:


 - Error 1
 JVM crashed

{code:java}
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7fa3ae68c0c7, pid=28382, tid=0x7fa281f8e700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_161-b12) (build 
1.8.0_161-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode linux-amd64 
compressed oops)
# Problematic frame:
# V  [libjvm.so+0xa9b0c7]  Unsafe_GetInt+0x47
#
# Failed to write core dump. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /huawei/xubo/git/carbondata2/hs_err_pid28382.log
Compiled method (nm)  106412 8135 n 0   sun.misc.Unsafe::getInt (native)
 total in heap  [0x7fa399a84b10,0x7fa399a84e70] = 864
 relocation [0x7fa399a84c38,0x7fa399a84c80] = 72
 main code  [0x7fa399a84c80,0x7fa399a84e70] = 496
Compiled method (nm)  106412 8135 n 0   sun.misc.Unsafe::getInt (native)
 total in heap  [0x7fa399a84b10,0x7fa399a84e70] = 864
 relocation [0x7fa399a84c38,0x7fa399a84c80] = 72
 main code  [0x7fa399a84c80,0x7fa399a84e70] = 496
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
{code}

## Error 2
 It will error that concurrentQueryBenchmark read data from HDFS in clster


> Support Search mode run in the cluster and fix some error
> -
>
> Key: CARBONDATA-2379
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2379
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: xubo245
>Assignee: xubo245
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> ##  Error 1
>  JVM crashed
> {code:java}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7fa3ae68c0c7, pid=28382, tid=0x7fa281f8e700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_161-b12) (build 
> 1.8.0_161-b12)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # V  [libjvm.so+0xa9b0c7]  Unsafe_GetInt+0x47
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core 
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /huawei/xubo/git/carbondata2/hs_err_pid28382.log
> Compiled method (nm)  106412 8135 n 0   sun.misc.Unsafe::getInt 
> (native)
>  total in heap  [0x7fa399a84b10,0x7fa399a84e70] = 864
>  relocation [0x7fa399a84c38,0x7fa399a84c80] = 72
>  main code  [0x7fa399a84c80,0x7fa399a84e70] = 496
> Compiled method (nm)  106412 8135 n 0   sun.misc.Unsafe::getInt 
> (native)
>  total in heap  [0x7fa399a84b10,0x7fa399a84e70] = 864
>  relocation [0x7fa399a84c38,0x7fa399a84c80] = 72
>  main code  [0x7fa399a84c80,0x7fa399a84e70] = 496
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> #
> {code}
> ## Error 2
>  It will error that 

[jira] [Updated] (CARBONDATA-2379) Support Search mode run in the cluster and fix some error

[xubo245 (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

xubo245 updated CARBONDATA-2379:

Description: 
 - Error 1
 JVM crashed

{code:java}
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7fa3ae68c0c7, pid=28382, tid=0x7fa281f8e700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_161-b12) (build 
1.8.0_161-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode linux-amd64 
compressed oops)
# Problematic frame:
# V  [libjvm.so+0xa9b0c7]  Unsafe_GetInt+0x47
#
# Failed to write core dump. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /huawei/xubo/git/carbondata2/hs_err_pid28382.log
Compiled method (nm)  106412 8135 n 0   sun.misc.Unsafe::getInt (native)
 total in heap  [0x7fa399a84b10,0x7fa399a84e70] = 864
 relocation [0x7fa399a84c38,0x7fa399a84c80] = 72
 main code  [0x7fa399a84c80,0x7fa399a84e70] = 496
Compiled method (nm)  106412 8135 n 0   sun.misc.Unsafe::getInt (native)
 total in heap  [0x7fa399a84b10,0x7fa399a84e70] = 864
 relocation [0x7fa399a84c38,0x7fa399a84c80] = 72
 main code  [0x7fa399a84c80,0x7fa399a84e70] = 496
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
{code}

 - Error 2
 It will error that concurrentQueryBenchmark read data from HDFS in clster

  was:
##  Error 1
 JVM crashed

{code:java}
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7fa3ae68c0c7, pid=28382, tid=0x7fa281f8e700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_161-b12) (build 
1.8.0_161-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode linux-amd64 
compressed oops)
# Problematic frame:
# V  [libjvm.so+0xa9b0c7]  Unsafe_GetInt+0x47
#
# Failed to write core dump. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /huawei/xubo/git/carbondata2/hs_err_pid28382.log
Compiled method (nm)  106412 8135 n 0   sun.misc.Unsafe::getInt (native)
 total in heap  [0x7fa399a84b10,0x7fa399a84e70] = 864
 relocation [0x7fa399a84c38,0x7fa399a84c80] = 72
 main code  [0x7fa399a84c80,0x7fa399a84e70] = 496
Compiled method (nm)  106412 8135 n 0   sun.misc.Unsafe::getInt (native)
 total in heap  [0x7fa399a84b10,0x7fa399a84e70] = 864
 relocation [0x7fa399a84c38,0x7fa399a84c80] = 72
 main code  [0x7fa399a84c80,0x7fa399a84e70] = 496
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
{code}

## Error 2
 It will error that concurrentQueryBenchmark read data from HDFS in clster


> Support Search mode run in the cluster and fix some error
> -
>
> Key: CARBONDATA-2379
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2379
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: xubo245
>Assignee: xubo245
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
>  - Error 1
>  JVM crashed
> {code:java}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7fa3ae68c0c7, pid=28382, tid=0x7fa281f8e700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_161-b12) (build 
> 1.8.0_161-b12)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # V  [libjvm.so+0xa9b0c7]  Unsafe_GetInt+0x47
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core 
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /huawei/xubo/git/carbondata2/hs_err_pid28382.log
> Compiled method (nm)  106412 8135 n 0   sun.misc.Unsafe::getInt 
> (native)
>  total in heap  [0x7fa399a84b10,0x7fa399a84e70] = 864
>  relocation [0x7fa399a84c38,0x7fa399a84c80] = 72
>  main code  [0x7fa399a84c80,0x7fa399a84e70] = 496
> Compiled method (nm)  106412 8135 n 0   sun.misc.Unsafe::getInt 
> (native)
>  total in heap  [0x7fa399a84b10,0x7fa399a84e70] = 864
>  relocation [0x7fa399a84c38,0x7fa399a84c80] = 72
>  main code  [0x7fa399a84c80,0x7fa399a84e70] = 496
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> #
> {code}
>  - Error 2
>  It will error that 

[jira] [Updated] (CARBONDATA-2379) Support Search mode run in the cluster and fix some error

[xubo245 (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

xubo245 updated CARBONDATA-2379:

Description: 


 - Error 1
 JVM crashed

{code:java}
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7fa3ae68c0c7, pid=28382, tid=0x7fa281f8e700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_161-b12) (build 
1.8.0_161-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode linux-amd64 
compressed oops)
# Problematic frame:
# V  [libjvm.so+0xa9b0c7]  Unsafe_GetInt+0x47
#
# Failed to write core dump. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /huawei/xubo/git/carbondata2/hs_err_pid28382.log
Compiled method (nm)  106412 8135 n 0   sun.misc.Unsafe::getInt (native)
 total in heap  [0x7fa399a84b10,0x7fa399a84e70] = 864
 relocation [0x7fa399a84c38,0x7fa399a84c80] = 72
 main code  [0x7fa399a84c80,0x7fa399a84e70] = 496
Compiled method (nm)  106412 8135 n 0   sun.misc.Unsafe::getInt (native)
 total in heap  [0x7fa399a84b10,0x7fa399a84e70] = 864
 relocation [0x7fa399a84c38,0x7fa399a84c80] = 72
 main code  [0x7fa399a84c80,0x7fa399a84e70] = 496
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
{code}

## Error 2
 It will error that concurrentQueryBenchmark read data from HDFS in clster
Summary: Support Search mode run in the cluster and fix some error  
(was: Support Search mode run in the cluster)

> Support Search mode run in the cluster and fix some error
> -
>
> Key: CARBONDATA-2379
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2379
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: xubo245
>Assignee: xubo245
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
>  - Error 1
>  JVM crashed
> {code:java}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7fa3ae68c0c7, pid=28382, tid=0x7fa281f8e700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_161-b12) (build 
> 1.8.0_161-b12)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # V  [libjvm.so+0xa9b0c7]  Unsafe_GetInt+0x47
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core 
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /huawei/xubo/git/carbondata2/hs_err_pid28382.log
> Compiled method (nm)  106412 8135 n 0   sun.misc.Unsafe::getInt 
> (native)
>  total in heap  [0x7fa399a84b10,0x7fa399a84e70] = 864
>  relocation [0x7fa399a84c38,0x7fa399a84c80] = 72
>  main code  [0x7fa399a84c80,0x7fa399a84e70] = 496
> Compiled method (nm)  106412 8135 n 0   sun.misc.Unsafe::getInt 
> (native)
>  total in heap  [0x7fa399a84b10,0x7fa399a84e70] = 864
>  relocation [0x7fa399a84c38,0x7fa399a84c80] = 72
>  main code  [0x7fa399a84c80,0x7fa399a84e70] = 496
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> #
> {code}
> ## Error 2
>  It will error that concurrentQueryBenchmark read data from HDFS in clster



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #2173: [CARBONDATA-2379] Support Search mode run in the clu...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2173
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4133/



---

[GitHub] carbondata pull request #2206: [CARBONDATA-2376] Improve Lucene datamap perf...

[QiangCai]

Github user QiangCai commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2206#discussion_r183271230
  
--- Diff: 
datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneDataMapWriter.java
 ---
@@ -102,24 +96,25 @@
 this.indexedCarbonColumns = indexedCarbonColumns;
   }
 
-  private String getIndexPath(long taskId) {
+  private String getIndexPath(String taskName) {
 if (isFineGrain) {
-  return genDataMapStorePathOnTaskId(identifier.getTablePath(), 
segmentId, dataMapName, taskId);
+  return genDataMapStorePathOnTaskId(identifier.getTablePath(), 
segmentId, dataMapName,
+  taskName);
 } else {
   // TODO: where write data in coarse grain data map
-  return genDataMapStorePathOnTaskId(identifier.getTablePath(), 
segmentId, dataMapName, taskId);
+  return genDataMapStorePathOnTaskId(identifier.getTablePath(), 
segmentId, dataMapName,
+  taskName);
 }
   }
 
   /**
* Start of new block notification.
*/
-  public void onBlockStart(String blockId, long taskId) throws IOException 
{
+  public void onBlockStart(String blockId, String indexShardName) throws 
IOException {
--- End diff --

In this method, we should only initialize indexWriter once.
It means all blocks of this task will share the index writer.


---

[GitHub] carbondata pull request #2206: [CARBONDATA-2376] Improve Lucene datamap perf...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2206#discussion_r183269416
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/datamap/dev/DataMapWriter.java ---
@@ -54,7 +54,7 @@ public DataMapWriter(AbsoluteTableIdentifier identifier, 
Segment segment,
*
* @param blockId file name of the carbondata file
*/
-  public abstract void onBlockStart(String blockId, long taskId) throws 
IOException;
+  public abstract void onBlockStart(String blockId, String taskName) 
throws IOException;
--- End diff --

ok


---

[GitHub] carbondata issue #2205: [CARBONDATA-2377][CarbonSearch] Support message thro...

[jackylk]

Github user jackylk commented on the issue:

https://github.com/apache/carbondata/pull/2205
  
retest this please


---

[GitHub] carbondata pull request #2181: [CARBONDATA-2355] Support run SQL on carbonda...

[xubo245]

Github user xubo245 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2181#discussion_r183268192
  
--- Diff: 
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateTableUsingSparkCarbonFileFormat.scala
 ---
@@ -153,6 +153,34 @@ class TestCreateTableUsingSparkCarbonFileFormat 
extends QueryTest with BeforeAnd
 cleanTestData()
   }
 
+  test("Running SQL directly and read carbondata files (sdk Writer Output) 
using the SparkCarbonFileFormat ") {
+buildTestData(false)
+assert(new File(filePath).exists())
+sql("DROP TABLE IF EXISTS sdkOutputTable")
+
+//data source file format
+if (sqlContext.sparkContext.version.startsWith("2.1")) {
+  //data source file format
+  sql(s"""CREATE TABLE sdkOutputTable USING carbonfile OPTIONS (PATH 
'$filePath') """)
--- End diff --

It's one of fileformat.


---

[GitHub] carbondata issue #2181: [CARBONDATA-2355] Support run SQL on carbondata file...

[xubo245]

Github user xubo245 commented on the issue:

https://github.com/apache/carbondata/pull/2181
  
It's function implementation.


---

[GitHub] carbondata issue #2205: [CARBONDATA-2377][CarbonSearch] Support message thro...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2205
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5311/



---

[GitHub] carbondata issue #2205: [CARBONDATA-2377][CarbonSearch] Support message thro...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2205
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4131/



---

[jira] [Created] (CARBONDATA-2380) Support visible/invisible datamap for performance tuning

[xuchuanyin (JIRA)]

xuchuanyin created CARBONDATA-2380:
--

 Summary: Support visible/invisible datamap for performance tuning
 Key: CARBONDATA-2380
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2380
 Project: CarbonData
  Issue Type: New Feature
  Components: data-query
Reporter: xuchuanyin
Assignee: xuchuanyin


Invisible datamap will not be used during the query, which can be used to 
verify whether to remove this datamap in the future.

 

This feature is similar to `Invisible indexed` in mysql 
(https://dev.mysql.com/doc/refman/8.0/en/invisible-indexes.html).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #2205: [CARBONDATA-2377][CarbonSearch] Support messa...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2205#discussion_r183262237
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
 ---
@@ -1671,6 +1673,17 @@
 
   public static final String CARBON_SEARCH_MODE_WORKER_PORT_DEFAULT = 
"10021";
 
+  /**
+   * If number of search request sent to Worker exceed this limit, Master 
will reschedule
+   * the request to another worker. In such case, locality will be lost in 
HDFS scenario, but
+   * it is find for S3 scenario.
--- End diff --

fixed


---

[GitHub] carbondata pull request #2205: [CARBONDATA-2377][CarbonSearch] Support messa...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2205#discussion_r183262230
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java ---
@@ -1501,4 +1501,16 @@ public static int getSearchWorkerPort() {
   return 
Integer.parseInt(CarbonCommonConstants.CARBON_SEARCH_MODE_WORKER_PORT_DEFAULT);
 }
   }
+
+  public static int getMaxWorkloadForWorker(int workerCores) {
+int defaultValue = workerCores * 10;
+try {
+  return Integer.parseInt(
--- End diff --

fixed


---

[jira] [Created] (CARBONDATA-2379) Support Search mode run in the cluster

[xubo245 (JIRA)]

xubo245 created CARBONDATA-2379:
---

 Summary: Support Search mode run in the cluster
 Key: CARBONDATA-2379
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2379
 Project: CarbonData
  Issue Type: Improvement
Reporter: xubo245
Assignee: xubo245






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #2181: [CARBONDATA-2355] Support run SQL on carbondata file...

[xuchuanyin]

Github user xuchuanyin commented on the issue:

https://github.com/apache/carbondata/pull/2181
  
@xubo245 This PR is only an example, not the function implementation, right?


---

[GitHub] carbondata pull request #2181: [CARBONDATA-2355] Support run SQL on carbonda...

[xuchuanyin]

Github user xuchuanyin commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2181#discussion_r183261488
  
--- Diff: 
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateTableUsingSparkCarbonFileFormat.scala
 ---
@@ -153,6 +153,34 @@ class TestCreateTableUsingSparkCarbonFileFormat 
extends QueryTest with BeforeAnd
 cleanTestData()
   }
 
+  test("Running SQL directly and read carbondata files (sdk Writer Output) 
using the SparkCarbonFileFormat ") {
+buildTestData(false)
+assert(new File(filePath).exists())
+sql("DROP TABLE IF EXISTS sdkOutputTable")
+
+//data source file format
+if (sqlContext.sparkContext.version.startsWith("2.1")) {
+  //data source file format
+  sql(s"""CREATE TABLE sdkOutputTable USING carbonfile OPTIONS (PATH 
'$filePath') """)
--- End diff --

Is `carbonfile` a reserved key word?


---

[GitHub] carbondata issue #1929: [CARBONDATA-2129][CARBONDATA-2094][CARBONDATA-1516] ...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1929
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4130/



---

[GitHub] carbondata issue #1929: [CARBONDATA-2129][CARBONDATA-2094][CARBONDATA-1516] ...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1929
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5310/



---

[GitHub] carbondata pull request #2205: [CARBONDATA-2377][CarbonSearch] Support messa...

[xuchuanyin]

Github user xuchuanyin commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2205#discussion_r183260157
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
 ---
@@ -1671,6 +1673,17 @@
 
   public static final String CARBON_SEARCH_MODE_WORKER_PORT_DEFAULT = 
"10021";
 
+  /**
+   * If number of search request sent to Worker exceed this limit, Master 
will reschedule
+   * the request to another worker. In such case, locality will be lost in 
HDFS scenario, but
+   * it is find for S3 scenario.
--- End diff --

'find' -> fine


---

[GitHub] carbondata pull request #2205: [CARBONDATA-2377][CarbonSearch] Support messa...

[xuchuanyin]

Github user xuchuanyin commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2205#discussion_r183260236
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java ---
@@ -1501,4 +1501,16 @@ public static int getSearchWorkerPort() {
   return 
Integer.parseInt(CarbonCommonConstants.CARBON_SEARCH_MODE_WORKER_PORT_DEFAULT);
 }
   }
+
+  public static int getMaxWorkloadForWorker(int workerCores) {
+int defaultValue = workerCores * 10;
+try {
+  return Integer.parseInt(
--- End diff --

validate number >= 0


---

[GitHub] carbondata issue #1929: [CARBONDATA-2129][CARBONDATA-2094][CARBONDATA-1516] ...

[xubo245]

Github user xubo245 commented on the issue:

https://github.com/apache/carbondata/pull/1929
  
retest this please


---

[GitHub] carbondata pull request #2169: [CARBONDATA-2344][DataMap] Fix bugs in mappin...

[xuchuanyin]

Github user xuchuanyin closed the pull request at:

https://github.com/apache/carbondata/pull/2169


---

[GitHub] carbondata issue #1570: [CARBONDATA-1730] Support skip.header.line.count opt...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1570
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4472/



---

[GitHub] carbondata issue #1640: [WIP] Annotate carbon property

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1640
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4471/



---

[GitHub] carbondata issue #1661: [CARBONDATA-1678] Fixed incorrect partitionCount on ...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1661
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4470/



---

[GitHub] carbondata issue #1664: [CARBONDATA-1890][Improvement]-Changes for Struct Op...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1664
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4469/



---

[GitHub] carbondata issue #1668: [CARBONDATA-1787] Updated data-management-on-carbond...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1668
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4468/



---

[GitHub] carbondata issue #2197: [CARBONDATA-2371] Add Profiler output in EXPLAIN com...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2197
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4129/



---

[GitHub] carbondata issue #1680: [WIP] fixing text parsing exception

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1680
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4467/



---

[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1751
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4466/



---

[GitHub] carbondata issue #2190: [CARBONDATA-2359] Support applicable load options an...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2190
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5308/



---

[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1812
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4465/



---

[GitHub] carbondata issue #2190: [CARBONDATA-2359] Support applicable load options an...

[ajantha-bhat]

Github user ajantha-bhat commented on the issue:

https://github.com/apache/carbondata/pull/2190
  
retest this please


---

[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[ajantha-bhat]

Github user ajantha-bhat commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2190#discussion_r183242493
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchemaBuilder.java
 ---
@@ -75,6 +78,11 @@ public TableSchemaBuilder tableName(String tableName) {
 return this;
   }
 
+  public TableSchemaBuilder resetTransactionalTable(boolean 
isTransactionalTable) {
--- End diff --

ok


---

[GitHub] carbondata issue #2187: [CARBONDATA-2362] Changing the Cacheable object from...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2187
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5307/



---

[GitHub] carbondata issue #2187: [CARBONDATA-2362] Changing the Cacheable object from...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2187
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4127/



---

[GitHub] carbondata issue #2208: [CARBONDATA-2378] Support enable/disable search mode...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2208
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5306/



---

[GitHub] carbondata issue #2208: [CARBONDATA-2378] Support enable/disable search mode...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2208
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4126/



---

[GitHub] carbondata issue #1904: [CARBONDATA-2059] - Changes to support compaction fo...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1904
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4462/



---

[GitHub] carbondata issue #1929: [CARBONDATA-2129][CARBONDATA-2094][CARBONDATA-1516] ...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1929
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4461/



---

[jira] [Resolved] (CARBONDATA-2375) Add CG prune before FG prune

[Jacky Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jacky Li resolved CARBONDATA-2375.
--
   Resolution: Fixed
Fix Version/s: 1.4.0

> Add CG prune before FG prune
> 
>
> Key: CARBONDATA-2375
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2375
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Ravindra Pesala
>Priority: Major
> Fix For: 1.4.0
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> CG prune before FG prune, and passes the pruned segments and indexfiles to FG 
> DataMap for further pruning.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #2204: [CARBONDATA-2375] Added CG prune before FG pr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/2204


---

[GitHub] carbondata issue #2187: [CARBONDATA-2362] Changing the Cacheable object from...

[dhatchayani]

Github user dhatchayani commented on the issue:

https://github.com/apache/carbondata/pull/2187
  
Retest this please


---

[GitHub] carbondata issue #2204: [CARBONDATA-2375] Added CG prune before FG prune.

[jackylk]

Github user jackylk commented on the issue:

https://github.com/apache/carbondata/pull/2204
  
LGTM


---

[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2190#discussion_r183239308
  
--- Diff: 
store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java 
---
@@ -210,11 +260,26 @@ private CarbonTable buildCarbonTable() {
   tableSchemaBuilder = tableSchemaBuilder.blockletSize(blockletSize);
 }
 
-List sortColumnsList;
-if (sortColumns != null) {
-  sortColumnsList = Arrays.asList(sortColumns);
+if (!isTransactionalTable) {
--- End diff --

always directly set value to tableSchemaBuilder


---

[GitHub] carbondata issue #2208: [CARBONDATA-2378] Support enable/disable search mode...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2208
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4125/



---

[GitHub] carbondata issue #2208: [CARBONDATA-2378] Support enable/disable search mode...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2208
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5305/



---

[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2190#discussion_r183239131
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchemaBuilder.java
 ---
@@ -75,6 +78,11 @@ public TableSchemaBuilder tableName(String tableName) {
 return this;
   }
 
+  public TableSchemaBuilder resetTransactionalTable(boolean 
isTransactionalTable) {
--- End diff --

change function name to setTransactionTable , there nothing resetting 
happening in this function


---

[GitHub] carbondata pull request #2208: [CARBONDATA-2378] Support enable/disable sear...

[jackylk]

GitHub user jackylk opened a pull request:

https://github.com/apache/carbondata/pull/2208

[CARBONDATA-2378] Support enable/disable search mode in ThriftServer

User can enable or disable search mode when using ThriftServer, by using:
`set carbon.search.enabled = true` or `set carbon.search.enabled = false`

 - [ ] Any interfaces changed?
 
 - [X] Any backward compatibility impacted?
 No
 - [X] Document update required?
Yes
 - [X] Testing done
Test case added
 - [X] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
NA

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jackylk/incubator-carbondata jdbc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2208.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2208


commit dce1b7ee76a80c68bea151017e4df679c9f7a9f9
Author: Jacky Li 
Date:   2018-04-22T14:32:57Z

add support for ThriftServer




---

[jira] [Created] (CARBONDATA-2378) Support enable/disable search mode in ThriftServer

[Jacky Li (JIRA)]

Jacky Li created CARBONDATA-2378:


 Summary: Support enable/disable search mode in ThriftServer
 Key: CARBONDATA-2378
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2378
 Project: CarbonData
  Issue Type: New Feature
Reporter: Jacky Li
Assignee: Jacky Li
 Fix For: 1.4.0


User should be able to enable or disable search mode when using ThriftServer



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #1933: [CARBONDATA-2132] [Partition] Fixed Error while load...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1933
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4460/



---

[GitHub] carbondata issue #2206: [CARBONDATA-2376] Improve Lucene datamap performance...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2206
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5304/



---

[GitHub] carbondata issue #2206: [CARBONDATA-2376] Improve Lucene datamap performance...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2206
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4124/



---

[GitHub] carbondata issue #1942: [CARBONDATA-2136] Fixed bug related to data load for...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1942
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4459/



---

[GitHub] carbondata issue #2190: [CARBONDATA-2359] Support applicable load options an...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2190
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5303/



---

[GitHub] carbondata issue #1963: [WIP][CARBONDATA-2153] Failed to update table status...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1963
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4458/



---

[GitHub] carbondata issue #2206: [CARBONDATA-2376] Improve Lucene datamap performance...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2206
  
@xuchuanyin I have fixed the Bloom Filter as well.


---

[GitHub] carbondata issue #2190: [CARBONDATA-2359] Support applicable load options an...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2190
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4123/



---

[GitHub] carbondata issue #2204: [CARBONDATA-2375] Added CG prune before FG prune.

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2204
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5302/



---

[GitHub] carbondata issue #2204: [CARBONDATA-2375] Added CG prune before FG prune.

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2204
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4122/



---

[GitHub] carbondata issue #1990: [CARBONDATA-2195] Add new test case for partition fe...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1990
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4457/



---

[GitHub] carbondata issue #2190: [CARBONDATA-2359] Support applicable load options an...

[ajantha-bhat]

Github user ajantha-bhat commented on the issue:

https://github.com/apache/carbondata/pull/2190
  
Retest this please


---

[GitHub] carbondata issue #2014: [CARBONDATA-2198] Fixed bug for streaming data for b...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2014
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4456/



---

[GitHub] carbondata issue #2206: [CARBONDATA-2376] Improve Lucene datamap performance...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2206
  
@xuchuanyin yes, blockletids keeps increasing in one task. Now blocklets 
are with respect to one task(index file). 


---

[GitHub] carbondata pull request #2206: [CARBONDATA-2376] Improve Lucene datamap perf...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2206#discussion_r183233543
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/datamap/Segment.java ---
@@ -39,6 +41,11 @@
 
   private String segmentFileName;
 
+  /**
+   * List of tasks which are already got filtered through CG index 
operation.
+   */
+  private Set filteredTaskNames = new HashSet<>();
--- End diff --

ok


---

[GitHub] carbondata pull request #2204: [CARBONDATA-2375] Added CG prune before FG pr...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2204#discussion_r183233442
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/datamap/DataMapChooser.java ---
@@ -93,6 +93,48 @@ public DataMapExprWrapper choose(CarbonTable 
carbonTable, FilterResolverIntf res
 resolverIntf);
   }
 
+  /**
+   * Return a chosen datamap based on input filter. See {@link 
DataMapChooser}
+   */
+  public DataMapExprWrapper chooseFG(CarbonTable carbonTable, 
FilterResolverIntf resolverIntf)
+  throws IOException {
+if (resolverIntf != null) {
+  Expression expression = resolverIntf.getFilterExpression();
+  // First check for FG datamaps if any exist
+  List allDataMapFG =
+  DataMapStoreManager.getInstance().getAllDataMap(carbonTable, 
DataMapLevel.FG);
+  ExpressionTuple tuple = selectDataMap(expression, allDataMapFG, 
resolverIntf);
+  if (tuple.dataMapExprWrapper != null) {
+return tuple.dataMapExprWrapper;
+  }
+}
+// Return the default datamap if no other datamap exists.
+return null;
+  }
+
+  /**
+   * Return a chosen datamap based on input filter. See {@link 
DataMapChooser}
+   */
+  public DataMapExprWrapper chooseCG(CarbonTable carbonTable, 
FilterResolverIntf resolverIntf)
--- End diff --

ok


---

[GitHub] carbondata pull request #2204: [CARBONDATA-2375] Added CG prune before FG pr...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2204#discussion_r183233446
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/datamap/DataMapChooser.java ---
@@ -93,6 +93,48 @@ public DataMapExprWrapper choose(CarbonTable 
carbonTable, FilterResolverIntf res
 resolverIntf);
   }
 
+  /**
+   * Return a chosen datamap based on input filter. See {@link 
DataMapChooser}
+   */
+  public DataMapExprWrapper chooseFG(CarbonTable carbonTable, 
FilterResolverIntf resolverIntf)
--- End diff --

ok


---

[GitHub] carbondata pull request #2204: [CARBONDATA-2375] Added CG prune before FG pr...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2204#discussion_r183233439
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/datamap/DataMapChooser.java ---
@@ -93,6 +93,48 @@ public DataMapExprWrapper choose(CarbonTable 
carbonTable, FilterResolverIntf res
 resolverIntf);
   }
 
+  /**
+   * Return a chosen datamap based on input filter. See {@link 
DataMapChooser}
--- End diff --

ok


---

[GitHub] carbondata pull request #2204: [CARBONDATA-2375] Added CG prune before FG pr...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2204#discussion_r183233431
  
--- Diff: 
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java ---
@@ -359,23 +359,27 @@ protected Expression 
getFilterPredicates(Configuration configuration) {
 .getProperty(CarbonCommonConstants.USE_DISTRIBUTED_DATAMAP,
 CarbonCommonConstants.USE_DISTRIBUTED_DATAMAP_DEFAULT));
 DataMapExprWrapper dataMapExprWrapper =
-
DataMapChooser.get().choose(getOrCreateCarbonTable(job.getConfiguration()), 
resolver);
+
DataMapChooser.get().chooseCG(getOrCreateCarbonTable(job.getConfiguration()), 
resolver);
 DataMapJob dataMapJob = getDataMapJob(job.getConfiguration());
 List partitionsToPrune = 
getPartitionsToPrune(job.getConfiguration());
 List prunedBlocklets;
-DataMapLevel dataMapLevel = dataMapExprWrapper.getDataMapType();
-if (dataMapJob != null &&
-(distributedCG ||
-(dataMapLevel == DataMapLevel.FG && 
isFgDataMapPruningEnable(job.getConfiguration() {
-  DistributableDataMapFormat datamapDstr =
-  new DistributableDataMapFormat(carbonTable, dataMapExprWrapper, 
segmentIds,
-  partitionsToPrune, BlockletDataMapFactory.class.getName());
-  prunedBlocklets = dataMapJob.execute(datamapDstr, resolver);
-  // Apply expression on the blocklets.
-  prunedBlocklets = dataMapExprWrapper.pruneBlocklets(prunedBlocklets);
+if (distributedCG) {
+  prunedBlocklets =
+  executeDataMapJob(carbonTable, resolver, segmentIds, 
dataMapExprWrapper, dataMapJob,
+  partitionsToPrune);
 } else {
   prunedBlocklets = dataMapExprWrapper.prune(segmentIds, 
partitionsToPrune);
 }
+dataMapExprWrapper =
+
DataMapChooser.get().chooseFG(getOrCreateCarbonTable(job.getConfiguration()), 
resolver);
+if (dataMapExprWrapper != null &&
+dataMapExprWrapper.getDataMapType() == DataMapLevel.FG &&
+isFgDataMapPruningEnable(job.getConfiguration())) {
--- End diff --

ok


---

[GitHub] carbondata pull request #2204: [CARBONDATA-2375] Added CG prune before FG pr...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2204#discussion_r183233432
  
--- Diff: 
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java ---
@@ -417,6 +421,38 @@ protected Expression getFilterPredicates(Configuration 
configuration) {
 return resultFilterredBlocks;
   }
 
+  private List executeDataMapJob(CarbonTable carbonTable,
+  FilterResolverIntf resolver, List segmentIds, 
DataMapExprWrapper dataMapExprWrapper,
+  DataMapJob dataMapJob, List partitionsToPrune) throws 
IOException {
+DistributableDataMapFormat datamapDstr =
+new DistributableDataMapFormat(carbonTable, dataMapExprWrapper, 
segmentIds,
+partitionsToPrune, BlockletDataMapFactory.class.getName());
+List prunedBlocklets = 
dataMapJob.execute(datamapDstr, resolver);
+// Apply expression on the blocklets.
+prunedBlocklets = dataMapExprWrapper.pruneBlocklets(prunedBlocklets);
+return prunedBlocklets;
+  }
+
+  private void updateSegments(List segments, 
List prunedBlocklets) {
--- End diff --

ok, added


---

[GitHub] carbondata pull request #2204: [CARBONDATA-2375] Added CG prune before FG pr...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2204#discussion_r183233426
  
--- Diff: 
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java ---
@@ -359,23 +359,27 @@ protected Expression 
getFilterPredicates(Configuration configuration) {
 .getProperty(CarbonCommonConstants.USE_DISTRIBUTED_DATAMAP,
 CarbonCommonConstants.USE_DISTRIBUTED_DATAMAP_DEFAULT));
 DataMapExprWrapper dataMapExprWrapper =
-
DataMapChooser.get().choose(getOrCreateCarbonTable(job.getConfiguration()), 
resolver);
+
DataMapChooser.get().chooseCG(getOrCreateCarbonTable(job.getConfiguration()), 
resolver);
 DataMapJob dataMapJob = getDataMapJob(job.getConfiguration());
 List partitionsToPrune = 
getPartitionsToPrune(job.getConfiguration());
 List prunedBlocklets;
-DataMapLevel dataMapLevel = dataMapExprWrapper.getDataMapType();
-if (dataMapJob != null &&
-(distributedCG ||
-(dataMapLevel == DataMapLevel.FG && 
isFgDataMapPruningEnable(job.getConfiguration() {
-  DistributableDataMapFormat datamapDstr =
-  new DistributableDataMapFormat(carbonTable, dataMapExprWrapper, 
segmentIds,
-  partitionsToPrune, BlockletDataMapFactory.class.getName());
-  prunedBlocklets = dataMapJob.execute(datamapDstr, resolver);
-  // Apply expression on the blocklets.
-  prunedBlocklets = dataMapExprWrapper.pruneBlocklets(prunedBlocklets);
+if (distributedCG) {
--- End diff --

Ok, corrected now. First always prune with Default datamap and then further 
pruned with remaining datamaps.


---

[GitHub] carbondata issue #2033: [CARBONDATA-2227] Added support to show partition de...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2033
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4455/



---

[GitHub] carbondata issue #2190: [CARBONDATA-2359] Support applicable load options an...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2190
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5301/



---

[GitHub] carbondata issue #2039: [CARBONDATA-2233] [Test] Improve test cases of DBLoc...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2039
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4454/



---

[GitHub] carbondata issue #2190: [CARBONDATA-2359] Support applicable load options an...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2190
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4121/



---

[GitHub] carbondata issue #2148: [CARBONDATA-2323]Distributed search mode using RPC

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2148
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4452/



---

[GitHub] carbondata issue #2205: [CARBONDATA-2377][CarbonSearch] Support message thro...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2205
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5300/



---

[GitHub] carbondata pull request #2202: [wip]change datamap factory interface to chec...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2202#discussion_r183230695
  
--- Diff: 
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableRenameCommand.scala
 ---
@@ -77,6 +78,11 @@ private[sql] case class CarbonAlterTableRenameCommand(
 var oldCarbonTable: CarbonTable = null
 oldCarbonTable = metastore.lookupRelation(Some(oldDatabaseName), 
oldTableName)(sparkSession)
   .asInstanceOf[CarbonRelation].carbonTable
+
+if (CarbonUtil.validateFeatureForDatamap(oldCarbonTable, 
FeaturesList.ALTER_RENAME)) {
--- End diff --

After you handle previous comments, I think this line will become:
` if (oldCarbonTable.canAccept(TableOperation.ALTER_RENAME))`


---

[GitHub] carbondata issue #2205: [CARBONDATA-2377][CarbonSearch] Support message thro...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2205
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4120/



---

[GitHub] carbondata pull request #2202: [wip]change datamap factory interface to chec...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2202#discussion_r183230585
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java ---
@@ -2942,5 +2947,31 @@ public static String 
getBlockId(AbsoluteTableIdentifier identifier, String fileP
 return blockId;
   }
 
+  /**
+   * methods returns whether the feature is supported for the 
corresponding datamap or not
+   * @param carbonTable
+   * @param feature
+   * @return
+   */
+  public static boolean validateFeatureForDatamap(CarbonTable carbonTable, 
FeaturesList feature) {
+boolean isSupported = false;
+try {
+  List datamaps = 
DataMapStoreManager.getInstance().getAllDataMap(carbonTable);
+  if (!datamaps.isEmpty()) {
+for (TableDataMap dataMap : datamaps) {
+  DataMapFactory factoryClass = DataMapStoreManager.getInstance()
+  .getDataMapFactoryClass(dataMap.getDataMapSchema());
+  isSupported = factoryClass.dataMapFeatureScope(feature);
--- End diff --

rename `dataMapFeatureScope` to `isOperationAllow`


---

[GitHub] carbondata pull request #2202: [wip]change datamap factory interface to chec...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2202#discussion_r183230534
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java ---
@@ -2942,5 +2947,31 @@ public static String 
getBlockId(AbsoluteTableIdentifier identifier, String fileP
 return blockId;
   }
 
+  /**
+   * methods returns whether the feature is supported for the 
corresponding datamap or not
+   * @param carbonTable
+   * @param feature
+   * @return
+   */
+  public static boolean validateFeatureForDatamap(CarbonTable carbonTable, 
FeaturesList feature) {
+boolean isSupported = false;
+try {
+  List datamaps = 
DataMapStoreManager.getInstance().getAllDataMap(carbonTable);
+  if (!datamaps.isEmpty()) {
+for (TableDataMap dataMap : datamaps) {
+  DataMapFactory factoryClass = DataMapStoreManager.getInstance()
--- End diff --

I suggest to change `DataMapFactory` class name to `IndexDataMap`


---

[GitHub] carbondata pull request #2202: [wip]change datamap factory interface to chec...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2202#discussion_r183230516
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java ---
@@ -2942,5 +2947,31 @@ public static String 
getBlockId(AbsoluteTableIdentifier identifier, String fileP
 return blockId;
   }
 
+  /**
+   * methods returns whether the feature is supported for the 
corresponding datamap or not
+   * @param carbonTable
+   * @param feature
+   * @return
+   */
+  public static boolean validateFeatureForDatamap(CarbonTable carbonTable, 
FeaturesList feature) {
--- End diff --

move this function to CarbonTable


---

[GitHub] carbondata issue #2190: [CARBONDATA-2359] Support applicable load options an...

[ajantha-bhat]

Github user ajantha-bhat commented on the issue:

https://github.com/apache/carbondata/pull/2190
  
Retest this please


---

[GitHub] carbondata pull request #2202: [wip]change datamap factory interface to chec...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2202#discussion_r183230388
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/features/FeaturesList.java ---
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.features;
+
+public enum FeaturesList {
--- End diff --

change to `TableCommand`


---

[GitHub] carbondata pull request #2206: [CARBONDATA-2376] Improve Lucene datamap perf...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2206#discussion_r183230325
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/datamap/Segment.java ---
@@ -39,6 +41,11 @@
 
   private String segmentFileName;
 
+  /**
+   * List of tasks which are already got filtered through CG index 
operation.
+   */
+  private Set filteredTaskNames = new HashSet<>();
--- End diff --

Instead of `taskName`, can we give a more formal one. I can suggest two: 
`indexShardName`, `segmentIndexName`


---