[jira] [Commented] (CARBONDATA-234) wrong message is printed in the logs each time when the compaction is done.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496993#comment-15496993
 ] 

ASF GitHub Bot commented on CARBONDATA-234:
---

Github user ravikiran23 commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/149#discussion_r79223519
  
--- Diff: 
integration/spark/src/main/java/org/apache/carbondata/integration/spark/merger/CarbonCompactionUtil.java
 ---
@@ -210,9 +210,12 @@ public static boolean 
deleteCompactionRequiredFile(String metaFolderPath,
   if (FileFactory.isFileExist(statusFile, 
FileFactory.getFileType(statusFile))) {
--- End diff --

http://136.243.101.176:8080/job/ApacheCarbonManualPRBuilder/225/



> wrong message is printed in the logs each time when the compaction is done.
> ---
>
> Key: CARBONDATA-234
> URL: https://issues.apache.org/jira/browse/CARBONDATA-234
> Project: CarbonData
>  Issue Type: Bug
>Reporter: ravikiran
>
> wrong message is printed in the logs each time when the compaction is done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496915#comment-15496915
 ] 

ASF GitHub Bot commented on CARBONDATA-244:
---

Github user ManoharVanam commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/156#discussion_r79217913
  
--- Diff: 
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
 ---
@@ -1077,19 +1077,38 @@ object CarbonDataRDDFactory extends Logging {
   val segmentStatusManager = new 
SegmentStatusManager(table.getAbsoluteTableIdentifier)
   val details = segmentStatusManager
 .readLoadMetadata(loadMetadataFilePath)
+  val carbonTableStatusLock = CarbonLockFactory
+
.getCarbonLockObj(table.getAbsoluteTableIdentifier.getCarbonTableIdentifier,
+  LockUsage.TABLE_STATUS_LOCK)
 
   // Delete marked loads
   val isUpdationRequired = DeleteLoadFolders
 .deleteLoadFoldersFromFileSystem(carbonLoadModel, 
hdfsStoreLocation,
   partitioner.partitionCount, isForceDeletion, details)
 
   if (isUpdationRequired) {
+try {
 // Update load metadate file after cleaning deleted nodes
-CarbonLoaderUtil.writeLoadMetadata(
-  carbonLoadModel.getCarbonDataLoadSchema,
-  carbonLoadModel.getDatabaseName,
-  carbonLoadModel.getTableName, details.toList.asJava
-)
+if (carbonTableStatusLock.lockWithRetries()) {
+  logger.info("Table status lock has been successfully acquired.")
+  CarbonLoaderUtil.writeLoadMetadata(
+carbonLoadModel.getCarbonDataLoadSchema,
+carbonLoadModel.getDatabaseName,
+carbonLoadModel.getTableName, details.toList.asJava
+  )
+}
+else {
+  val errorMsg = "Clean files request is failed for " + 
carbonLoadModel.getDatabaseName +
+ "." + carbonLoadModel.getTableName +
+ ". Not able to acquire the table status lock."
--- End diff --

ok


> Load and delete segment by id queries giving inconsistent results when we 
> execute parallely
> ---
>
> Key: CARBONDATA-244
> URL: https://issues.apache.org/jira/browse/CARBONDATA-244
> Project: CarbonData
>  Issue Type: Bug
>Reporter: Manohar Vanam
>Assignee: Manohar Vanam
>
> Delete segment by id behavior is inconsistent when  we Execute load and 
> delete segment by id queries parallely,  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CARBONDATA-239) Failure of one compaction in queue should not affect the others.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496804#comment-15496804
 ] 

ASF GitHub Bot commented on CARBONDATA-239:
---

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/155#discussion_r79208835
  
--- Diff: 
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
 ---
@@ -590,16 +605,25 @@ object CarbonDataRDDFactory extends Logging {
 executor, sqlContext, kettleHomePath, storeLocation
   )
 }
+catch {
+  case e: Exception =>
+logger.error("Exception in compaction thread for table 
" + tableForCompaction
+  .carbonTable.getDatabaseName + "." +
+ tableForCompaction.carbonTableIdentifier
+   .getTableName)
+  // not handling the exception. only logging as this is 
not the table triggered
+  // by user.
+}
 finally {
-  // delete the compaction required file
+  // delete the compaction required file in case of 
failure or success also.
--- End diff --

If compact file deletion fails, add failed table to skipped list.


> Failure of one compaction in queue should not affect the others.
> 
>
> Key: CARBONDATA-239
> URL: https://issues.apache.org/jira/browse/CARBONDATA-239
> Project: CarbonData
>  Issue Type: Bug
>Reporter: ravikiran
>
> Failure of one compaction in queue should not affect the others.
> If a compaction is triggered by the user on table1 , and other requests will 
> go to queue.  and if the compaction is failed for table1 then the requests in 
> queue should continue and at the end the beeline will show the failure 
> message to the user.
> if any compaction gets failed for a table which is other than the user 
> requested table then the error in the beeline should not appear.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CARBONDATA-241) OOM error during query execution in long run

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496781#comment-15496781
 ] 

ASF GitHub Bot commented on CARBONDATA-241:
---

Github user kumarvishal09 commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/158#discussion_r79207294
  
--- Diff: 
integration/spark/src/main/scala/org/apache/spark/sql/CarbonDatasourceHadoopRelation.scala
 ---
@@ -20,20 +20,13 @@ package org.apache.spark.sql
 import java.text.SimpleDateFormat
 import java.util.Date
 
-import org.apache.carbondata.core.carbon.AbsoluteTableIdentifier
--- End diff --

done


> OOM error during query execution in long run
> 
>
> Key: CARBONDATA-241
> URL: https://issues.apache.org/jira/browse/CARBONDATA-241
> Project: CarbonData
>  Issue Type: Bug
>Reporter: kumar vishal
>Assignee: kumar vishal
>
> **Problem:** During long run query execution is taking more time and it is 
> throwing out of memory issue.
> **Reason**: In compaction we are compacting segments and each segment 
> metadata is loaded in memory. So after compaction compacted segments are 
> invalid but its meta data is not removed from memory because of this 
> duplicate metadata is pile up and it is taking more memory and after few days 
> query exeution is throwing OOM
> **Solution**: Need to remove invalid blocks from memory
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CARBONDATA-242) NOT IN with Null filter results are not compatible With Hive

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496747#comment-15496747
 ] 

ASF GitHub Bot commented on CARBONDATA-242:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-carbondata/pull/159


> NOT IN with Null filter results are not compatible With Hive
> 
>
> Key: CARBONDATA-242
> URL: https://issues.apache.org/jira/browse/CARBONDATA-242
> Project: CarbonData
>  Issue Type: Bug
>Reporter: Sujith
>Assignee: Sujith
>Priority: Minor
>
> When user provides Null member inside NOT IN filter condition the resultset 
> is not compatible with hive result.
> Eg: 
> select country from t3 where country not in ('china','france',null) group by 
> country



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496744#comment-15496744
 ] 

ASF GitHub Bot commented on CARBONDATA-244:
---

Github user sujith71955 commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/156#discussion_r79204842
  
--- Diff: 
processing/src/main/java/org/apache/carbondata/lcm/status/SegmentStatusManager.java
 ---
@@ -306,13 +330,18 @@ private Integer compareDateValues(Long loadValue, 
Long userValue) {
* @return
*/
   public List updateDeletionStatus(String loadDate, String 
tableFolderPath,
-  Long loadStartTime) {
-ICarbonLock carbonLock = CarbonLockFactory
-
.getCarbonLockObj(absoluteTableIdentifier.getCarbonTableIdentifier(),
-LockUsage.METADATA_LOCK);
+  Long loadStartTime) throws Exception {
+CarbonTableIdentifier carbonTableIdentifier =
+absoluteTableIdentifier.getCarbonTableIdentifier();
+ICarbonLock carbonMetadataLock =
+CarbonLockFactory.getCarbonLockObj(carbonTableIdentifier, 
LockUsage.METADATA_LOCK);
+ICarbonLock carbonTableStatusLock =
+CarbonLockFactory.getCarbonLockObj(carbonTableIdentifier, 
LockUsage.TABLE_STATUS_LOCK);
+String tableDetails =
+carbonTableIdentifier.getDatabaseName() + "." + 
carbonTableIdentifier.getTableName();
 List invalidLoadTimestamps = new ArrayList(0);
 try {
-  if (carbonLock.lockWithRetries()) {
+  if (carbonMetadataLock.lockWithRetries()) {
--- End diff --

make it as operational log


> Load and delete segment by id queries giving inconsistent results when we 
> execute parallely
> ---
>
> Key: CARBONDATA-244
> URL: https://issues.apache.org/jira/browse/CARBONDATA-244
> Project: CarbonData
>  Issue Type: Bug
>Reporter: Manohar Vanam
>Assignee: Manohar Vanam
>
> Delete segment by id behavior is inconsistent when  we Execute load and 
> delete segment by id queries parallely,  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496519#comment-15496519
 ] 

ASF GitHub Bot commented on CARBONDATA-244:
---

Github user sujith71955 commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/156#discussion_r79187047
  
--- Diff: 
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
 ---
@@ -1077,19 +1077,38 @@ object CarbonDataRDDFactory extends Logging {
   val segmentStatusManager = new 
SegmentStatusManager(table.getAbsoluteTableIdentifier)
   val details = segmentStatusManager
 .readLoadMetadata(loadMetadataFilePath)
+  val carbonTableStatusLock = CarbonLockFactory
+
.getCarbonLockObj(table.getAbsoluteTableIdentifier.getCarbonTableIdentifier,
+  LockUsage.TABLE_STATUS_LOCK)
 
   // Delete marked loads
   val isUpdationRequired = DeleteLoadFolders
 .deleteLoadFoldersFromFileSystem(carbonLoadModel, 
hdfsStoreLocation,
   partitioner.partitionCount, isForceDeletion, details)
 
   if (isUpdationRequired) {
+try {
 // Update load metadate file after cleaning deleted nodes
-CarbonLoaderUtil.writeLoadMetadata(
-  carbonLoadModel.getCarbonDataLoadSchema,
-  carbonLoadModel.getDatabaseName,
-  carbonLoadModel.getTableName, details.toList.asJava
-)
+if (carbonTableStatusLock.lockWithRetries()) {
+  logger.info("Table status lock has been successfully acquired.")
+  CarbonLoaderUtil.writeLoadMetadata(
+carbonLoadModel.getCarbonDataLoadSchema,
+carbonLoadModel.getDatabaseName,
+carbonLoadModel.getTableName, details.toList.asJava
+  )
+}
+else {
+  val errorMsg = "Clean files request is failed for " + 
carbonLoadModel.getDatabaseName +
+ "." + carbonLoadModel.getTableName +
+ ". Not able to acquire the table status lock."
--- End diff --

also mention that "due to  other operation running in the  background ", 
this message will give more clear picture to user about the task failure


> Load and delete segment by id queries giving inconsistent results when we 
> execute parallely
> ---
>
> Key: CARBONDATA-244
> URL: https://issues.apache.org/jira/browse/CARBONDATA-244
> Project: CarbonData
>  Issue Type: Bug
>Reporter: Manohar Vanam
>Assignee: Manohar Vanam
>
> Delete segment by id behavior is inconsistent when  we Execute load and 
> delete segment by id queries parallely,  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CARBONDATA-224) Fixed data mismatch issue in case of Dictionary Exclude column for Numeric data type

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496504#comment-15496504
 ] 

ASF GitHub Bot commented on CARBONDATA-224:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-carbondata/pull/139


> Fixed data mismatch issue in case of Dictionary Exclude column for Numeric 
> data type
> 
>
> Key: CARBONDATA-224
> URL: https://issues.apache.org/jira/browse/CARBONDATA-224
> Project: CarbonData
>  Issue Type: Bug
>Reporter: kumar vishal
>Assignee: kumar vishal
>
> Problem: In case of greater than query on dictionary exclude column of for 
> numeric data type
> This is because data is sorted based on string because of this if data is 
> 1,10,2,3 , data will be sorted like 1,10,2,3 but if we search greater than 3 
> then while applying min max will return false as last value is 3
> Solution:we need to sort based on actual data type for this we should have 
> chain comparator based on data type while loading the data, currently 
> disabling DictionaryExclude column for numeric data type and will throw 
> exception. Will raise jira issue to for sorting the based on actual data type



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CARBONDATA-234) wrong message is printed in the logs each time when the compaction is done.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496466#comment-15496466
 ] 

ASF GitHub Bot commented on CARBONDATA-234:
---

Github user kumarvishal09 commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/149#discussion_r79180388
  
--- Diff: 
integration/spark/src/main/java/org/apache/carbondata/integration/spark/merger/CarbonCompactionUtil.java
 ---
@@ -210,9 +210,12 @@ public static boolean 
deleteCompactionRequiredFile(String metaFolderPath,
   if (FileFactory.isFileExist(statusFile, 
FileFactory.getFileType(statusFile))) {
--- End diff --

change statusFile to compactionRequestFile


> wrong message is printed in the logs each time when the compaction is done.
> ---
>
> Key: CARBONDATA-234
> URL: https://issues.apache.org/jira/browse/CARBONDATA-234
> Project: CarbonData
>  Issue Type: Bug
>Reporter: ravikiran
>
> wrong message is printed in the logs each time when the compaction is done.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CARBONDATA-242) NOT IN with Null filter results are not compatible With Hive

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496181#comment-15496181
 ] 

ASF GitHub Bot commented on CARBONDATA-242:
---

GitHub user sujith71955 opened a pull request:

https://github.com/apache/incubator-carbondata/pull/159

[Issue Number] CARBONDATA-242

[Problem] Filter result was not compatible with hive result when a null 
filter member is present in not in
filter model, as per hive no result shall be return if a NOT IN filter 
model has null object for comparison.
eg: select country from t3 where country not in (null,'china','france') 
group by country
[Description] When user provides Null member inside NOT IN filter condition 
the resultset is not compatible with hive result.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sujith71955/incubator-carbondata 
master_StartsWithIssue

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-carbondata/pull/159.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #159


commit 8709af0cebc07e8ac25f40f4a31dfdc95b4eb5e6
Author: sujith71955 
Date:   2016-09-15T09:23:04Z

[Issue Number] CARBONDATA-242
[Description] When user provides Null member inside NOT IN filter condition 
the resultset is not compatible with hive result




> NOT IN with Null filter results are not compatible With Hive
> 
>
> Key: CARBONDATA-242
> URL: https://issues.apache.org/jira/browse/CARBONDATA-242
> Project: CarbonData
>  Issue Type: Bug
>Reporter: Sujith
>Assignee: Sujith
>Priority: Minor
>
> When user provides Null member inside NOT IN filter condition the resultset 
> is not compatible with hive result.
> Eg: 
> select country from t3 where country not in ('china','france',null) group by 
> country



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CARBONDATA-104) To support varchar datatype

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495942#comment-15495942
 ] 

ASF GitHub Bot commented on CARBONDATA-104:
---

Github user Zhangshunyu closed the pull request at:

https://github.com/apache/incubator-carbondata/pull/52


> To support varchar datatype
> ---
>
> Key: CARBONDATA-104
> URL: https://issues.apache.org/jira/browse/CARBONDATA-104
> Project: CarbonData
>  Issue Type: New Feature
>Reporter: zhangshunyu
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely

[Manohar Vanam (JIRA)]

Manohar Vanam created CARBONDATA-244:


 Summary: Load and delete segment by id queries giving inconsistent 
results when we execute parallely
 Key: CARBONDATA-244
 URL: https://issues.apache.org/jira/browse/CARBONDATA-244
 Project: CarbonData
  Issue Type: Bug
Reporter: Manohar Vanam
Assignee: Manohar Vanam


Delete segment by id behavior is inconsistent when  we Execute load and delete 
segment by id queries parallely,  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)