[jira] [Commented] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely
[
https://issues.apache.org/jira/browse/CARBONDATA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496519#comment-15496519
]
ASF GitHub Bot commented on CARBONDATA-244:
---
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/156#discussion_r79187047
--- Diff:
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
---
@@ -1077,19 +1077,38 @@ object CarbonDataRDDFactory extends Logging {
val segmentStatusManager = new
SegmentStatusManager(table.getAbsoluteTableIdentifier)
val details = segmentStatusManager
.readLoadMetadata(loadMetadataFilePath)
+ val carbonTableStatusLock = CarbonLockFactory
+
.getCarbonLockObj(table.getAbsoluteTableIdentifier.getCarbonTableIdentifier,
+ LockUsage.TABLE_STATUS_LOCK)
// Delete marked loads
val isUpdationRequired = DeleteLoadFolders
.deleteLoadFoldersFromFileSystem(carbonLoadModel,
hdfsStoreLocation,
partitioner.partitionCount, isForceDeletion, details)
if (isUpdationRequired) {
+try {
// Update load metadate file after cleaning deleted nodes
-CarbonLoaderUtil.writeLoadMetadata(
- carbonLoadModel.getCarbonDataLoadSchema,
- carbonLoadModel.getDatabaseName,
- carbonLoadModel.getTableName, details.toList.asJava
-)
+if (carbonTableStatusLock.lockWithRetries()) {
+ logger.info("Table status lock has been successfully acquired.")
+ CarbonLoaderUtil.writeLoadMetadata(
+carbonLoadModel.getCarbonDataLoadSchema,
+carbonLoadModel.getDatabaseName,
+carbonLoadModel.getTableName, details.toList.asJava
+ )
+}
+else {
+ val errorMsg = "Clean files request is failed for " +
carbonLoadModel.getDatabaseName +
+ "." + carbonLoadModel.getTableName +
+ ". Not able to acquire the table status lock."
--- End diff --
also mention that "due to other operation running in the background ",
this message will give more clear picture to user about the task failure
> Load and delete segment by id queries giving inconsistent results when we
> execute parallely
> ---
>
> Key: CARBONDATA-244
> URL: https://issues.apache.org/jira/browse/CARBONDATA-244
> Project: CarbonData
> Issue Type: Bug
>Reporter: Manohar Vanam
>Assignee: Manohar Vanam
>
> Delete segment by id behavior is inconsistent when we Execute load and
> delete segment by id queries parallely,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely
[
https://issues.apache.org/jira/browse/CARBONDATA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496744#comment-15496744
]
ASF GitHub Bot commented on CARBONDATA-244:
---
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/156#discussion_r79204842
--- Diff:
processing/src/main/java/org/apache/carbondata/lcm/status/SegmentStatusManager.java
---
@@ -306,13 +330,18 @@ private Integer compareDateValues(Long loadValue,
Long userValue) {
* @return
*/
public List updateDeletionStatus(String loadDate, String
tableFolderPath,
- Long loadStartTime) {
-ICarbonLock carbonLock = CarbonLockFactory
-
.getCarbonLockObj(absoluteTableIdentifier.getCarbonTableIdentifier(),
-LockUsage.METADATA_LOCK);
+ Long loadStartTime) throws Exception {
+CarbonTableIdentifier carbonTableIdentifier =
+absoluteTableIdentifier.getCarbonTableIdentifier();
+ICarbonLock carbonMetadataLock =
+CarbonLockFactory.getCarbonLockObj(carbonTableIdentifier,
LockUsage.METADATA_LOCK);
+ICarbonLock carbonTableStatusLock =
+CarbonLockFactory.getCarbonLockObj(carbonTableIdentifier,
LockUsage.TABLE_STATUS_LOCK);
+String tableDetails =
+carbonTableIdentifier.getDatabaseName() + "." +
carbonTableIdentifier.getTableName();
List invalidLoadTimestamps = new ArrayList(0);
try {
- if (carbonLock.lockWithRetries()) {
+ if (carbonMetadataLock.lockWithRetries()) {
--- End diff --
make it as operational log
> Load and delete segment by id queries giving inconsistent results when we
> execute parallely
> ---
>
> Key: CARBONDATA-244
> URL: https://issues.apache.org/jira/browse/CARBONDATA-244
> Project: CarbonData
> Issue Type: Bug
>Reporter: Manohar Vanam
>Assignee: Manohar Vanam
>
> Delete segment by id behavior is inconsistent when we Execute load and
> delete segment by id queries parallely,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely
[
https://issues.apache.org/jira/browse/CARBONDATA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496915#comment-15496915
]
ASF GitHub Bot commented on CARBONDATA-244:
---
Github user ManoharVanam commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/156#discussion_r79217913
--- Diff:
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
---
@@ -1077,19 +1077,38 @@ object CarbonDataRDDFactory extends Logging {
val segmentStatusManager = new
SegmentStatusManager(table.getAbsoluteTableIdentifier)
val details = segmentStatusManager
.readLoadMetadata(loadMetadataFilePath)
+ val carbonTableStatusLock = CarbonLockFactory
+
.getCarbonLockObj(table.getAbsoluteTableIdentifier.getCarbonTableIdentifier,
+ LockUsage.TABLE_STATUS_LOCK)
// Delete marked loads
val isUpdationRequired = DeleteLoadFolders
.deleteLoadFoldersFromFileSystem(carbonLoadModel,
hdfsStoreLocation,
partitioner.partitionCount, isForceDeletion, details)
if (isUpdationRequired) {
+try {
// Update load metadate file after cleaning deleted nodes
-CarbonLoaderUtil.writeLoadMetadata(
- carbonLoadModel.getCarbonDataLoadSchema,
- carbonLoadModel.getDatabaseName,
- carbonLoadModel.getTableName, details.toList.asJava
-)
+if (carbonTableStatusLock.lockWithRetries()) {
+ logger.info("Table status lock has been successfully acquired.")
+ CarbonLoaderUtil.writeLoadMetadata(
+carbonLoadModel.getCarbonDataLoadSchema,
+carbonLoadModel.getDatabaseName,
+carbonLoadModel.getTableName, details.toList.asJava
+ )
+}
+else {
+ val errorMsg = "Clean files request is failed for " +
carbonLoadModel.getDatabaseName +
+ "." + carbonLoadModel.getTableName +
+ ". Not able to acquire the table status lock."
--- End diff --
ok
> Load and delete segment by id queries giving inconsistent results when we
> execute parallely
> ---
>
> Key: CARBONDATA-244
> URL: https://issues.apache.org/jira/browse/CARBONDATA-244
> Project: CarbonData
> Issue Type: Bug
>Reporter: Manohar Vanam
>Assignee: Manohar Vanam
>
> Delete segment by id behavior is inconsistent when we Execute load and
> delete segment by id queries parallely,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely
[
https://issues.apache.org/jira/browse/CARBONDATA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496970#comment-15496970
]
ASF GitHub Bot commented on CARBONDATA-244:
---
Github user ManoharVanam commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/156#discussion_r79222066
--- Diff:
processing/src/main/java/org/apache/carbondata/lcm/status/SegmentStatusManager.java
---
@@ -306,13 +330,18 @@ private Integer compareDateValues(Long loadValue,
Long userValue) {
* @return
*/
public List updateDeletionStatus(String loadDate, String
tableFolderPath,
- Long loadStartTime) {
-ICarbonLock carbonLock = CarbonLockFactory
-
.getCarbonLockObj(absoluteTableIdentifier.getCarbonTableIdentifier(),
-LockUsage.METADATA_LOCK);
+ Long loadStartTime) throws Exception {
+CarbonTableIdentifier carbonTableIdentifier =
+absoluteTableIdentifier.getCarbonTableIdentifier();
+ICarbonLock carbonMetadataLock =
+CarbonLockFactory.getCarbonLockObj(carbonTableIdentifier,
LockUsage.METADATA_LOCK);
+ICarbonLock carbonTableStatusLock =
+CarbonLockFactory.getCarbonLockObj(carbonTableIdentifier,
LockUsage.TABLE_STATUS_LOCK);
+String tableDetails =
+carbonTableIdentifier.getDatabaseName() + "." +
carbonTableIdentifier.getTableName();
List invalidLoadTimestamps = new ArrayList(0);
try {
- if (carbonLock.lockWithRetries()) {
+ if (carbonMetadataLock.lockWithRetries()) {
--- End diff --
ok
> Load and delete segment by id queries giving inconsistent results when we
> execute parallely
> ---
>
> Key: CARBONDATA-244
> URL: https://issues.apache.org/jira/browse/CARBONDATA-244
> Project: CarbonData
> Issue Type: Bug
>Reporter: Manohar Vanam
>Assignee: Manohar Vanam
>
> Delete segment by id behavior is inconsistent when we Execute load and
> delete segment by id queries parallely,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely
[
https://issues.apache.org/jira/browse/CARBONDATA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499620#comment-15499620
]
ASF GitHub Bot commented on CARBONDATA-244:
---
Github user akashrn5 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/156#discussion_r79291255
--- Diff:
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
---
@@ -,25 +1131,32 @@ object CarbonDataRDDFactory extends Logging {
val table =
org.apache.carbondata.core.carbon.metadata.CarbonMetadata.getInstance
.getCarbonTable(carbonLoadModel.getDatabaseName + "_" +
carbonLoadModel.getTableName)
val metaDataPath: String = table.getMetaDataFilepath
-val carbonLock = CarbonLockFactory
+val carbonCleanFilesLock = CarbonLockFactory
.getCarbonLockObj(table.getAbsoluteTableIdentifier.getCarbonTableIdentifier,
-LockUsage.METADATA_LOCK
+LockUsage.CLEAN_FILES_LOCK
)
try {
- if (carbonLock.lockWithRetries()) {
+ if (carbonCleanFilesLock.lockWithRetries()) {
+logger.info("Clean files lock has been successfully acquired.")
deleteLoadsAndUpdateMetadata(carbonLoadModel,
table,
partitioner,
hdfsStoreLocation,
isForceDeletion = true)
}
+ else {
+val errorMsg = "Clean files request is failed for " +
carbonLoadModel.getDatabaseName +
--- End diff --
"Clean file request is running in the background, please try after some
time". u[date the comment since already we know the operation
> Load and delete segment by id queries giving inconsistent results when we
> execute parallely
> ---
>
> Key: CARBONDATA-244
> URL: https://issues.apache.org/jira/browse/CARBONDATA-244
> Project: CarbonData
> Issue Type: Bug
>Reporter: Manohar Vanam
>Assignee: Manohar Vanam
>
> Delete segment by id behavior is inconsistent when we Execute load and
> delete segment by id queries parallely,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely
[ https://issues.apache.org/jira/browse/CARBONDATA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499624#comment-15499624 ] ASF GitHub Bot commented on CARBONDATA-244: --- Github user sujith71955 commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/156#discussion_r79291307 --- Diff: processing/src/main/java/org/apache/carbondata/lcm/locks/CarbonLockUtil.java --- @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.carbondata.lcm.locks; + +import org.apache.carbondata.common.logging.LogService; +import org.apache.carbondata.common.logging.LogServiceFactory; + +/** + * This is a singleton class for initialization of zookeeper client. --- End diff -- provide proper comment > Load and delete segment by id queries giving inconsistent results when we > execute parallely > --- > > Key: CARBONDATA-244 > URL: https://issues.apache.org/jira/browse/CARBONDATA-244 > Project: CarbonData > Issue Type: Bug >Reporter: Manohar Vanam >Assignee: Manohar Vanam > > Delete segment by id behavior is inconsistent when we Execute load and > delete segment by id queries parallely, -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely
[ https://issues.apache.org/jira/browse/CARBONDATA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499627#comment-15499627 ] ASF GitHub Bot commented on CARBONDATA-244: --- Github user sujith71955 commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/156#discussion_r79291346 --- Diff: processing/src/main/java/org/apache/carbondata/lcm/locks/LockUsage.java --- @@ -28,5 +28,7 @@ public static String COMPACTION_LOCK = "compaction.lock"; public static String SYSTEMLEVEL_COMPACTION_LOCK = "system_level_compaction.lock"; public static String TABLE_STATUS_LOCK = "tablestatus.lock"; + public static String DELETE_SEGMENT_LOCK = "delete_segment.lock"; --- End diff -- make it final/use enum > Load and delete segment by id queries giving inconsistent results when we > execute parallely > --- > > Key: CARBONDATA-244 > URL: https://issues.apache.org/jira/browse/CARBONDATA-244 > Project: CarbonData > Issue Type: Bug >Reporter: Manohar Vanam >Assignee: Manohar Vanam > > Delete segment by id behavior is inconsistent when we Execute load and > delete segment by id queries parallely, -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely
[
https://issues.apache.org/jira/browse/CARBONDATA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499631#comment-15499631
]
ASF GitHub Bot commented on CARBONDATA-244:
---
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/156#discussion_r79291381
--- Diff:
processing/src/main/java/org/apache/carbondata/lcm/status/SegmentStatusManager.java
---
@@ -287,12 +307,18 @@ private Integer compareDateValues(Long loadValue,
Long userValue) {
}
} else {
-LOG.error("Unable to acquire the metadata lock");
+String errorMsg = "Delete segment by id is failed for " +
tableDetails
++ ". Not able to acquire the delete segment lock due to other
operation running "
--- End diff --
update message, provide operation name
> Load and delete segment by id queries giving inconsistent results when we
> execute parallely
> ---
>
> Key: CARBONDATA-244
> URL: https://issues.apache.org/jira/browse/CARBONDATA-244
> Project: CarbonData
> Issue Type: Bug
>Reporter: Manohar Vanam
>Assignee: Manohar Vanam
>
> Delete segment by id behavior is inconsistent when we Execute load and
> delete segment by id queries parallely,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely
[
https://issues.apache.org/jira/browse/CARBONDATA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499665#comment-15499665
]
ASF GitHub Bot commented on CARBONDATA-244:
---
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/156#discussion_r79291710
--- Diff:
processing/src/main/java/org/apache/carbondata/lcm/status/SegmentStatusManager.java
---
@@ -287,12 +307,18 @@ private Integer compareDateValues(Long loadValue,
Long userValue) {
}
} else {
-LOG.error("Unable to acquire the metadata lock");
+String errorMsg = "Delete segment by id is failed for " +
tableDetails
++ ". Not able to acquire the delete segment lock due to other
operation running "
--- End diff --
LGTM
> Load and delete segment by id queries giving inconsistent results when we
> execute parallely
> ---
>
> Key: CARBONDATA-244
> URL: https://issues.apache.org/jira/browse/CARBONDATA-244
> Project: CarbonData
> Issue Type: Bug
>Reporter: Manohar Vanam
>Assignee: Manohar Vanam
>
> Delete segment by id behavior is inconsistent when we Execute load and
> delete segment by id queries parallely,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-244) Load and delete segment by id queries giving inconsistent results when we execute parallely
[ https://issues.apache.org/jira/browse/CARBONDATA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499717#comment-15499717 ] ASF GitHub Bot commented on CARBONDATA-244: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/156 > Load and delete segment by id queries giving inconsistent results when we > execute parallely > --- > > Key: CARBONDATA-244 > URL: https://issues.apache.org/jira/browse/CARBONDATA-244 > Project: CarbonData > Issue Type: Bug >Reporter: Manohar Vanam >Assignee: Manohar Vanam > > Delete segment by id behavior is inconsistent when we Execute load and > delete segment by id queries parallely, -- This message was sent by Atlassian JIRA (v6.3.4#6332)
