[jira] [Commented] (SPARK-13796) Lock release errors occur frequently in executor logs
[ https://issues.apache.org/jira/browse/SPARK-13796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15217100#comment-15217100 ] Apache Spark commented on SPARK-13796: -- User 'nishkamravi2' has created a pull request for this issue: https://github.com/apache/spark/pull/12052 > Lock release errors occur frequently in executor logs > - > > Key: SPARK-13796 > URL: https://issues.apache.org/jira/browse/SPARK-13796 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.0 >Reporter: Nishkam Ravi >Priority: Minor > > Executor logs contain a lot of these error messages (irrespective of the > workload): > 16/03/08 17:53:07 ERROR executor.Executor: 1 block locks were not released by > TID = 1119 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13796) Lock release errors occur frequently in executor logs
[
https://issues.apache.org/jira/browse/SPARK-13796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208692#comment-15208692
]
Josh Rosen commented on SPARK-13796:
We should probably downgrade the log message's severity to WARNING because
there are currently some non-error cases where locks won't be released before
the end of the task (such as the {{take()}} operation's per-partition limits,
which partially consume an iterator but may still leave remaining elements
without performing a close() or dispose() operation).
> Lock release errors occur frequently in executor logs
> -
>
> Key: SPARK-13796
> URL: https://issues.apache.org/jira/browse/SPARK-13796
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.0.0
>Reporter: Nishkam Ravi
>Priority: Minor
>
> Executor logs contain a lot of these error messages (irrespective of the
> workload):
> 16/03/08 17:53:07 ERROR executor.Executor: 1 block locks were not released by
> TID = 1119
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13796) Lock release errors occur frequently in executor logs
[ https://issues.apache.org/jira/browse/SPARK-13796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208016#comment-15208016 ] Ernest commented on SPARK-13796: I find this ERROR is still exist in executor's log. This error can be reproduced by running _LiveJournalPageRank_ example. By calling BlockManager.getOrElseUpdate, as mentioned in https://github.com/apache/spark/blob/f2e855fba8eb73475cf312cdf880c1297d4323bb/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L660 , both of _doPut_ and _getLocalValues_ acquired a read lock, but only released one at line 661. The remaining lock should be release by _CompletionIterator_ located in https://github.com/apache/spark/blob/f2e855fba8eb73475cf312cdf880c1297d4323bb/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L427 after successfully iterates through all the elements, however, this do not always work. At the last of task, _releaseAllLocksForTask_ is called, and blocks that still have read lock are added to this method's result. And lead to lots of this kind of error in executor. > Lock release errors occur frequently in executor logs > - > > Key: SPARK-13796 > URL: https://issues.apache.org/jira/browse/SPARK-13796 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.0 >Reporter: Nishkam Ravi >Priority: Minor > > Executor logs contain a lot of these error messages (irrespective of the > workload): > 16/03/08 17:53:07 ERROR executor.Executor: 1 block locks were not released by > TID = 1119 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13796) Lock release errors occur frequently in executor logs
[ https://issues.apache.org/jira/browse/SPARK-13796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194171#comment-15194171 ] Nishkam Ravi commented on SPARK-13796: -- Looks like this issue has been fixed by a recent commit which adds the missing releaseLock (among other changes): https://github.com/apache/spark/commit/81d48532d954a8aea28d7e1fb3aa32a78c708b63 Closing the JIRA. Thanks [~joshrosen] > Lock release errors occur frequently in executor logs > - > > Key: SPARK-13796 > URL: https://issues.apache.org/jira/browse/SPARK-13796 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.0 >Reporter: Nishkam Ravi >Priority: Minor > > Executor logs contain a lot of these error messages (irrespective of the > workload): > 16/03/08 17:53:07 ERROR executor.Executor: 1 block locks were not released by > TID = 1119 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13796) Lock release errors occur frequently in executor logs
[ https://issues.apache.org/jira/browse/SPARK-13796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189957#comment-15189957 ] Nishkam Ravi commented on SPARK-13796: -- Would it make sense to change the condition to check for unreleased locks > 1 instead of 0 or redirect the error message to logWarning? > Lock release errors occur frequently in executor logs > - > > Key: SPARK-13796 > URL: https://issues.apache.org/jira/browse/SPARK-13796 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.0 >Reporter: Nishkam Ravi >Priority: Minor > > Executor logs contain a lot of these error messages (irrespective of the > workload): > 16/03/08 17:53:07 ERROR executor.Executor: 1 block locks were not released by > TID = 1119 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13796) Lock release errors occur frequently in executor logs
[ https://issues.apache.org/jira/browse/SPARK-13796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189939#comment-15189939 ] Josh Rosen commented on SPARK-13796: This is likely to be a spurious warning due to things like limit() not releasing locks early, so it's not a correctness issue per se. > Lock release errors occur frequently in executor logs > - > > Key: SPARK-13796 > URL: https://issues.apache.org/jira/browse/SPARK-13796 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.0 >Reporter: Nishkam Ravi >Priority: Minor > > Executor logs contain a lot of these error messages (irrespective of the > workload): > 16/03/08 17:53:07 ERROR executor.Executor: 1 block locks were not released by > TID = 1119 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13796) Lock release errors occur frequently in executor logs
[ https://issues.apache.org/jira/browse/SPARK-13796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189930#comment-15189930 ] Nishkam Ravi commented on SPARK-13796: -- [~joshrosen] Retested with latest master. FWIW I haven't seen any error messages with > 1 block locks not released. > Lock release errors occur frequently in executor logs > - > > Key: SPARK-13796 > URL: https://issues.apache.org/jira/browse/SPARK-13796 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.0 >Reporter: Nishkam Ravi >Priority: Minor > > Executor logs contain a lot of these error messages (irrespective of the > workload): > 16/03/08 17:53:07 ERROR executor.Executor: 1 block locks were not released by > TID = 1119 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13796) Lock release errors occur frequently in executor logs
[ https://issues.apache.org/jira/browse/SPARK-13796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189919#comment-15189919 ] Nishkam Ravi commented on SPARK-13796: -- Hey [~srowen] this doesn't seem to impact functionality. I checked with Josh offline and we decided to open a JIRA anyways. Can be reproduced with SparkPageRank example among others. If we conclude this isn't a real problem, the error message could be redirected to logWarning. > Lock release errors occur frequently in executor logs > - > > Key: SPARK-13796 > URL: https://issues.apache.org/jira/browse/SPARK-13796 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.0 >Reporter: Nishkam Ravi >Priority: Minor > > Executor logs contain a lot of these error messages (irrespective of the > workload): > 16/03/08 17:53:07 ERROR executor.Executor: 1 block locks were not released by > TID = 1119 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13796) Lock release errors occur frequently in executor logs
[ https://issues.apache.org/jira/browse/SPARK-13796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189389#comment-15189389 ] Benjamin Herta commented on SPARK-13796: We have been seeing this as well, running with trunk as of this morning. It doesn't appear to affect our results, so it may be just log noise, or maybe our testing isn't thorough enough to see the problem. One easy way to reproduce it is to run the org.apache.spark.examples.mllib.LBFGSExample example program. > Lock release errors occur frequently in executor logs > - > > Key: SPARK-13796 > URL: https://issues.apache.org/jira/browse/SPARK-13796 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.0 >Reporter: Nishkam Ravi >Priority: Minor > > Executor logs contain a lot of these error messages (irrespective of the > workload): > 16/03/08 17:53:07 ERROR executor.Executor: 1 block locks were not released by > TID = 1119 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13796) Lock release errors occur frequently in executor logs
[ https://issues.apache.org/jira/browse/SPARK-13796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189197#comment-15189197 ] Sean Owen commented on SPARK-13796: --- [~joshrosen] does that seem like a problem? > Lock release errors occur frequently in executor logs > - > > Key: SPARK-13796 > URL: https://issues.apache.org/jira/browse/SPARK-13796 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.0 >Reporter: Nishkam Ravi >Priority: Minor > > Executor logs contain a lot of these error messages (irrespective of the > workload): > 16/03/08 17:53:07 ERROR executor.Executor: 1 block locks were not released by > TID = 1119 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13796) Lock release errors occur frequently in executor logs
[ https://issues.apache.org/jira/browse/SPARK-13796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188762#comment-15188762 ] Nishkam Ravi commented on SPARK-13796: -- Running with master from March 7th (e52e597db48d069b98c1d404b221d3365f38fbb8) Error introduced by 633d63a48ad98754dc7c56f9ac150fc2aa4e42c5 > Lock release errors occur frequently in executor logs > - > > Key: SPARK-13796 > URL: https://issues.apache.org/jira/browse/SPARK-13796 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.0 >Reporter: Nishkam Ravi > > Executor logs contain a lot of these error messages (irrespective of the > workload): > 16/03/08 17:53:07 ERROR executor.Executor: 1 block locks were not released by > TID = 1119 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
