[jira] [Updated] (SPARK-38881) PySpark Kinesis Streaming should expose metricsLevel CloudWatch config that is already supported in the Scala/Java APIs

[ https://issues.apache.org/jira/browse/SPARK-38881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Khaitman updated SPARK-38881: -- Description: This relates to https://issues.apache.org/jira/browse/SPARK-27420 which was

[jira] [Updated] (SPARK-38881) PySpark Kinesis Streaming should expose metricsLevel CloudWatch config that is already supported in the Scala/Java APIs

[ https://issues.apache.org/jira/browse/SPARK-38881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Khaitman updated SPARK-38881: -- Description: This relates to https://issues.apache.org/jira/browse/SPARK-27420 which was

[jira] [Created] (SPARK-38881) PySpark Kinesis Streaming should expose metricsLevel CloudWatch config that is already supported in the Scala/Java APIs

Mark Khaitman created SPARK-38881: - Summary: PySpark Kinesis Streaming should expose metricsLevel CloudWatch config that is already supported in the Scala/Java APIs Key: SPARK-38881 URL:

[jira] [Created] (SPARK-10702) Dynamic Allocation in Standalone Breaking Parallelism

Mark Khaitman created SPARK-10702: - Summary: Dynamic Allocation in Standalone Breaking Parallelism Key: SPARK-10702 URL: https://issues.apache.org/jira/browse/SPARK-10702 Project: Spark

[jira] [Updated] (SPARK-5782) Python Worker / Pyspark Daemon Memory Issue

[ https://issues.apache.org/jira/browse/SPARK-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Khaitman updated SPARK-5782: - External issue URL: https://github.com/apache/spark/pull/1977 Python Worker / Pyspark Daemon

[jira] [Commented] (SPARK-5782) Python Worker / Pyspark Daemon Memory Issue

[ https://issues.apache.org/jira/browse/SPARK-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371394#comment-14371394 ] Mark Khaitman commented on SPARK-5782: -- I didn't think this would be an extreme

[jira] [Comment Edited] (SPARK-5782) Python Worker / Pyspark Daemon Memory Issue

[ https://issues.apache.org/jira/browse/SPARK-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371533#comment-14371533 ] Mark Khaitman edited comment on SPARK-5782 at 3/20/15 4:10 PM:

[jira] [Commented] (SPARK-5782) Python Worker / Pyspark Daemon Memory Issue

[ https://issues.apache.org/jira/browse/SPARK-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371533#comment-14371533 ] Mark Khaitman commented on SPARK-5782: -- True, there would be billions of elements in

[jira] [Commented] (SPARK-5782) Python Worker / Pyspark Daemon Memory Issue

[ https://issues.apache.org/jira/browse/SPARK-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372039#comment-14372039 ] Mark Khaitman commented on SPARK-5782: -- This might also be a problem specific to

[jira] [Comment Edited] (SPARK-5782) Python Worker / Pyspark Daemon Memory Issue

[ https://issues.apache.org/jira/browse/SPARK-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371927#comment-14371927 ] Mark Khaitman edited comment on SPARK-5782 at 3/20/15 7:33 PM:

[jira] [Commented] (SPARK-5782) Python Worker / Pyspark Daemon Memory Issue

[ https://issues.apache.org/jira/browse/SPARK-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371927#comment-14371927 ] Mark Khaitman commented on SPARK-5782: -- Interesting scala equivalent. I ran it and

[jira] [Commented] (SPARK-4105) FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14370613#comment-14370613 ] Mark Khaitman commented on SPARK-4105: -- Can also confirm it's happened once so far

[jira] [Updated] (SPARK-5782) Python Worker / Pyspark Daemon Memory Issue

[ https://issues.apache.org/jira/browse/SPARK-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Khaitman updated SPARK-5782: - Priority: Blocker (was: Critical) Python Worker / Pyspark Daemon Memory Issue

[jira] [Commented] (SPARK-5782) Python Worker / Pyspark Daemon Memory Issue

[ https://issues.apache.org/jira/browse/SPARK-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363894#comment-14363894 ] Mark Khaitman commented on SPARK-5782: -- I've upped this JIRA ticket to blocker since

[jira] [Created] (SPARK-5899) Viewing specific stage information which contains thousands of tasks will freak out the driver and CPU cores from where it runs

Mark Khaitman created SPARK-5899: Summary: Viewing specific stage information which contains thousands of tasks will freak out the driver and CPU cores from where it runs Key: SPARK-5899 URL:

[jira] [Commented] (SPARK-5782) Python Worker / Pyspark Daemon Memory Issue

[ https://issues.apache.org/jira/browse/SPARK-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320309#comment-14320309 ] Mark Khaitman commented on SPARK-5782: -- Would it make sense to instead make the

[jira] [Updated] (SPARK-5782) Python Worker / Pyspark Daemon Memory Issue

[ https://issues.apache.org/jira/browse/SPARK-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Khaitman updated SPARK-5782: - Priority: Critical (was: Major) Python Worker / Pyspark Daemon Memory Issue

[jira] [Issue Comment Deleted] (SPARK-5782) Python Worker / Pyspark Daemon Memory Issue

[ https://issues.apache.org/jira/browse/SPARK-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Khaitman updated SPARK-5782: - Comment: was deleted (was: Would it make sense to instead make the _next_limit return the MIN of

[jira] [Created] (SPARK-5782) Python Worker / Pyspark Daemon Memory Issue

Mark Khaitman created SPARK-5782: Summary: Python Worker / Pyspark Daemon Memory Issue Key: SPARK-5782 URL: https://issues.apache.org/jira/browse/SPARK-5782 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-5782) Python Worker / Pyspark Daemon Memory Issue

[ https://issues.apache.org/jira/browse/SPARK-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Khaitman updated SPARK-5782: - Description: I'm including the Shuffle component on this, as a brief scan through the code

[jira] [Updated] (SPARK-5782) Python Worker / Pyspark Daemon Memory Issue

[ https://issues.apache.org/jira/browse/SPARK-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Khaitman updated SPARK-5782: - Description: I'm including the Shuffle component on this, as a brief scan through the code

[jira] [Commented] (SPARK-4105) FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313652#comment-14313652 ] Mark Khaitman commented on SPARK-4105: -- We're running 1.2.1-rc2 on our cluster and

[jira] [Commented] (SPARK-5395) Large number of Python workers causing resource depletion

[ https://issues.apache.org/jira/browse/SPARK-5395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14295416#comment-14295416 ] Mark Khaitman commented on SPARK-5395: -- Thanks! Hopefully that tackles the same

[jira] [Commented] (SPARK-5395) Large number of Python workers causing resource depletion

[ https://issues.apache.org/jira/browse/SPARK-5395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293653#comment-14293653 ] Mark Khaitman commented on SPARK-5395: -- Actually I think I know why this happens...

[jira] [Comment Edited] (SPARK-5395) Large number of Python workers causing resource depletion

[ https://issues.apache.org/jira/browse/SPARK-5395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293653#comment-14293653 ] Mark Khaitman edited comment on SPARK-5395 at 1/27/15 4:26 PM:

[jira] [Commented] (SPARK-5395) Large number of Python workers causing resource depletion

[ https://issues.apache.org/jira/browse/SPARK-5395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292121#comment-14292121 ] Mark Khaitman commented on SPARK-5395: -- Having the same issue in standalone

[jira] [Commented] (SPARK-5395) Large number of Python workers causing resource depletion

[ https://issues.apache.org/jira/browse/SPARK-5395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292769#comment-14292769 ] Mark Khaitman commented on SPARK-5395: -- This may prove to be useful... I'm watching

[jira] [Commented] (SPARK-5395) Large number of Python workers causing resource depletion

[ https://issues.apache.org/jira/browse/SPARK-5395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292668#comment-14292668 ] Mark Khaitman commented on SPARK-5395: -- [~skrasser], I actually only managed to have

[jira] [Comment Edited] (SPARK-5395) Large number of Python workers causing resource depletion

[ https://issues.apache.org/jira/browse/SPARK-5395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292668#comment-14292668 ] Mark Khaitman edited comment on SPARK-5395 at 1/26/15 11:57 PM: