[jira] [Commented] (SPARK-7736) Exception not failing Python applications (in yarn cluster mode)

[ https://issues.apache.org/jira/browse/SPARK-7736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952263#comment-14952263 ] Shay Rojansky commented on SPARK-7736: -- Have just tested this with Spark 1.5.1 on Yarn 2.7.1 and the

[jira] [Commented] (SPARK-8119) HeartbeatReceiver should not adjust application executor resources

[ https://issues.apache.org/jira/browse/SPARK-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630799#comment-14630799 ] Shay Rojansky commented on SPARK-8119: -- Thanks Andrew! HeartbeatReceiver should not

[jira] [Commented] (SPARK-8119) HeartbeatReceiver should not adjust application executor resources

[ https://issues.apache.org/jira/browse/SPARK-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630380#comment-14630380 ] Shay Rojansky commented on SPARK-8119: -- Will this really not be fixed before 1.5?

[jira] [Commented] (SPARK-7736) Exception not failing Python applications (in yarn cluster mode)

[ https://issues.apache.org/jira/browse/SPARK-7736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621832#comment-14621832 ] Shay Rojansky commented on SPARK-7736: -- Neelesh, not sure I understood what you're

[jira] [Commented] (SPARK-8374) Job frequently hangs after YARN preemption

[ https://issues.apache.org/jira/browse/SPARK-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14605425#comment-14605425 ] Shay Rojansky commented on SPARK-8374: -- Thanks for your comment and sure, I can help

[jira] [Commented] (SPARK-8374) Job frequently hangs after YARN preemption

[ https://issues.apache.org/jira/browse/SPARK-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14605117#comment-14605117 ] Shay Rojansky commented on SPARK-8374: -- Any chance someone can look at this bug, at

[jira] [Commented] (SPARK-7736) Exception not failing Python applications (in yarn cluster mode)

[ https://issues.apache.org/jira/browse/SPARK-7736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14601923#comment-14601923 ] Shay Rojansky commented on SPARK-7736: -- The problem is simply with the YARN status

[jira] [Created] (SPARK-8374) Job frequently hangs after YARN preemption

Shay Rojansky created SPARK-8374: Summary: Job frequently hangs after YARN preemption Key: SPARK-8374 URL: https://issues.apache.org/jira/browse/SPARK-8374 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-7725) --py-files doesn't seem to work in YARN cluster mode

[ https://issues.apache.org/jira/browse/SPARK-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14564421#comment-14564421 ] Shay Rojansky commented on SPARK-7725: -- Here you go:

[jira] [Created] (SPARK-7725) --py-files doesn't seem to work in YARN cluster mode

Shay Rojansky created SPARK-7725: Summary: --py-files doesn't seem to work in YARN cluster mode Key: SPARK-7725 URL: https://issues.apache.org/jira/browse/SPARK-7725 Project: Spark Issue

[jira] [Closed] (SPARK-7709) spark-submit option to quit after submitting in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shay Rojansky closed SPARK-7709. Resolution: Duplicate Oops, it seems this was already implemented... spark-submit option to quit

[jira] [Created] (SPARK-7736) Exception not failing Python applications (in yarn cluster mode)

Shay Rojansky created SPARK-7736: Summary: Exception not failing Python applications (in yarn cluster mode) Key: SPARK-7736 URL: https://issues.apache.org/jira/browse/SPARK-7736 Project: Spark

[jira] [Created] (SPARK-7709) spark-submit option to quit after submitting in cluster mode

Shay Rojansky created SPARK-7709: Summary: spark-submit option to quit after submitting in cluster mode Key: SPARK-7709 URL: https://issues.apache.org/jira/browse/SPARK-7709 Project: Spark

[jira] [Commented] (SPARK-3644) REST API for Spark application info (jobs / stages / tasks / storage info)

[ https://issues.apache.org/jira/browse/SPARK-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547073#comment-14547073 ] Shay Rojansky commented on SPARK-3644: -- +1 on this, our main use would be to get

[jira] [Created] (SPARK-3470) Have JavaSparkContext implement Closeable/AutoCloseable

Shay Rojansky created SPARK-3470: Summary: Have JavaSparkContext implement Closeable/AutoCloseable Key: SPARK-3470 URL: https://issues.apache.org/jira/browse/SPARK-3470 Project: Spark Issue

[jira] [Created] (SPARK-3471) Automatic resource manager for SparkContext in Scala?

Shay Rojansky created SPARK-3471: Summary: Automatic resource manager for SparkContext in Scala? Key: SPARK-3471 URL: https://issues.apache.org/jira/browse/SPARK-3471 Project: Spark Issue

[jira] [Closed] (SPARK-2972) APPLICATION_COMPLETE not created in Python unless context explicitly stopped

[ https://issues.apache.org/jira/browse/SPARK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shay Rojansky closed SPARK-2972. Resolution: Won't Fix APPLICATION_COMPLETE not created in Python unless context explicitly stopped

[jira] [Commented] (SPARK-3470) Have JavaSparkContext implement Closeable/AutoCloseable

[ https://issues.apache.org/jira/browse/SPARK-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128285#comment-14128285 ] Shay Rojansky commented on SPARK-3470: -- Good point about AutoCloseable. Yes, the idea

[jira] [Commented] (SPARK-2972) APPLICATION_COMPLETE not created in Python unless context explicitly stopped

[ https://issues.apache.org/jira/browse/SPARK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126686#comment-14126686 ] Shay Rojansky commented on SPARK-2972: -- you're right! imho, this means your program

[jira] [Created] (SPARK-3457) ConcurrentModificationException starting up pyspark

Shay Rojansky created SPARK-3457: Summary: ConcurrentModificationException starting up pyspark Key: SPARK-3457 URL: https://issues.apache.org/jira/browse/SPARK-3457 Project: Spark Issue

[jira] [Updated] (SPARK-3457) ConcurrentModificationException starting up pyspark

[ https://issues.apache.org/jira/browse/SPARK-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shay Rojansky updated SPARK-3457: - Description: Just downloaded Spark 1.1.0-rc4. Launching pyspark for the very first time in

[jira] [Commented] (SPARK-2972) APPLICATION_COMPLETE not created in Python unless context explicitly stopped

[ https://issues.apache.org/jira/browse/SPARK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14127171#comment-14127171 ] Shay Rojansky commented on SPARK-2972: -- I'd love to help on this, but I know 0 Scala

[jira] [Commented] (SPARK-2972) APPLICATION_COMPLETE not created in Python unless context explicitly stopped

[ https://issues.apache.org/jira/browse/SPARK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124873#comment-14124873 ] Shay Rojansky commented on SPARK-2972: -- Thanks for answering. I guess it's a

[jira] [Commented] (SPARK-3183) Add option for requesting full YARN cluster

[ https://issues.apache.org/jira/browse/SPARK-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114368#comment-14114368 ] Shay Rojansky commented on SPARK-3183: -- +1. As a current workaround for cores, we

[jira] [Created] (SPARK-2971) Orphaned YARN ApplicationMaster lingers forever

Shay Rojansky created SPARK-2971: Summary: Orphaned YARN ApplicationMaster lingers forever Key: SPARK-2971 URL: https://issues.apache.org/jira/browse/SPARK-2971 Project: Spark Issue Type:

[jira] [Created] (SPARK-2972) APPLICATION_COMPLETE not created in Python unless context explicitly stopped

Shay Rojansky created SPARK-2972: Summary: APPLICATION_COMPLETE not created in Python unless context explicitly stopped Key: SPARK-2972 URL: https://issues.apache.org/jira/browse/SPARK-2972 Project:

[jira] [Commented] (SPARK-2945) Allow specifying num of executors in the context configuration

[ https://issues.apache.org/jira/browse/SPARK-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092058#comment-14092058 ] Shay Rojansky commented on SPARK-2945: -- I just did a quick test on Spark 1.0.2, and

[jira] [Updated] (SPARK-2960) Spark executables fail to start via symlinks

[ https://issues.apache.org/jira/browse/SPARK-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shay Rojansky updated SPARK-2960: - Priority: Minor (was: Major) Spark executables fail to start via symlinks

[jira] [Updated] (SPARK-2960) Spark executables fail to start via symlinks

[ https://issues.apache.org/jira/browse/SPARK-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shay Rojansky updated SPARK-2960: - Summary: Spark executables fail to start via symlinks (was: Spark executables failed to start

[jira] [Created] (SPARK-2960) Spark executables failed to start via symlinks

Shay Rojansky created SPARK-2960: Summary: Spark executables failed to start via symlinks Key: SPARK-2960 URL: https://issues.apache.org/jira/browse/SPARK-2960 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-2945) Allow specifying num of executors in the context configuration

Shay Rojansky created SPARK-2945: Summary: Allow specifying num of executors in the context configuration Key: SPARK-2945 URL: https://issues.apache.org/jira/browse/SPARK-2945 Project: Spark

[jira] [Created] (SPARK-2946) Allow specifying * for --num-executors in YARN

Shay Rojansky created SPARK-2946: Summary: Allow specifying * for --num-executors in YARN Key: SPARK-2946 URL: https://issues.apache.org/jira/browse/SPARK-2946 Project: Spark Issue Type:

[jira] [Commented] (SPARK-2880) spark-submit processes app cmdline options

[ https://issues.apache.org/jira/browse/SPARK-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090880#comment-14090880 ] Shay Rojansky commented on SPARK-2880: -- It's indeed a duplicate of that bug, great to

[jira] [Created] (SPARK-2880) spark-submit processes app cmdline options

Shay Rojansky created SPARK-2880: Summary: spark-submit processes app cmdline options Key: SPARK-2880 URL: https://issues.apache.org/jira/browse/SPARK-2880 Project: Spark Issue Type: Bug