[jira] [Created] (SPARK-1900) Fix running PySpark files on YARN

2014-05-22 Thread Andrew Or (JIRA)
Andrew Or created SPARK-1900: Summary: Fix running PySpark files on YARN Key: SPARK-1900 URL: https://issues.apache.org/jira/browse/SPARK-1900 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-1870) Jars specified via --jars in spark-submit are not added to executor classpath for YARN

2014-05-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-1870: - Affects Version/s: (was: 1.0.0) Jars specified via --jars in spark-submit are not added to

[jira] [Resolved] (SPARK-1870) Jars specified via --jars in spark-submit are not added to executor classpath for YARN

2014-05-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-1870. -- Resolution: Fixed Fix Version/s: 1.0.0 Fixed in

[jira] [Commented] (SPARK-1901) Standalone worker update exector's state ahead of executor process exit

2014-05-22 Thread Zhen Peng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005749#comment-14005749 ] Zhen Peng commented on SPARK-1901: -- https://github.com/apache/spark/pull/854 Standalone

[jira] [Updated] (SPARK-1900) Fix running PySpark files on YARN

2014-05-22 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1900: - Description: If I run the following on a YARN cluster ``` bin/spark-submit sheep.py --master yarn-client

[jira] [Updated] (SPARK-1900) Fix running PySpark files on YARN

2014-05-22 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1900: - Description: If I run the following on a YARN cluster {code} bin/spark-submit sheep.py --master

[jira] [Updated] (SPARK-1900) Fix running PySpark files on YARN

2014-05-22 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-1900: - Description: If I run the following on a YARN cluster {code} bin/spark-submit sheep.py --master

[jira] [Commented] (SPARK-983) External hashing sorting support

2014-05-22 Thread Madhu Siddalingaiah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006161#comment-14006161 ] Madhu Siddalingaiah commented on SPARK-983: --- Can someone comment on the scope of

[jira] [Commented] (SPARK-983) External hashing sorting support

2014-05-22 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006221#comment-14006221 ] Andrew Ash commented on SPARK-983: -- I think the main intention is to make the .sortByKey()

[jira] [Commented] (SPARK-983) External hashing sorting support

2014-05-22 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006251#comment-14006251 ] Aaron Davidson commented on SPARK-983: -- This JIRA is pretty vague. We've already

[jira] [Updated] (SPARK-983) Support external sorting for RDD#sortByKey()

2014-05-22 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-983: - Assignee: (was: Aaron Davidson) Support external sorting for RDD#sortByKey()

[jira] [Created] (SPARK-1902) Spark shell prints error when :4040 port already in use

2014-05-22 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-1902: - Summary: Spark shell prints error when :4040 port already in use Key: SPARK-1902 URL: https://issues.apache.org/jira/browse/SPARK-1902 Project: Spark Issue Type:

[jira] [Updated] (SPARK-1902) Spark shell prints error when :4040 port already in use

2014-05-22 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-1902: -- Description: When running two shells on the same machine, I get the below error. The issue is that

[jira] [Created] (SPARK-1903) Document Spark's network connections

2014-05-22 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-1903: - Summary: Document Spark's network connections Key: SPARK-1903 URL: https://issues.apache.org/jira/browse/SPARK-1903 Project: Spark Issue Type: Documentation

[jira] [Updated] (SPARK-1904) ZooKeeper URI in spark-env.sh no longer working w/ bin/pyspark

2014-05-22 Thread Eric Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Lee updated SPARK-1904: Description: Attempting to run my bin/pyspark interactive shell against a Mesos cluster this assignment

[jira] [Updated] (SPARK-1904) ZooKeeper URI in spark-env.sh no longer working w/ bin/pyspark

2014-05-22 Thread Eric Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Lee updated SPARK-1904: Description: Attempting to run my bin/pyspark interactive shell against a Mesos cluster this assignment

[jira] [Updated] (SPARK-1904) ZooKeeper URI in spark-env.sh no longer working w/ bin/pyspark

2014-05-22 Thread Eric Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Lee updated SPARK-1904: Description: Attempting to run my bin/pyspark interactive shell against a Mesos cluster this assignment

[jira] [Commented] (SPARK-1898) In deploy.yarn.Client, use YarnClient rather than YarnClientImpl

2014-05-22 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006494#comment-14006494 ] Sandy Ryza commented on SPARK-1898: --- The change only applies to Spark's stable/2.2+

[jira] [Commented] (SPARK-1904) ZooKeeper URI in spark-env.sh no longer working w/ bin/pyspark

2014-05-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006514#comment-14006514 ] Tathagata Das commented on SPARK-1904: -- This seems to be a regression. Since we are

[jira] [Updated] (SPARK-1870) Jars specified via --jars in spark-submit are not added to executor classpath for YARN

2014-05-22 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-1870: - Issue Type: Sub-task (was: Bug) Parent: SPARK-1905 Jars specified via --jars in

[jira] [Created] (SPARK-1907) spark-submit: add exec at the end of the script

2014-05-22 Thread Colin Patrick McCabe (JIRA)
Colin Patrick McCabe created SPARK-1907: --- Summary: spark-submit: add exec at the end of the script Key: SPARK-1907 URL: https://issues.apache.org/jira/browse/SPARK-1907 Project: Spark

[jira] [Created] (SPARK-1906) spark-submit doesn't send master URL to Driver in standalone cluster mode

2014-05-22 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-1906: Summary: spark-submit doesn't send master URL to Driver in standalone cluster mode Key: SPARK-1906 URL: https://issues.apache.org/jira/browse/SPARK-1906 Project:

[jira] [Created] (SPARK-1908) Support local app jar in standalone cluster mode

2014-05-22 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-1908: Summary: Support local app jar in standalone cluster mode Key: SPARK-1908 URL: https://issues.apache.org/jira/browse/SPARK-1908 Project: Spark Issue Type:

[jira] [Updated] (SPARK-1904) ZooKeeper URI in spark-env.sh no longer working w/ bin/pyspark

2014-05-22 Thread Eric Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Lee updated SPARK-1904: Environment: Ubuntu AMI in EC2, build taken from tags/spark-v1.0.0 tag {code} $ lsb_release -r Release:

[jira] [Created] (SPARK-1909) --jars is not supported in standalone cluster mode

2014-05-22 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-1909: Summary: --jars is not supported in standalone cluster mode Key: SPARK-1909 URL: https://issues.apache.org/jira/browse/SPARK-1909 Project: Spark Issue Type:

[jira] [Commented] (SPARK-1898) In deploy.yarn.Client, use YarnClient rather than YarnClientImpl

2014-05-22 Thread Colin Patrick McCabe (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006540#comment-14006540 ] Colin Patrick McCabe commented on SPARK-1898: - Yeah, we have the

[jira] [Updated] (SPARK-1900) Fix running PySpark files on YARN

2014-05-22 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-1900: - Issue Type: Sub-task (was: Bug) Parent: SPARK-1652 Fix running PySpark files on YARN

[jira] [Updated] (SPARK-1900) Fix running PySpark files on YARN

2014-05-22 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-1900: - Issue Type: Bug (was: Sub-task) Parent: (was: SPARK-1905) Fix running PySpark

[jira] [Updated] (SPARK-1870) Jars specified via --jars in spark-submit are not added to executor classpath for YARN

2014-05-22 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-1870: - Issue Type: Sub-task (was: Improvement) Parent: SPARK-1652 Jars specified via --jars

[jira] [Updated] (SPARK-1906) spark-submit doesn't send master URL to Driver in standalone cluster mode

2014-05-22 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-1906: - Issue Type: Improvement (was: Sub-task) Parent: (was: SPARK-1905) spark-submit

[jira] [Updated] (SPARK-1909) --jars is not supported in standalone cluster mode

2014-05-22 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-1909: - Issue Type: Improvement (was: Sub-task) Parent: (was: SPARK-1905) --jars is not

[jira] [Updated] (SPARK-1900) Fix running PySpark files on YARN

2014-05-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-1900: - Component/s: Deploy Fix running PySpark files on YARN --

[jira] [Updated] (SPARK-1908) Support local app jar in standalone cluster mode

2014-05-22 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-1908: - Issue Type: Bug (was: Sub-task) Parent: (was: SPARK-1905) Support local app jar in

[jira] [Updated] (SPARK-1908) Support local app jar in standalone cluster mode

2014-05-22 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-1908: - Issue Type: Sub-task (was: Bug) Parent: SPARK-1652 Support local app jar in standalone

[jira] [Updated] (SPARK-1906) spark-submit doesn't send master URL to Driver in standalone cluster mode

2014-05-22 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-1906: - Issue Type: Sub-task (was: Improvement) Parent: SPARK-1652 spark-submit doesn't send

[jira] [Resolved] (SPARK-1905) Issues with `spark-submit`

2014-05-22 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-1905. -- Resolution: Duplicate Issues with `spark-submit` --

[jira] [Updated] (SPARK-1904) ZooKeeper URI in spark-env.sh no longer working w/ bin/pyspark

2014-05-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-1904: - Assignee: Andrew Or (was: Tathagata Das) ZooKeeper URI in spark-env.sh no longer working w/

[jira] [Assigned] (SPARK-1904) ZooKeeper URI in spark-env.sh no longer working w/ bin/pyspark

2014-05-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-1904: Assignee: Tathagata Das ZooKeeper URI in spark-env.sh no longer working w/ bin/pyspark

[jira] [Created] (SPARK-1910) Add onBlockComplete API to receiver

2014-05-22 Thread Hari Shreedharan (JIRA)
Hari Shreedharan created SPARK-1910: --- Summary: Add onBlockComplete API to receiver Key: SPARK-1910 URL: https://issues.apache.org/jira/browse/SPARK-1910 Project: Spark Issue Type: Bug

[jira] [Closed] (SPARK-1904) ZooKeeper URI in spark-env.sh no longer working w/ bin/pyspark

2014-05-22 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-1904. Resolution: Not a Problem ZooKeeper URI in spark-env.sh no longer working w/ bin/pyspark

[jira] [Commented] (SPARK-983) Support external sorting for RDD#sortByKey()

2014-05-22 Thread Madhu Siddalingaiah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006664#comment-14006664 ] Madhu Siddalingaiah commented on SPARK-983: --- Looking at

[jira] [Resolved] (SPARK-1897) Spark shell --jars (or spark.jars) doesn't work

2014-05-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-1897. -- Resolution: Fixed Fix Version/s: 1.0.0 https://github.com/apache/spark/pull/849 Spark

[jira] [Resolved] (SPARK-1896) MASTER masks spark.master in spark-shell

2014-05-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-1896. -- Resolution: Fixed Fix Version/s: 1.0.0 https://github.com/apache/spark/pull/846/

[jira] [Created] (SPARK-1912) Compression memory issue during shuffle

2014-05-22 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-1912: -- Summary: Compression memory issue during shuffle Key: SPARK-1912 URL: https://issues.apache.org/jira/browse/SPARK-1912 Project: Spark Issue Type: Bug