[jira] [Created] (SPARK-2013) Add Python pickleFile to programming guide

Matei Zaharia created SPARK-2013: Summary: Add Python pickleFile to programming guide Key: SPARK-2013 URL: https://issues.apache.org/jira/browse/SPARK-2013 Project: Spark Issue Type:

[jira] [Created] (SPARK-2014) Make PySpark store RDDs in MEMORY_ONLY_SER with compression by default

Matei Zaharia created SPARK-2014: Summary: Make PySpark store RDDs in MEMORY_ONLY_SER with compression by default Key: SPARK-2014 URL: https://issues.apache.org/jira/browse/SPARK-2014 Project: Spark

[jira] [Commented] (SPARK-1977) mutable.BitSet in ALS not serializable with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017477#comment-14017477 ] Xiangrui Meng commented on SPARK-1977: -- This is more likely a version conflict in

[jira] [Created] (SPARK-2016) rdd in-memory storage UI becomes unresponsive when the number of RDD partitions is large

Reynold Xin created SPARK-2016: -- Summary: rdd in-memory storage UI becomes unresponsive when the number of RDD partitions is large Key: SPARK-2016 URL: https://issues.apache.org/jira/browse/SPARK-2016

[jira] [Created] (SPARK-2017) web ui stage page becomes unresponsive when the number of tasks is large

Reynold Xin created SPARK-2017: -- Summary: web ui stage page becomes unresponsive when the number of tasks is large Key: SPARK-2017 URL: https://issues.apache.org/jira/browse/SPARK-2017 Project: Spark

[jira] [Updated] (SPARK-2016) rdd in-memory storage UI becomes unresponsive when the number of RDD partitions is large

[ https://issues.apache.org/jira/browse/SPARK-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-2016: --- Labels: starter (was: ) rdd in-memory storage UI becomes unresponsive when the number of RDD

[jira] [Comment Edited] (SPARK-1977) mutable.BitSet in ALS not serializable with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017508#comment-14017508 ] Neville Li edited comment on SPARK-1977 at 6/4/14 8:45 AM: --- We

[jira] [Commented] (SPARK-1977) mutable.BitSet in ALS not serializable with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017508#comment-14017508 ] Neville Li commented on SPARK-1977: --- We submit 1 spark-assembly and 1 job assembly jar

[jira] [Issue Comment Deleted] (SPARK-1999) UI : StorageLevel in storage tab and RDD Storage Info never changes

[ https://issues.apache.org/jira/browse/SPARK-1999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Chao updated SPARK-1999: - Comment: was deleted (was: https://github.com/apache/spark/pull/950 sorry，i will repost soon, the above

[jira] [Commented] (SPARK-1999) UI : StorageLevel in storage tab and RDD Storage Info never changes

[ https://issues.apache.org/jira/browse/SPARK-1999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017519#comment-14017519 ] Chen Chao commented on SPARK-1999: -- PR：https://github.com/apache/spark/pull/968 UI :

[jira] [Issue Comment Deleted] (SPARK-1999) UI : StorageLevel in storage tab and RDD Storage Info never changes

[ https://issues.apache.org/jira/browse/SPARK-1999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Chao updated SPARK-1999: - Comment: was deleted (was: I have fixed and tested fine. Please assign it to me , I will post a PR

[jira] [Commented] (SPARK-2018) Big-Endian (IBM Power7) Spark Serialization issue

[ https://issues.apache.org/jira/browse/SPARK-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017528#comment-14017528 ] Sean Owen commented on SPARK-2018: -- The meaning of the error is that Java thinks two

[jira] [Created] (SPARK-2019) Spark workers die/disappear when job fails for nearly any reason

sam created SPARK-2019: -- Summary: Spark workers die/disappear when job fails for nearly any reason Key: SPARK-2019 URL: https://issues.apache.org/jira/browse/SPARK-2019 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-1520) Assembly Jar with more than 65536 files won't work when compiled on JDK7 and run on JDK6

[ https://issues.apache.org/jira/browse/SPARK-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017753#comment-14017753 ] Qiuzhuang Lian commented on SPARK-1520: --- I can run the assembly jar via

[jira] [Commented] (SPARK-2019) Spark workers die/disappear when job fails for nearly any reason

[ https://issues.apache.org/jira/browse/SPARK-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017776#comment-14017776 ] Mark Hamstra commented on SPARK-2019: - Please don't leave the Affects Version/s

[jira] [Commented] (SPARK-1817) RDD zip erroneous when partitions do not divide RDD count

[ https://issues.apache.org/jira/browse/SPARK-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017812#comment-14017812 ] Kan Zhang commented on SPARK-1817: -- There are 2 issues related to this bug. One is that

[jira] [Issue Comment Deleted] (SPARK-1817) RDD zip erroneous when partitions do not divide RDD count

[ https://issues.apache.org/jira/browse/SPARK-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated SPARK-1817: - Comment: was deleted (was: PR: https://github.com/apache/spark/pull/760) RDD zip erroneous when

[jira] [Updated] (SPARK-2013) Add Python pickleFile to programming guide

[ https://issues.apache.org/jira/browse/SPARK-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matei Zaharia updated SPARK-2013: - Assignee: Kan Zhang Add Python pickleFile to programming guide

[jira] [Resolved] (SPARK-1973) Add randomSplit to JavaRDD (with tests, and tidy Java tests)

[ https://issues.apache.org/jira/browse/SPARK-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-1973. -- Resolution: Implemented PR: https://github.com/apache/spark/pull/919 Add randomSplit to

[jira] [Updated] (SPARK-1973) Add randomSplit to JavaRDD (with tests, and tidy Java tests)

[ https://issues.apache.org/jira/browse/SPARK-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-1973: - Assignee: Sean Owen Add randomSplit to JavaRDD (with tests, and tidy Java tests)

[jira] [Issue Comment Deleted] (SPARK-1704) java.lang.AssertionError: assertion failed: No plan for ExplainCommand (Project [*])

[ https://issues.apache.org/jira/browse/SPARK-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-1704: Comment: was deleted (was: [~marmbrus] I am attaching the link to the PR.) java.lang.AssertionError:

[jira] [Created] (SPARK-2020) Spark 1.0.0 fails to run in coarse-grained mesos mode

Ajay Viswanathan created SPARK-2020: --- Summary: Spark 1.0.0 fails to run in coarse-grained mesos mode Key: SPARK-2020 URL: https://issues.apache.org/jira/browse/SPARK-2020 Project: Spark

[jira] [Commented] (SPARK-2020) Spark 1.0.0 fails to run in coarse-grained mesos mode

[ https://issues.apache.org/jira/browse/SPARK-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018022#comment-14018022 ] Ajay Viswanathan commented on SPARK-2020: - Do I have to use Java 8 to rectify this

[jira] [Commented] (SPARK-2019) Spark workers die/disappear when job fails for nearly any reason

[ https://issues.apache.org/jira/browse/SPARK-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018048#comment-14018048 ] sam commented on SPARK-2019: Sorry. Its 0.9.1 Spark workers die/disappear when job fails for

[jira] [Commented] (SPARK-1508) Add support for reading from SparkConf

[ https://issues.apache.org/jira/browse/SPARK-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018043#comment-14018043 ] Zongheng Yang commented on SPARK-1508: -- WIP PR:

[jira] [Commented] (SPARK-1508) Add support for reading from SparkConf

[ https://issues.apache.org/jira/browse/SPARK-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018053#comment-14018053 ] Michael Armbrust commented on SPARK-1508: - It is likely we will fix this issue

[jira] [Updated] (SPARK-2019) Spark workers die/disappear when job fails for nearly any reason

[ https://issues.apache.org/jira/browse/SPARK-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Hamstra updated SPARK-2019: Affects Version/s: 0.9.1 Spark workers die/disappear when job fails for nearly any reason

[jira] [Updated] (SPARK-2019) Spark workers die/disappear when job fails for nearly any reason

[ https://issues.apache.org/jira/browse/SPARK-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2019: --- Fix Version/s: 0.9.2 Spark workers die/disappear when job fails for nearly any reason

[jira] [Commented] (SPARK-2019) Spark workers die/disappear when job fails for nearly any reason

[ https://issues.apache.org/jira/browse/SPARK-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018086#comment-14018086 ] Patrick Wendell commented on SPARK-2019: We should dig into this and figure out

[jira] [Updated] (SPARK-2019) Spark workers die/disappear when job fails for nearly any reason

[ https://issues.apache.org/jira/browse/SPARK-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2019: --- Priority: Critical (was: Major) Spark workers die/disappear when job fails for nearly any

[jira] [Commented] (SPARK-1977) mutable.BitSet in ALS not serializable with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018155#comment-14018155 ] Xiangrui Meng commented on SPARK-1977: -- In our example code, we only register

[jira] [Updated] (SPARK-1912) Compression memory issue during reduce

[ https://issues.apache.org/jira/browse/SPARK-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matei Zaharia updated SPARK-1912: - Target Version/s: 0.9.2, 1.0.1, 1.1.0 (was: 0.9.2, 1.0.1) Compression memory issue during

[jira] [Updated] (SPARK-1912) Compression memory issue during reduce

[ https://issues.apache.org/jira/browse/SPARK-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matei Zaharia updated SPARK-1912: - Target Version/s: 0.9.2, 1.0.1 Compression memory issue during reduce

[jira] [Commented] (SPARK-1977) mutable.BitSet in ALS not serializable with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018254#comment-14018254 ] Neville Li commented on SPARK-1977: --- Yes we did register 'Rating'. And we had to

[jira] [Created] (SPARK-2023) PySpark reduce does a map side reduce and then sends the results to the driver for final reduce, instead do this more like Scala Spark.

holdenk created SPARK-2023: -- Summary: PySpark reduce does a map side reduce and then sends the results to the driver for final reduce, instead do this more like Scala Spark. Key: SPARK-2023 URL:

[jira] [Created] (SPARK-2024) Add saveAsSequenceFile to PySpark

Matei Zaharia created SPARK-2024: Summary: Add saveAsSequenceFile to PySpark Key: SPARK-2024 URL: https://issues.apache.org/jira/browse/SPARK-2024 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-1790) Update EC2 scripts to support r3 instance types

[ https://issues.apache.org/jira/browse/SPARK-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1790: --- Fix Version/s: 1.1.0 0.9.2 Update EC2 scripts to support r3 instance

[jira] [Resolved] (SPARK-1790) Update EC2 scripts to support r3 instance types

[ https://issues.apache.org/jira/browse/SPARK-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1790. Resolution: Fixed Update EC2 scripts to support r3 instance types

[jira] [Commented] (SPARK-2011) Eliminate duplicate join in Pregel

[ https://issues.apache.org/jira/browse/SPARK-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018327#comment-14018327 ] Tim Weninger commented on SPARK-2011: - I also think that there is a memory leak

[jira] [Commented] (SPARK-1977) mutable.BitSet in ALS not serializable with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018328#comment-14018328 ] Shuo Xiang commented on SPARK-1977: --- Hi [~neville], I just run the MovieLens example on

[jira] [Created] (SPARK-2025) EdgeRDD persists after pregel iteration

Tim Weninger created SPARK-2025: --- Summary: EdgeRDD persists after pregel iteration Key: SPARK-2025 URL: https://issues.apache.org/jira/browse/SPARK-2025 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-2025) EdgeRDD persists after pregel iteration

[ https://issues.apache.org/jira/browse/SPARK-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Weninger updated SPARK-2025: Description: Symptoms: During execution of a pregel script/function a copy of an intermediate

[jira] [Created] (SPARK-2026) Maven hadoop* Profiles Should Set the expected Hadoop Version.

Bernardo Gomez Palacio created SPARK-2026: - Summary: Maven hadoop* Profiles Should Set the expected Hadoop Version. Key: SPARK-2026 URL: https://issues.apache.org/jira/browse/SPARK-2026

[jira] [Created] (SPARK-2027) spark-ec2 puts Hadoop's log4j ahead of Spark's in classpath

Aaron Davidson created SPARK-2027: - Summary: spark-ec2 puts Hadoop's log4j ahead of Spark's in classpath Key: SPARK-2027 URL: https://issues.apache.org/jira/browse/SPARK-2027 Project: Spark

[jira] [Assigned] (SPARK-2025) EdgeRDD persists after pregel iteration

[ https://issues.apache.org/jira/browse/SPARK-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Dave reassigned SPARK-2025: - Assignee: Ankur Dave EdgeRDD persists after pregel iteration

[jira] [Commented] (SPARK-2025) EdgeRDD persists after pregel iteration

[ https://issues.apache.org/jira/browse/SPARK-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018389#comment-14018389 ] Tim Weninger commented on SPARK-2025: - adding

[jira] [Commented] (SPARK-1977) mutable.BitSet in ALS not serializable with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018390#comment-14018390 ] Neville Li commented on SPARK-1977: --- Our YARN cluster runs 2.2.0. We built

[jira] [Commented] (SPARK-2025) EdgeRDD persists after pregel iteration

[ https://issues.apache.org/jira/browse/SPARK-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018391#comment-14018391 ] Tim Weninger commented on SPARK-2025: - I'll leave it to you to make the bug fix. You

[jira] [Commented] (SPARK-2025) EdgeRDD persists after pregel iteration

[ https://issues.apache.org/jira/browse/SPARK-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018402#comment-14018402 ] Ankur Dave commented on SPARK-2025: --- Proposed fix:

[jira] [Updated] (SPARK-1988) Enable storing edges out-of-core

[ https://issues.apache.org/jira/browse/SPARK-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Dave updated SPARK-1988: -- Priority: Minor (was: Major) Enable storing edges out-of-core

[jira] [Commented] (SPARK-2018) Big-Endian (IBM Power7) Spark Serialization issue

[ https://issues.apache.org/jira/browse/SPARK-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018409#comment-14018409 ] Yanjie Gao commented on SPARK-2018: --- Thanks for your quick reply! I believe they use

[jira] [Created] (SPARK-2028) Users of HadoopRDD cannot access the partition InputSplits

Aaron Davidson created SPARK-2028: - Summary: Users of HadoopRDD cannot access the partition InputSplits Key: SPARK-2028 URL: https://issues.apache.org/jira/browse/SPARK-2028 Project: Spark

[jira] [Updated] (SPARK-2028) Users of HadoopRDD cannot access the partition InputSplits

[ https://issues.apache.org/jira/browse/SPARK-2028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2028: --- Issue Type: New Feature (was: Bug) Users of HadoopRDD cannot access the partition

[jira] [Commented] (SPARK-2028) Let users of HadoopRDD access the partition InputSplits

[ https://issues.apache.org/jira/browse/SPARK-2028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018474#comment-14018474 ] Patrick Wendell commented on SPARK-2028: I wantonly changed this from a Bug to a

[jira] [Updated] (SPARK-2027) spark-ec2 puts Hadoop's log4j ahead of Spark's in classpath

[ https://issues.apache.org/jira/browse/SPARK-2027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2027: --- Component/s: EC2 spark-ec2 puts Hadoop's log4j ahead of Spark's in classpath

[jira] [Updated] (SPARK-2028) Let users of HadoopRDD access the partition InputSplits

[ https://issues.apache.org/jira/browse/SPARK-2028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2028: --- Summary: Let users of HadoopRDD access the partition InputSplits (was: Users of HadoopRDD

[jira] [Commented] (SPARK-2024) Add saveAsSequenceFile to PySpark

[ https://issues.apache.org/jira/browse/SPARK-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018485#comment-14018485 ] Kan Zhang commented on SPARK-2024: -- You meant SPARK-1416? Add saveAsSequenceFile to

[jira] [Created] (SPARK-2029) Bump pom.xml version number of master branch to 1.1.0-SNAPSHOT.

Takuya Ueshin created SPARK-2029: Summary: Bump pom.xml version number of master branch to 1.1.0-SNAPSHOT. Key: SPARK-2029 URL: https://issues.apache.org/jira/browse/SPARK-2029 Project: Spark

[jira] [Commented] (SPARK-2029) Bump pom.xml version number of master branch to 1.1.0-SNAPSHOT.

[ https://issues.apache.org/jira/browse/SPARK-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018492#comment-14018492 ] Takuya Ueshin commented on SPARK-2029: -- PRed:

[jira] [Created] (SPARK-2030) Bump SparkBuild.scala version number of branch-1.0 to 1.0.1-SNAPSHOT.

Takuya Ueshin created SPARK-2030: Summary: Bump SparkBuild.scala version number of branch-1.0 to 1.0.1-SNAPSHOT. Key: SPARK-2030 URL: https://issues.apache.org/jira/browse/SPARK-2030 Project: Spark

[jira] [Commented] (SPARK-2030) Bump SparkBuild.scala version number of branch-1.0 to 1.0.1-SNAPSHOT.

[ https://issues.apache.org/jira/browse/SPARK-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018493#comment-14018493 ] Takuya Ueshin commented on SPARK-2030: -- PRed: