[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/156 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39687170 Merged, thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39657291 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39657292 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13815/ --- If your project

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39656279 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39656284 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39656235 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39652010 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13801/ --- If your project is set up for it, you can r

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39652009 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39649892 Awesome, thanks! I'll merge it once it gets past Jenkins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39649894 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39649898 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39649888 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/156#discussion_r11322315 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -664,6 +664,18 @@ abstract class RDD[T: ClassTag]( } /** + * Retu

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user epahomov commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39630218 Checked laziness. Good to go. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user epahomov commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39630047 Sure, I like this approach, I will change it on Sunday. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. I

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39601555 @epahomov Do you mind changing the code to use Iterators and flatMap as suggested too? It would be simpler than what's there now. --- If your project is set up for it, you

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user epahomov commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39538362 Doc's changed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39499615 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39499617 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13731/ --- If your project is set up for it, you can r

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/156#discussion_r11266579 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -664,6 +664,24 @@ abstract class RDD[T: ClassTag]( } /** + * Return

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39490196 Made some more comments to simplify this now that we return an Iterator. --- If your project is set up for it, you can reply to this email and have your reply appear on Git

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/156#discussion_r11266448 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -664,6 +664,24 @@ abstract class RDD[T: ClassTag]( } /** + * Return

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/156#discussion_r11266352 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -664,6 +664,24 @@ abstract class RDD[T: ClassTag]( } /** + * Return

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39489554 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39489572 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/156#discussion_r11266266 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -664,6 +664,24 @@ abstract class RDD[T: ClassTag]( } /** + * Return

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/156#discussion_r11266262 --- Diff: core/src/main/scala/org/apache/spark/api/java/JavaRDDLike.scala --- @@ -282,6 +283,17 @@ trait JavaRDDLike[T, This <: JavaRDDLike[T, This]] extends S

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-39489375 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-38881877 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your proj

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user epahomov commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-38361789 Mark, Ok. Let this review be about simple changes for some user cases and next one I try to cover more. --- If your project is set up for it, you can reply to this emai

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user markhamstra commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-38358029 @epahomov This PR may do everything that you want in terms of a streaming iterator, but I can tell you with certainty that only being able to fetch a single partition

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-38353582 Some time in future (not in this PR), we would want to add support for disk-backed support for this feature : in case a partition is too large to fit into memory. There is

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user markhamstra commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-38339261 Sure, these locally iterable results are useful even without async and the ability to cancel part way through the iteration; but that functionality is also very useful

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-38338520 @markhamstra I'd be okay if this doesn't support an async version right now since it's a tricky one to make async. It still seems useful as is. --- If your project is set

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/156#discussion_r10859655 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -664,6 +664,24 @@ abstract class RDD[T: ClassTag]( } /** + * Return

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/156#discussion_r10859640 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -664,6 +664,24 @@ abstract class RDD[T: ClassTag]( } /** + * Return

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/156#discussion_r10859630 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -664,6 +664,24 @@ abstract class RDD[T: ClassTag]( } /** + * Return

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user epahomov commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-38143969 Hi, what do you think about new changes? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project d

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user epahomov commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-37912860 Code added. Tests passed. Let's do it again) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your proje

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user markhamstra commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-37828372 Looks like this effort should be coordinated with this PR that should be brought over from incubator-spark: https://github.com/apache/incubator-spark/pull/421

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-37790992 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13208/ --- If your p

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-37790991 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-37790929 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-37790928 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-37789500 Hey Egor, this looks good, but a few comments: * We should add a Java API method for it too, in JavaRDDLike. Unfortunately, `toStream` will not be a good name there beca

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/156#discussion_r10647019 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -664,6 +664,22 @@ abstract class RDD[T: ClassTag]( } /** + * Return

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/156#discussion_r10647011 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -664,6 +664,22 @@ abstract class RDD[T: ClassTag]( } /** + * Return

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/156#issuecomment-37789306 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-1259] Make RDD locally iterable

Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/156#discussion_r10647006 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -664,6 +664,22 @@ abstract class RDD[T: ClassTag]( } /** + * Return