[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-30 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-61056802 @harishreedharan I dont think so. The block location is called only once in both, and the hdfs location is called only once and only if required. I dont think there is any

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-30 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19591383 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-30 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-61057132 @tdas you also missed my two other comments --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-30 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-61071886 @rxin, crap, i missed that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-61073837 [Test build #22537 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22537/consoleFull) for PR 2931 at commit

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-61080839 [Test build #22537 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22537/consoleFull) for PR 2931 at commit

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-61080845 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-30 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19612661 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-30 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19634527 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-30 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19634884 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-30 Thread harishreedharan
Github user harishreedharan commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19635029 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-61167138 [Test build #22562 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22562/consoleFull) for PR 2931 at commit

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-30 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-61167522 Alright! I think we have converged to the best solution here. I am going to wait for the tests to pass and then converge. Thanks @rxin and @JoshRosen for all the feedback!

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread harishreedharan
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60969642 This one is ready to go in? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60983359 This looks good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19573331 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala --- @@ -0,0 +1,124 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19573363 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala --- @@ -0,0 +1,124 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19573690 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDDSuite.scala --- @@ -0,0 +1,155 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19573756 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDDSuite.scala --- @@ -0,0 +1,155 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19574104 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala --- @@ -0,0 +1,124 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19574168 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala --- @@ -0,0 +1,124 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-61009625 @harishreedharan / @tdas I made a few more comments. Most are just nits that I've left earlier. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread harishreedharan
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-61011552 Thanks @rxin. Updates coming soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19587299 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19587321 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala --- @@ -0,0 +1,124 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19587361 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala --- @@ -0,0 +1,124 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19587457 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala --- @@ -0,0 +1,124 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19587466 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDDSuite.scala --- @@ -0,0 +1,155 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-61044672 @rxin I updated. Only part i am not in agreement is the preferred location logic. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-61044841 [Test build #22521 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22521/consoleFull) for PR 2931 at commit

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread harishreedharan
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-61044847 Apart from the readability, does one have a performance benefit over the other? --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-61048431 [Test build #22521 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22521/consoleFull) for PR 2931 at commit

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-61048438 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19454510 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19454502 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19454508 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19454515 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19454523 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19454541 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19454543 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19454568 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19454583 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19454599 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19454635 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19454662 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19454664 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19454808 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19454823 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19454844 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19456819 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19456864 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19457227 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19483666 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19484208 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,163 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19484474 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,163 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19484412 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,163 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19485339 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60799300 @harishreedharan I am taking care of these comments to unblock other PRs. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread harishreedharan
Github user harishreedharan commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19490482 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19491638 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread harishreedharan
Github user harishreedharan commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19492065 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread harishreedharan
Github user harishreedharan commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19495119 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread harishreedharan
Github user harishreedharan commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19495201 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19499543 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19506065 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19506096 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19506112 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19506143 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19506184 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19506160 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19506198 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60837159 Updated! - Renamed the class, - Simplified the unit test code, added one more unit test and added lots of documentation. --- If your project is set up for it, you

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60837595 [Test build #22380 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22380/consoleFull) for PR 2931 at commit

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread harishreedharan
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60837536 Shouldn't it be WriteAheadLogBackedRDD not WriteAheadLogBasedBackedRDD (Based and Backed together without anything in between does not make sense - no?) --- If

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread harishreedharan
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60837781 OK, I wasn't reading properly - still drinking coffee. Ignore my last comment please! --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60837839 [Test build #22380 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22380/consoleFull) for PR 2931 at commit

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60837845 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread harishreedharan
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60845224 +1. The changes look good (apart from the style comments that Lord Jenkins made above) --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60846631 Dummy me. My IntelliJ is misconfigured to warn lines 120 chars long, not 100. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60847243 [Test build #22387 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22387/consoleFull) for PR 2931 at commit

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60847372 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60847367 [Test build #22387 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22387/consoleFull) for PR 2931 at commit

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60853490 [Test build #22398 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22398/consoleFull) for PR 2931 at commit

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60860158 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60860153 [Test build #22398 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22398/consoleFull) for PR 2931 at commit

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-27 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19422693 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60637477 [Test build #22300 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22300/consoleFull) for PR 2931 at commit

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60649869 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60649863 [Test build #22300 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22300/consoleFull) for PR 2931 at commit

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-27 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19428914 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,152 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-27 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19428929 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,152 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-27 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19428909 --- Diff: streaming/src/test/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDDSuite.scala --- @@ -0,0 +1,152 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-27 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/2931#discussion_r19429864 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/HDFSBackedBlockRDD.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-27 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60653950 Jenkins, test this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60659980 [Test build #22302 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22302/consoleFull) for PR 2931 at commit

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60664704 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-27 Thread shaneknapp
Github user shaneknapp commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60664941 jenkins, test this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-27 Thread shaneknapp
Github user shaneknapp commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60668550 jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60668710 [Test build #22306 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22306/consoleFull) for PR 2931 at commit

[GitHub] spark pull request: [SPARK-4027][Streaming] HDFSBasedBlockRDD to r...

2014-10-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2931#issuecomment-60678825 [Test build #22306 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22306/consoleFull) for PR 2931 at commit

  1   2   >