[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-52013782 QA tests have started for PR 1907. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18420/consoleFull --- If

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-52013487 QA tests have started for PR 1907. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18419/consoleFull --- If

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-52013468 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-52013347 QA results for PR 1907:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-52013277 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-52011849 QA results for PR 1907:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-52009548 QA tests have started for PR 1907. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18410/consoleFull --- If

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-52008263 QA tests have started for PR 1907. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18408/consoleFull --- If

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-52007898 QA results for PR 1907:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-52005111 QA results for PR 1907:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-52004801 QA tests have started for PR 1907. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18406/consoleFull --- If

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-52003747 QA tests have started for PR 1907. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18405/consoleFull --- If

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51995339 QA results for PR 1907:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51988946 QA tests have started for PR 1907. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18391/consoleFull --- If

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16139683 --- Diff: core/src/main/scala/org/apache/spark/network/netty/ReferenceCountedBuffer.scala --- @@ -17,15 +17,25 @@ package org.apache.spark.network.ne

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51970648 I'm not sure what akka.io would bring here. The main benefit seems to be more native Scala APIs. In this case I am not sure if we care since this is a low level module. Akka

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread mubarak
Github user mubarak commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51969476 @rxin Sorry for this question and not related to code review at all. Any thoughts on using [Akka IO](http://doc.akka.io/docs/akka/snapshot/scala/io.html) package tha

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16135674 --- Diff: core/src/main/scala/org/apache/spark/network/netty/client.scala --- @@ -0,0 +1,276 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16134921 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala --- @@ -262,77 +271,49 @@ object BlockFetcherIterator { readMetrics:

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread coderplay
Github user coderplay commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16134600 --- Diff: core/src/main/scala/org/apache/spark/network/netty/server.scala --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread coderplay
Github user coderplay commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16134481 --- Diff: core/src/main/scala/org/apache/spark/network/netty/server.scala --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread coderplay
Github user coderplay commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16134163 --- Diff: core/src/main/scala/org/apache/spark/network/netty/server.scala --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread dcapwell
Github user dcapwell commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16134103 --- Diff: core/src/main/scala/org/apache/spark/network/netty/client.scala --- @@ -0,0 +1,276 @@ +/* + * Licensed to the Apache Software Foundation (A

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread coderplay
Github user coderplay commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16134006 --- Diff: core/src/main/scala/org/apache/spark/network/netty/server.scala --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread dcapwell
Github user dcapwell commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16133840 --- Diff: core/src/main/scala/org/apache/spark/network/netty/ReferenceCountedBuffer.scala --- @@ -17,15 +17,25 @@ package org.apache.spark.networ

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51958814 QA results for PR 1907:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51958686 QA tests have started for PR 1907. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18380/consoleFull --- If

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51957274 QA results for PR 1907:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51957233 That's part of the TODO :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16132782 --- Diff: core/src/main/scala/org/apache/spark/network/netty/server.scala --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51957157 QA tests have started for PR 1907. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18378/consoleFull --- If

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51957070 @rxin Very nice ! Do you have any benchmarks of how fast things are ? In terms of say % of network bandwidth we can use ? --- If your project is set up for it, you can

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51956530 QA results for PR 1907:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51956457 QA tests have started for PR 1907. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18377/consoleFull --- If

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/1907 [WIP] [SPARK-2468] Netty based block server / client module This is a rough draft, but it can't be much worse than the old Netty module since the old one didn't really work :) Compared with th