[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157116469
  
**[Test build #45999 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45999/consoleFull)**
 for PR 9619 at commit 
[`180456c`](https://github.com/apache/spark/commit/180456ce3d25a126a74fa0f57c22665ef002b6cd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157111920
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157149640
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45999/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157149438
  
**[Test build #45999 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45999/consoleFull)**
 for PR 9619 at commit 
[`180456c`](https://github.com/apache/spark/commit/180456ce3d25a126a74fa0f57c22665ef002b6cd).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157149637
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157194098
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157194247
  
> Test FAILed.
> Refer to this link for build results (access rights to CI server needed): 
> https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46015/
> Test FAILed.

This is a bug that will be fixed in #9707


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157179395
  
**[Test build #46019 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46019/consoleFull)**
 for PR 9619 at commit 
[`7fe9617`](https://github.com/apache/spark/commit/7fe96174edde015c8c3bc43ca734e809c5041192).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/9619#discussion_r44994208
  
--- Diff: 
network/common/src/main/java/org/apache/spark/network/util/TransportFrameDecoder.java
 ---
@@ -56,32 +56,43 @@ public void channelRead(ChannelHandlerContext ctx, 
Object data) throws Exception
   buffer = in.alloc().compositeBuffer();
 }
 
-buffer.writeBytes(in);
+buffer.addComponent(in).writerIndex(buffer.writerIndex() + 
in.readableBytes());
 
 while (buffer.isReadable()) {
-  feedInterceptor();
-  if (interceptor != null) {
-continue;
-  }
+  discardReadBytes();
+  if (!feedInterceptor()) {
+ByteBuf frame = decodeNext();
+if (frame == null) {
+  break;
+}
 
-  ByteBuf frame = decodeNext();
-  if (frame != null) {
 ctx.fireChannelRead(frame);
-  } else {
-break;
   }
 }
 
-// We can't discard read sub-buffers if there are other references to 
the buffer (e.g.
-// through slices used for framing). This assumes that code that 
retains references
-// will call retain() from the thread that called "fireChannelRead()" 
above, otherwise
-// ref counting will go awry.
-if (buffer != null && buffer.refCnt() == 1) {
+discardReadBytes();
+  }
+
+  private void discardReadBytes() {
+// If the buffer's been retained by downstream code, then make a copy 
of the remaining
+// bytes into a new buffer. Otherwise, just discard stale components.
+if (buffer.refCnt() > 1) {
--- End diff --

Yeah. Just saw `retain()` in `ChunkFetchSuccess`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157193465
  
**[Test build #46015 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46015/consoleFull)**
 for PR 9619 at commit 
[`dcdfc31`](https://github.com/apache/spark/commit/dcdfc31bfa7461c16a5a3cd17ae66cc999697664).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157193563
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46015/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157196189
  
**[Test build #46024 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46024/consoleFull)**
 for PR 9619 at commit 
[`7fe9617`](https://github.com/apache/spark/commit/7fe96174edde015c8c3bc43ca734e809c5041192).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157191902
  
LGTM pending tests.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/9619#discussion_r44986915
  
--- Diff: 
network/common/src/main/java/org/apache/spark/network/util/TransportFrameDecoder.java
 ---
@@ -56,32 +56,43 @@ public void channelRead(ChannelHandlerContext ctx, 
Object data) throws Exception
   buffer = in.alloc().compositeBuffer();
 }
 
-buffer.writeBytes(in);
+buffer.addComponent(in).writerIndex(buffer.writerIndex() + 
in.readableBytes());
 
 while (buffer.isReadable()) {
-  feedInterceptor();
-  if (interceptor != null) {
-continue;
-  }
+  discardReadBytes();
+  if (!feedInterceptor()) {
+ByteBuf frame = decodeNext();
+if (frame == null) {
+  break;
+}
 
-  ByteBuf frame = decodeNext();
-  if (frame != null) {
 ctx.fireChannelRead(frame);
-  } else {
-break;
   }
 }
 
-// We can't discard read sub-buffers if there are other references to 
the buffer (e.g.
-// through slices used for framing). This assumes that code that 
retains references
-// will call retain() from the thread that called "fireChannelRead()" 
above, otherwise
-// ref counting will go awry.
-if (buffer != null && buffer.refCnt() == 1) {
+discardReadBytes();
+  }
+
+  private void discardReadBytes() {
+// If the buffer's been retained by downstream code, then make a copy 
of the remaining
+// bytes into a new buffer. Otherwise, just discard stale components.
+if (buffer.refCnt() > 1) {
--- End diff --

@vanzin could you give a real case? Or this is just for correctness, even 
if downstream in Spark doesn't use `retain`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/9619#discussion_r44987526
  
--- Diff: 
network/common/src/main/java/org/apache/spark/network/util/TransportFrameDecoder.java
 ---
@@ -56,32 +56,43 @@ public void channelRead(ChannelHandlerContext ctx, 
Object data) throws Exception
   buffer = in.alloc().compositeBuffer();
 }
 
-buffer.writeBytes(in);
+buffer.addComponent(in).writerIndex(buffer.writerIndex() + 
in.readableBytes());
 
 while (buffer.isReadable()) {
-  feedInterceptor();
-  if (interceptor != null) {
-continue;
-  }
+  discardReadBytes();
+  if (!feedInterceptor()) {
+ByteBuf frame = decodeNext();
+if (frame == null) {
+  break;
+}
 
-  ByteBuf frame = decodeNext();
-  if (frame != null) {
 ctx.fireChannelRead(frame);
-  } else {
-break;
   }
 }
 
-// We can't discard read sub-buffers if there are other references to 
the buffer (e.g.
-// through slices used for framing). This assumes that code that 
retains references
-// will call retain() from the thread that called "fireChannelRead()" 
above, otherwise
-// ref counting will go awry.
-if (buffer != null && buffer.refCnt() == 1) {
+discardReadBytes();
+  }
+
+  private void discardReadBytes() {
+// If the buffer's been retained by downstream code, then make a copy 
of the remaining
+// bytes into a new buffer. Otherwise, just discard stale components.
+if (buffer.refCnt() > 1) {
--- End diff --

Actually Spark does use retain() when fetching shuffle blocks, and for some 
reason that causes problems. I think the real problem is somewhere in netty 
code, but this is the workaround the netty code itself uses (see 
ByteToMessageDecoder).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157193560
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157167789
  
**[Test build #46015 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46015/consoleFull)**
 for PR 9619 at commit 
[`dcdfc31`](https://github.com/apache/spark/commit/dcdfc31bfa7461c16a5a3cd17ae66cc999697664).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157204377
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157204311
  
**[Test build #46019 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46019/consoleFull)**
 for PR 9619 at commit 
[`7fe9617`](https://github.com/apache/spark/commit/7fe96174edde015c8c3bc43ca734e809c5041192).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157204379
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46019/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157220720
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46024/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157220718
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157220593
  
**[Test build #46024 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46024/consoleFull)**
 for PR 9619 at commit 
[`7fe9617`](https://github.com/apache/spark/commit/7fe96174edde015c8c3bc43ca734e809c5041192).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/9619


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-157230740
  
Merging this (master / 1.6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-156523931
  
/cc @zsxwing @rxin 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-156600994
  
**[Test build #45909 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45909/consoleFull)**
 for PR 9619 at commit 
[`180456c`](https://github.com/apache/spark/commit/180456ce3d25a126a74fa0f57c22665ef002b6cd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-156629319
  
**[Test build #45909 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45909/consoleFull)**
 for PR 9619 at commit 
[`180456c`](https://github.com/apache/spark/commit/180456ce3d25a126a74fa0f57c22665ef002b6cd).
 * This patch **fails from timeout after a configured wait of \`250m\`**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-156629520
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-156629524
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45909/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-156245083
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-156245036
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-156247639
  
**[Test build #45779 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45779/consoleFull)**
 for PR 9619 at commit 
[`8a7d194`](https://github.com/apache/spark/commit/8a7d19496c8c61f0e3e2c5180857f1773ba10372).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-156288552
  
**[Test build #45779 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45779/consoleFull)**
 for PR 9619 at commit 
[`8a7d194`](https://github.com/apache/spark/commit/8a7d19496c8c61f0e3e2c5180857f1773ba10372).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-156288677
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-156288679
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45779/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-155703979
  
**[Test build #45608 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45608/consoleFull)**
 for PR 9619 at commit 
[`ed0c1d7`](https://github.com/apache/spark/commit/ed0c1d7e6df3357344574b3ef2dccb526ed3f9b7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-155704108
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45608/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-155704106
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-155700576
  
**[Test build #45603 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45603/consoleFull)**
 for PR 9619 at commit 
[`ed0c1d7`](https://github.com/apache/spark/commit/ed0c1d7e6df3357344574b3ef2dccb526ed3f9b7).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * 
`public class JavaLBFGSExample `\n  * `class LDA @Since(\"1.6.0\") (`\n  * `  
case class Metadata(`\n  * `  require(className == expectedClassName, 
s\"Error loading metadata: Expected class name\" +`\n  * `class 
SlidingRDDPartition[T](val idx: Int, val prev: Partition, val tail: Seq[T], val 
offset: Int)`\n  * `class SlidingRDD[T: ClassTag](@transient val parent: 
RDD[T], val windowSize: Int, val step: Int)`\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-155700636
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-155700638
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45603/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[vanzin]

GitHub user vanzin opened a pull request:

https://github.com/apache/spark/pull/9619

[SPARK-11617] [network] Fix leak in TransportFrameDecoder.

The code was using the wrong API to add data to the internal composite
buffer, causing buffers to leak in certain situations. Use the right
API and enhance the tests to catch memory leaks.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vanzin/spark SPARK-11617

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9619.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9619


commit ed0c1d7e6df3357344574b3ef2dccb526ed3f9b7
Author: Marcelo Vanzin 
Date:   2015-11-11T06:00:27Z

[SPARK-11617] [network] Fix leak in TransportFrameDecoder.

The code was using the wrong API to add data to the internal composite
buffer, causing buffers to leak in certain situations. Use the right
API and enhance the tests to catch memory leaks.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-155678500
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-155678518
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-155678833
  
**[Test build #45603 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45603/consoleFull)**
 for PR 9619 at commit 
[`ed0c1d7`](https://github.com/apache/spark/commit/ed0c1d7e6df3357344574b3ef2dccb526ed3f9b7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-155680397
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-155680629
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-155680615
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11617] [network] Fix leak in TransportF...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9619#issuecomment-155680777
  
**[Test build #45608 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45608/consoleFull)**
 for PR 9619 at commit 
[`ed0c1d7`](https://github.com/apache/spark/commit/ed0c1d7e6df3357344574b3ef2dccb526ed3f9b7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org