Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18388
oh i see, it's orthogonal to the current approach. Makes sense.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18388
the idea of the queue is not to queue entire reqeusts, its just to flow
contol the # chunks being sent at once. for example you only create 5 outgoing
chunks at a time per connection, once one
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18388
OK then let's go with the flow control direction.
> For the flow control part, I'm just worrying the queue will be too large
and causing memory issue.
We can make an external
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18388
its not ok to break the shuffle service backward compatibility though.
Especially not in a minor release. We may choose to do it in like a 3.0 but
even then it makes upgrading very hard to
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18388
if it's ok to break shuffle service backward compatibility(by default this
config is off), I think we should introduce a new response type to tell the
client that, the shuffle service is still up
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18388
@tgravescs
Thanks for help.
> I think we should expand the description of the config to say what
happens when the limit is hit. Since its not using real flow control a user
might set
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18388
https://issues.apache.org/jira/browse/SPARK-21530
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18388
sorry I didn't get a chance to review this. Started but kept getting
distracted by other higher priority things. I think we should expand the
description of the config to say what happens when
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18388
Thanks for merging !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18388
thanks, merging to master!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79923/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79923 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79923/testReport)**
for PR 18388 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79923 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79923/testReport)**
for PR 18388 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79886/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79886 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79886/testReport)**
for PR 18388 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79886 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79886/testReport)**
for PR 18388 at commit
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18388
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79879/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79879 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79879/testReport)**
for PR 18388 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79879 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79879/testReport)**
for PR 18388 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79858/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79858 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79858/testReport)**
for PR 18388 at commit
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/18388
@jinxing64 Sorry, I forgot to mention one request. Could you add a unit
test? Right now it's disabled so the new codes are not tested. It will help
avoid some obvious mistakes, such as the missing
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79858 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79858/testReport)**
for PR 18388 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18388
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79857/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79857 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79857/testReport)**
for PR 18388 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79856/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79856 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79856/testReport)**
for PR 18388 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79855/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79855 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79855/testReport)**
for PR 18388 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79857 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79857/testReport)**
for PR 18388 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79856 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79856/testReport)**
for PR 18388 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79855 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79855/testReport)**
for PR 18388 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79840/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79840 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79840/testReport)**
for PR 18388 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79840 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79840/testReport)**
for PR 18388 at commit
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18388
@cloud-fan
I understand your concern. A `TransportRequestHandler` is for a
channel/connection. We want to track the sending chunks of all connections. So
I guess we must have a manager for
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79800/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79800 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79800/testReport)**
for PR 18388 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18388
a question: can we track the number of sending chunks in
`TransportRequestHandler` instead of `StreamManager`? It looks weird to me that
only `OneForOneStreamManager` does the tracking while we
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79800 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79800/testReport)**
for PR 18388 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79785/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79785 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79785/testReport)**
for PR 18388 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79785 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79785/testReport)**
for PR 18388 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79754/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79754 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79754/testReport)**
for PR 18388 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79754 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79754/testReport)**
for PR 18388 at commit
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18388
yeah it would be interesting to see if those are all active.I guess the
question still is how much memory are those using vs what the actual blocks are
using. You said the memory was being
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18388
Thanks for reply. I will figure out what I can do for this :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/18388
@jinxing64 Since
[ExternalShuffleService](https://github.com/apache/spark/blob/a0fe32a219253f0abe9d67cf178c73daf5f6fcc1/core/src/main/scala/org/apache/spark/deploy/ExternalShuffleService.scala#L55)
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18388
@tgravescs
Thanks a lot for reviewing this pr thus much. I think I'm making a stupid
mistake. Can I ask a question, how to decide the number of connections? I'm
just counting the
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18388
200k+ connections seems to be your problem then. Is this all a single
application? You say 6000 nodes with 64 executors on each host, how many cores
per executor? Or do you mean basically
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18388
We didn't change `spark.shuffle.io.numConnectionsPerPeer`. Our biggest
cluster has 6000 `NodeManager`s. There are 50 executors running on a same host
at the same time.
---
If your project is
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18388
@cloud-fan
To be honest, it's a little bit tricky to reject "open blocks" by closing
the connection. The following reconnection will surely have extra cost. In
current change we are relying
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/18388
> there are 200K+ connections and 3.5M blocks(FileSegmentManagedBuffer)
being fetched.
Did you use a large `spark.shuffle.io.numConnectionsPerPeer`? If not, the
number of connections seems
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18388
Analyzing the heap dump, there are 200K+ connections and 3.5M
blocks(`FileSegmentManagedBuffer`) being fetched. Yes, flow control is a good
idea. But I still think it make much sense to control
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18388
ok sorry I forgot you had the screenshot there. so as you mention in that
post if we are just creating to many outboundbuffers before they can actual be
sent over the network then we should try
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79280/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79280 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79280/testReport)**
for PR 18388 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18388
@jinxing64 what's the downside if we don't have the `OpenBlocksFailed`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18388
I removed the `OpenBlocksFailed` for compatibility. In current change, the
server reject the "open blocks" request by closing the connection. Then
`RetryingBlockFetcher` will retry.
---
If your
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18388
@tgravescs
As in the screenshot, we have tons of `ChunkOutboundBuffer$Entry`. Yes we
are using `transferTo`. Netty will put the `Entry`(containing reference to the
`MessageWithHeader`) into
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #79280 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79280/testReport)**
for PR 18388 at commit
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18388
So that is an issue. If users are running spark 1.6 or spark 2.1 on the
same cluster as the new one with this feature, you can't upgrade the shuffle
service until no one runs those. We run
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18388
@tgravescs
I think it's not that hurt.
In current change, new client is compatible with the old and new shuffle
service. In our clusters, we always upgrade the client first and then server
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18388
making the external shuffle service incompatible is a huge issue for
deployments. For the yarn side you would have to have the nodemanager run 2
versions (which as far as I know hasn't been
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18388
Very gentle ping @zsxwing , would you mind help comment on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18388
cc @zsxwing how strictly we require for shuffle service compatibility?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18388
Yes, there is a change. Server side may return `OpenBlocksFailed` for the
"open blocks" request, which means that old client is not compatible with new
server. Is it acceptable ?
---
If your
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18388
does this patch require server side change?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78914/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #78914 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78914/testReport)**
for PR 18388 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #78914 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78914/testReport)**
for PR 18388 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18388
LGTM. Can we add descriptions of these new configs in `configuration.md`?
thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78897/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #78897 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78897/testReport)**
for PR 18388 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #78897 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78897/testReport)**
for PR 18388 at commit
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18388
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18388
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78880/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #78880 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78880/testReport)**
for PR 18388 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18388
**[Test build #78880 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78880/testReport)**
for PR 18388 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18388
LGTM except some minor comments, thanks for working on it!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18388
@jiangxb1987
Thanks a lot for taking time review this pr.
I will read your comments very carefully and refine it.
---
If your project is set up for it, you can reply to this email and
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18388
https://user-images.githubusercontent.com/4058918/27620790-a0a27c8e-5bfe-11e7-8a78-5ebbf4839437.png;>
As the screenshot, there are tons of
`io.netty.channel.ChannelOutboundBuffer$Entry`. If I
1 - 100 of 119 matches
Mail list logo
