[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225960#comment-14225960
]
zzc commented on SPARK-2468:
AH, Aaron Davidson, With patch #3465, I can run successful previo
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225790#comment-14225790
]
zzc commented on SPARK-2468:
Great. I will test it later.
> Netty-based block server / client
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225778#comment-14225778
]
Reynold Xin commented on SPARK-2468:
Glad that we are able to resolve this!
> Netty-b
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225773#comment-14225773
]
Lianhui Wang commented on SPARK-2468:
-
[~adav] that 's very great. With patch #3465, I
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225681#comment-14225681
]
Aaron Davidson commented on SPARK-2468:
---
Hey guys, I finally got a chance to run a m
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14212064#comment-14212064
]
zzc commented on SPARK-2468:
Hi, Aaron Davidson, I send a email to you about shuffle data per
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14210074#comment-14210074
]
Aaron Davidson commented on SPARK-2468:
---
Here is my spark configuration (note 32 cor
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209489#comment-14209489
]
zzc commented on SPARK-2468:
Aaron Davidson, I find that #3155 has merged into master, but
sp
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209477#comment-14209477
]
zzc commented on SPARK-2468:
Aaron Davidson, thank you for your reply. I will try it again.
C
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209407#comment-14209407
]
Aaron Davidson commented on SPARK-2468:
---
My test was significantly less strict in it
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14208013#comment-14208013
]
Lianhui Wang commented on SPARK-2468:
-
[~adav]yes i re-run test with preferDirectBufs
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14207839#comment-14207839
]
zzc commented on SPARK-2468:
Hi, Aaron Davidson, can you describe your test, including the env
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14207826#comment-14207826
]
zzc commented on SPARK-2468:
Hi, Aaron Davidson, I am sure that I ran my last test with the pa
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14207774#comment-14207774
]
Aaron Davidson commented on SPARK-2468:
---
[~zzcclp] Ah, man, that's not good. Just to
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14206178#comment-14206178
]
zzc commented on SPARK-2468:
@Aaron Davidson, I try with preferDirectBufs=true and maxUsableCo
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204222#comment-14204222
]
zzc commented on SPARK-2468:
@Aaron Davidson, Thank you for the suggestion.
The configuration
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14203316#comment-14203316
]
Lianhui Wang commented on SPARK-2468:
-
ok, thanks.[~adav] i will try to do as you say.
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14202490#comment-14202490
]
Aaron Davidson commented on SPARK-2468:
---
[~lianhuiwang] Can you try again with prefe
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14202136#comment-14202136
]
zzc commented on SPARK-2468:
The performance of Netty is worse than NIO in my test. Why?@Aaron
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14202056#comment-14202056
]
Lianhui Wang commented on SPARK-2468:
-
[~adav] yes,with https://github.com/apache/spa
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201865#comment-14201865
]
zzc commented on SPARK-2468:
Hi, Aaron Davidson, what do you mean that " Is it really the case
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201849#comment-14201849
]
Aaron Davidson commented on SPARK-2468:
---
[~zzcclp] Thank you for the writeup. Is it
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201844#comment-14201844
]
zzc commented on SPARK-2468:
By the way, My test code:
val mapR = textFile.map(line => {
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201835#comment-14201835
]
zzc commented on SPARK-2468:
Hi, Aaron Davidson, I can't download logs from server, so I just
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201832#comment-14201832
]
Aaron Davidson commented on SPARK-2468:
---
[~lianhuiwang] I have created
[#3155|https
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201812#comment-14201812
]
Aaron Davidson commented on SPARK-2468:
---
Looking at the netty code a bit more, it se
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201783#comment-14201783
]
Aaron Davidson commented on SPARK-2468:
---
Thanks a lot for those diagnostics. Can you
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201778#comment-14201778
]
Lianhui Wang commented on SPARK-2468:
-
[~zzcclp] in am's log, you can find this log:
E
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201774#comment-14201774
]
Aaron Davidson commented on SPARK-2468:
---
Yup, that would work.
> Netty-based block
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201772#comment-14201772
]
zzc commented on SPARK-2468:
aa...@databricks.com?
> Netty-based block server / client module
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201766#comment-14201766
]
Aaron Davidson commented on SPARK-2468:
---
[~zzcclp] Yes, please do. What's the memory
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201765#comment-14201765
]
zzc commented on SPARK-2468:
@Lianhui Wang, How to view the associated logs with " yarn still
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201754#comment-14201754
]
Lianhui Wang commented on SPARK-2468:
-
[~adav] when i set the spark config "spark.shuf
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201684#comment-14201684
]
zzc commented on SPARK-2468:
@Aaron Davidson, error still occur by setting the spark config
"
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201487#comment-14201487
]
zzc commented on SPARK-2468:
YES, I see it, and start to compile and test.
> Netty-based bloc
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201483#comment-14201483
]
Reynold Xin commented on SPARK-2468:
It's been merged.
> Netty-based block server / c
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201482#comment-14201482
]
zzc commented on SPARK-2468:
Thank you.
> Netty-based block server / client module
>
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201432#comment-14201432
]
Aaron Davidson commented on SPARK-2468:
---
[~zzcclp] I believe it is close to merging.
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201378#comment-14201378
]
zzc commented on SPARK-2468:
@Aaron Davidson, Thank you for your recommendation. By the way,
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200519#comment-14200519
]
Aaron Davidson commented on SPARK-2468:
---
[~zzcclp] Use of epoll mode is highly depen
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200096#comment-14200096
]
Lianhui Wang commented on SPARK-2468:
-
[~adav] i use your branch and memory overhead o
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200037#comment-14200037
]
zzc commented on SPARK-2468:
Hi, Aaron Davidson, I set spark.shuffle.blockTransferService=nett
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199945#comment-14199945
]
zzc commented on SPARK-2468:
Thank you for your reply. I will focus on it.
> Netty-based bloc
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199911#comment-14199911
]
Aaron Davidson commented on SPARK-2468:
---
This could be due to the netty transfer ser
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14196251#comment-14196251
]
zzc commented on SPARK-2468:
If I use less data, it can run successfully, such as 24G snappy f
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14195952#comment-14195952
]
zzc commented on SPARK-2468:
Yes, running on yarn client mode, but application is running, not
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14195884#comment-14195884
]
Reynold Xin commented on SPARK-2468:
Are you running on YARN? It seems like YARN just
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14189825#comment-14189825
]
zzc commented on SPARK-2468:
Hi, Reynold Xin, [SPARK-3453] Netty-based BlockTransferService, e
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187881#comment-14187881
]
zzc commented on SPARK-2468:
thanks
> Netty-based block server / client module
>
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187837#comment-14187837
]
Reynold Xin commented on SPARK-2468:
Take a look here https://cwiki.apache.org/conflue
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187832#comment-14187832
]
zzc commented on SPARK-2468:
Hi，Reynold Xin, When version 1.2 released, approximate time?
> N
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187183#comment-14187183
]
Reynold Xin commented on SPARK-2468:
Scheduled to go in in 1.2.
> Netty-based block s
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186910#comment-14186910
]
zzc commented on SPARK-2468:
Hi，Reynold Xin, What time does this issue can be solved?
I need t
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14098805#comment-14098805
]
Apache Spark commented on SPARK-2468:
-
User 'rxin' has created a pull request for this
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14094449#comment-14094449
]
Apache Spark commented on SPARK-2468:
-
User 'rxin' has created a pull request for this
55 matches
Mail list logo
