[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14145541#comment-14145541
]
Hari Shreedharan commented on SPARK-3129:
-
Sure. Thanks Matei!
> Prevent data los
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14145537#comment-14145537
]
Matei Zaharia commented on SPARK-3129:
--
Alright, in that case, this sounds pretty goo
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14145411#comment-14145411
]
Hari Shreedharan commented on SPARK-3129:
-
It is per node, single threaded.
> Pre
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14145324#comment-14145324
]
Matei Zaharia commented on SPARK-3129:
--
Is that 100 MB/s per node or in total? That s
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143775#comment-14143775
]
Hari Shreedharan commented on SPARK-3129:
-
I did multiple rounds of testing and it
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141382#comment-14141382
]
Matei Zaharia commented on SPARK-3129:
--
So Hari, what is the maximum sustainable rate
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14140049#comment-14140049
]
Hari Shreedharan commented on SPARK-3129:
-
Do these numbers look ok enough to you
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14140048#comment-14140048
]
Hari Shreedharan commented on SPARK-3129:
-
Reducing the buffer size decreases the
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14140033#comment-14140033
]
Hari Shreedharan commented on SPARK-3129:
-
So I did some benchmarking on EC2, writ
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138706#comment-14138706
]
Saisai Shao commented on SPARK-3129:
Strongly agree with Matei's comment, I think we c
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138281#comment-14138281
]
Matei Zaharia commented on SPARK-3129:
--
Great, it will be nice to see how fast this i
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138032#comment-14138032
]
Hari Shreedharan commented on SPARK-3129:
-
Thanks Matei for the background. I had
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138014#comment-14138014
]
Matei Zaharia commented on SPARK-3129:
--
Hari, have you actually benchmarked a WAL bas
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135841#comment-14135841
]
Hari Shreedharan commented on SPARK-3129:
-
As long as at least one executor contai
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135789#comment-14135789
]
Patrick Wendell commented on SPARK-3129:
I think for this it's worth considering a
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135116#comment-14135116
]
Hari Shreedharan commented on SPARK-3129:
-
It looks like Akka makes it difficult t
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129279#comment-14129279
]
Hari Shreedharan commented on SPARK-3129:
-
[~sowen] Thanks! That fixed the issue!
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129256#comment-14129256
]
Sean Owen commented on SPARK-3129:
--
[~hshreedharan] Just manually add the src dir in the
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129252#comment-14129252
]
Hari Shreedharan commented on SPARK-3129:
-
FYI here is the branch where I am doing
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14125981#comment-14125981
]
Thomas Graves commented on SPARK-3129:
--
yes that should be enough.
> Prevent data lo
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14125828#comment-14125828
]
Hari Shreedharan commented on SPARK-3129:
-
Correct me if I am wrong here, it looks
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14125810#comment-14125810
]
Hari Shreedharan commented on SPARK-3129:
-
(I am not too familiar with how UGI get
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14125790#comment-14125790
]
Hari Shreedharan commented on SPARK-3129:
-
[~tgraves] - It looks like the Security
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14125784#comment-14125784
]
Hari Shreedharan commented on SPARK-3129:
-
Hi Saisai,
You are correct that there
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122274#comment-14122274
]
Saisai Shao commented on SPARK-3129:
Hi [~hshreedharan]], thanks for your reply, is th
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122114#comment-14122114
]
Hari Shreedharan commented on SPARK-3129:
-
Looks like simply moving the code that
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122103#comment-14122103
]
Hari Shreedharan commented on SPARK-3129:
-
I am less worried about client mode, si
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122064#comment-14122064
]
Thomas Graves commented on SPARK-3129:
--
On yarn, it generates the secret automaticall
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122017#comment-14122017
]
Hari Shreedharan commented on SPARK-3129:
-
[~tgraves] - Am I correct in assuming t
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122006#comment-14122006
]
Hari Shreedharan commented on SPARK-3129:
-
Yes, so my initial goal is to be able t
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14121160#comment-14121160
]
Saisai Shao commented on SPARK-3129:
Hi [~hshreedharan], one more question:
Is your d
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105928#comment-14105928
]
Hari Shreedharan commented on SPARK-3129:
-
[~tgraves] - Thanks for the pointers. Y
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105086#comment-14105086
]
Saisai Shao commented on SPARK-3129:
Hi Hari, I have some high level questions about t
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14104077#comment-14104077
]
Thomas Graves commented on SPARK-3129:
--
Yes that probably means using reflection.
I
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14102579#comment-14102579
]
Hari Shreedharan commented on SPARK-3129:
-
The way the driver "finds" the executor
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14102504#comment-14102504
]
Thomas Graves commented on SPARK-3129:
--
A couple of random thoughts on this for yarn.
[
https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14102501#comment-14102501
]
Hari Shreedharan commented on SPARK-3129:
-
This doc is an early list of fixes. I m
37 matches
Mail list logo
