Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
Sure, thanks @felixcheung. I will let you know if I see something odds!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pr
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/18465
merged to master.
let's keep an eye on this, ok? thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
Yes, I believe these were all worth being tested.
I have veen 9 times with the current change and I believe it is enough.
---
If your project is set up for it, you can reply to this email a
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/18465
ah cool, this
https://github.com/apache/spark/pull/18465#issuecomment-313671617 was what I
meant and want to see if it recovers (or dies) properly.
have you seen enough test passes runn
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
The comments here are a bit messy. I think
https://github.com/apache/spark/pull/18465#issuecomment-313559240 summarises
all results with some links.
---
If your project is set up for it, you c
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
I rushed to read the comment. I also try to get rid of that try/catch in
c++ as below:
**R test alone**
```
vi tmp.R
```
copy and paste the codes in **Before** and *
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
Doh! Not inside try/catch. Sure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enable
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
I tested above both on MacOS and CentOS. Basically, they are same but it
is, as expected, worse on CentOS causing a crash in the master.
---
If your project is set up for it, you can reply to t
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
I referred this - http://adv-r.had.co.nz/Rcpp.html and your link.
I did as below:
**R test alone**
```
vi tmp.R
```
copy and paste the codes in **Before
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18465
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79308/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18465
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18465
**[Test build #79308 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79308/testReport)**
for PR 18465 at commit
[`c08ccd5`](https://github.com/apache/spark/commit/c
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18465
**[Test build #79308 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79308/testReport)**
for PR 18465 at commit
[`c08ccd5`](https://github.com/apache/spark/commit/c0
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
(simply rebased)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so,
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
Mac OS
- [ ] jobs with many stages -
https://github.com/apache/spark/pull/18320#issuecomment-310312766
- [ ] jobs with long stages -
https://github.com/apache/spark/pull/18320#issuecom
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
Yea, I get your point and thanks for details. Will go thought the list
first and then maybe give a shot for that
---
If your project is set up for it, you can reply to this email and have your
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/18465
hey, thanks! I think it's reasonable.
I do want to check out what happens with a native error/exception (ie.
[here](http://gallery.rcpp.org/articles/intro-to-exceptions/) a std:: exception
*
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
@felixcheung and @shivaram, do you guys see it sounds okay to go through
the checklist I used before for the current state? (of course, not asking a
sing-off but just a rough judgement)
---
If
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
Also, just checked via manually calling `stop()` as end-to-end tests after
building it.
```
df <- createDataFrame(list(list(1L, 1, "1", 0.1)), c("a", "b", "c", "d"))
collect(gap
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
@felixcheung, it looks the codes below:
```R
for(i in 0:1) {
p <- parallel:::mcfork()
if (inherits(p, "masterProcess")) {
stop("unexpected failure")
}
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
Sounds good to.check. I will be back after investigating. BTW, I guess the
original state does not handle that case too.
---
If your project is set up for it, you can reply to this email and ha
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/18465
thinking more about this, how would it handle the case when the forked
process is in a bad state?
say, if we put `stop()` in the gapply UDF. Or something more serious say
some sort of native
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
Just FYI, -9 error has occurred consistently without particularly
increasing or decreasing given my observations. I think code changes are not
particularly related with it.
---
If your project
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
Just to be clear, -9 error has occured consistently without paricularly
increasing or decreasing given my observations. I don't believe R change is
partocularly related with it.
---
If your pr
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/18465
I see, quite possibly it is bubbled up more because of that change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your proje
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
FWIW, I don't know but I guess it happens randomly in the middle of any
tests. My wild guess is it is related with triggering many tests (or maybe
rebasing a lot to trigger the build). I saw it
Github user shivaram commented on the issue:
https://github.com/apache/spark/pull/18465
@felixcheung are these failures happening from the gapply tests ? Also do
we have a way to map the error code to an error reason ?
---
If your project is set up for it, you can reply to this email
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/18465
btw, even when the earlier commit is revert seems like there's still a high
number of test failure with
`fails due to an unknown error code, -9.` - do you know how that might be
related?
-
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
> isn't being "terminated" enough? that we have to pskill it again?
Indeed, this is the point. I really tested so many times but `exit` does
not terminate (I think I might have to change
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/18465
hmm, this is a fairly small but crucial change.
One point we were discussing, earlier, we are saying that we don't know if
the child is going away, or merely slow starting, so we explicitly s
Github user shivaram commented on the issue:
https://github.com/apache/spark/pull/18465
Thanks @HyukjinKwon - I will try to look at this later tonight
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does n
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18465
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18465
**[Test build #78900 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78900/testReport)**
for PR 18465 at commit
[`9ff89a7`](https://github.com/apache/spark/commit/9
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18465
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78900/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18465
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78899/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18465
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18465
**[Test build #78899 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78899/testReport)**
for PR 18465 at commit
[`9ff89a7`](https://github.com/apache/spark/commit/9
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
If these tests all pass, 8 times with Jenkins pass. I believe this solves
the flakiness if so.
---
If your project is set up for it, you can reply to this email and have your
reply appear on Gi
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18465
**[Test build #78900 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78900/testReport)**
for PR 18465 at commit
[`9ff89a7`](https://github.com/apache/spark/commit/9f
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes s
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18465
**[Test build #78899 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78899/testReport)**
for PR 18465 at commit
[`9ff89a7`](https://github.com/apache/spark/commit/9f
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes s
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18465
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18465
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78896/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18465
**[Test build #78896 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78896/testReport)**
for PR 18465 at commit
[`9ff89a7`](https://github.com/apache/spark/commit/9
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
Also tested with this:
```r
kill <- function(children) {
lapply(children, function(child) {
pid <- parallel:::readChild(child)
if (is.integer(pid)) {
i
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
For race condition concern, I tested this as below:
```
vi tmp.R
```
copied and pasted:
```r
kill <- function(children) {
lapply(children, functio
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
To cut this short, the new approach is roughly
From
```R
exitCode <- 1
...
data <- parallel:::readChild(child)
if (is.raw(data)) {
if (unserialize(
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18465
**[Test build #78896 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78896/testReport)**
for PR 18465 at commit
[`9ff89a7`](https://github.com/apache/spark/commit/9f
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18465
cc @felixchung and @shivaram, I ran some simple tests on Mac OS (described
one in https://github.com/apache/spark/pull/18320). Likewise, I would like to
be sure before going through the checklis
50 matches
Mail list logo
