Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

2018-11-05 Thread Felix Cheung
I don’t know what the cause is yet. The test should be skipped because of this check https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L21 And this https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L57 But it ran:

Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

2018-11-05 Thread Dongjoon Hyun
I'm wondering if we should change the order of publishing next time. Although it's not announced, we already have uploaded artifacts for (1), (2), (3). 1. Download: https://www-us.apache.org/dist/spark/spark-2.4.0/ 2. Maven:

Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

2018-11-05 Thread Sean Owen
What can we do to get the release through? is there any way to circumvent these tests or otherwise hack it? or does it need a maintenance release? On Mon, Nov 5, 2018 at 8:53 PM Felix Cheung wrote: > > FYI. SparkR submission failed. It seems to detect Java 11 correctly with > vignettes but not

Fwd: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

2018-11-05 Thread Felix Cheung
FYI. SparkR submission failed. It seems to detect Java 11 correctly with vignettes but not skipping tests as would be expected. Error: processing vignette ‘sparkr-vignettes.Rmd’ failed with diagnostics: Java version 8 is required for this package; found version: 11.0.1 Execution halted *

Re: need assistance debugging a strange build failure...

2018-11-05 Thread shane knapp
the maven build is quite happy: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7-ubuntu-testing/ i've wiped all the cached deps from .ivy2 on one ubuntu worker, pinned the build to that (research-jenkins-worker-08) and re-launched the sbt build. let's see if it fails

Re: need assistance debugging a strange build failure...

2018-11-05 Thread Wenchen Fan
Have you tried Maven instead of SBT? This looks like a Java dependency problem, e.g. a wrong version of Avro is picked. On Tue, Nov 6, 2018 at 8:30 AM shane knapp wrote: > i'm really close (for real: really close!) on the ubuntu port... but one > build has been a thorn in my side and i was

need assistance debugging a strange build failure...

2018-11-05 Thread shane knapp
i'm really close (for real: really close!) on the ubuntu port... but one build has been a thorn in my side and i was wondering if i could get some extra eyes on this as i grind through the remaining few pieces of my own personal system dependency hell. :) the job in question is:

spark ML PR committers

2018-11-05 Thread Ilya Matiach
Hi spark devs, Who are the main committers/reviewers for the [ML] and [MLLib] namespace currently? Could someone please review & merge this simple PR which I reviewed: https://github.com/apache/spark/pull/22087 I also have two PRs for review, one of which was approved by one reviewer back in

Integration with YARN Timeline Service v2

2018-11-05 Thread Jonathan Bender
Hello, Has there been any discussion of integrating the spark project with the updated YARN timeline service? I had seen https://issues.apache.org/jira/browse/SPARK-1537 for the original Timeline Server integration but activity seems to have stalled. Would that be the place to track any new

Re: Equivalent of emptyDataFrame in StructuredStreaming

2018-11-05 Thread Arun Manivannan
I also looked into the MemoryStream as a workaround but I do see that it is not recommended for production use. But then, since I am converting this to a DS immediately, I couldn't see a problem with mutation. Could you let me know if this is a better idea than to drop in an empty file (which is