Do we need to create a JIRA ticket for it and list it as a known issue in 2.4.0 release notes?
On Wed, Nov 7, 2018 at 11:26 PM Shivaram Venkataraman < shiva...@eecs.berkeley.edu> wrote: > Agree with the points Felix made. > > One thing is that it looks like the only problem is vignettes and the > tests are being skipped as designed. If you see > > https://win-builder.r-project.org/incoming_pretest/SparkR_2.4.0_20181105_165757/Windows/00check.log > and > https://win-builder.r-project.org/incoming_pretest/SparkR_2.4.0_20181105_165757/Debian/00check.log > , > the tests run in 1s. > On Tue, Nov 6, 2018 at 1:29 PM Felix Cheung <felixcheun...@hotmail.com> > wrote: > > > > I’d rather not mess with 2.4.0 at this point. On CRAN is nice but users > can also install from Apache Mirror. > > > > Also I had attempted and failed to get vignettes not to build, it was > non trivial and could t get it to work. It I have an idea. > > > > As for tests I don’t know exact why is it not skipped. Need to > investigate but worse case test_package can run with 0 test. > > > > > > > > ________________________________ > > From: Sean Owen <sro...@gmail.com> > > Sent: Tuesday, November 6, 2018 10:51 AM > > To: Shivaram Venkataraman > > Cc: Felix Cheung; Wenchen Fan; Matei Zaharia; dev > > Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0 > > > > I think the second option, to skip the tests, is best right now, if > > the alternative is to have no SparkR release at all! > > Can we monkey-patch the 2.4.0 release for SparkR in this way, bless it > > from the PMC, and release that? It's drastic but so is not being able > > to release, I think. > > Right? or is CRAN not actually an important distribution path for > > SparkR in particular? > > > > On Tue, Nov 6, 2018 at 12:49 PM Shivaram Venkataraman > > <shiva...@eecs.berkeley.edu> wrote: > > > > > > Right - I think we should move on with 2.4.0. > > > > > > In terms of what can be done to avoid this error there are two > strategies > > > - Felix had this other thread about JDK 11 that should at least let > > > Spark run on the CRAN instance. In general this strategy isn't > > > foolproof because the JDK version and other dependencies on that > > > machine keep changing over time and we dont have much control over it. > > > Worse we also dont have much control > > > - The other solution is to not run code to build the vignettes > > > document and just have static code blocks there that have been > > > pre-evaluated / pre-populated. We can open a JIRA to discuss the > > > pros/cons of this ? > > > > > > Thanks > > > Shivaram > > > > > > On Tue, Nov 6, 2018 at 10:57 AM Felix Cheung < > felixcheun...@hotmail.com> wrote: > > > > > > > > We have not been able to publish to CRAN for quite some time (since > 2.3.0 was archived - the cause is Java 11) > > > > > > > > I think it’s ok to announce the release of 2.4.0 > > > > > > > > > > > > ________________________________ > > > > From: Wenchen Fan <cloud0...@gmail.com> > > > > Sent: Tuesday, November 6, 2018 8:51 AM > > > > To: Felix Cheung > > > > Cc: Matei Zaharia; Sean Owen; Spark dev list; Shivaram Venkataraman > > > > Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0 > > > > > > > > Do you mean we should have a 2.4.0 release without CRAN and then do > a 2.4.1 immediately? > > > > > > > > On Wed, Nov 7, 2018 at 12:34 AM Felix Cheung < > felixcheun...@hotmail.com> wrote: > > > >> > > > >> Shivaram and I were discussing. > > > >> Actually we worked with them before. Another possible approach is > to remove the vignettes eval and all test from the source package... in the > next release. > > > >> > > > >> > > > >> ________________________________ > > > >> From: Matei Zaharia <matei.zaha...@gmail.com> > > > >> Sent: Tuesday, November 6, 2018 12:07 AM > > > >> To: Felix Cheung > > > >> Cc: Sean Owen; dev; Shivaram Venkataraman > > > >> Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0 > > > >> > > > >> Maybe it’s wroth contacting the CRAN maintainers to ask for help? > Perhaps we aren’t disabling it correctly, or perhaps they can ignore this > specific failure. +Shivaram who might have some ideas. > > > >> > > > >> Matei > > > >> > > > >> > On Nov 5, 2018, at 9:09 PM, Felix Cheung < > felixcheun...@hotmail.com> wrote: > > > >> > > > > >> > I don¡Št know what the cause is yet. > > > >> > > > > >> > The test should be skipped because of this check > > > >> > > https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L21 > > > >> > > > > >> > And this > > > >> > > https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L57 > > > >> > > > > >> > But it ran: > > > >> > > callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", > "fit", formula, > > > >> > > > > >> > The earlier release was achived because of Java 11+ too so this > unfortunately isn¡Št new. > > > >> > > > > >> > > > > >> > From: Sean Owen <sro...@gmail.com> > > > >> > Sent: Monday, November 5, 2018 7:22 PM > > > >> > To: Felix Cheung > > > >> > Cc: dev > > > >> > Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0 > > > >> > > > > >> > What can we do to get the release through? is there any way to > > > >> > circumvent these tests or otherwise hack it? or does it need a > > > >> > maintenance release? > > > >> > On Mon, Nov 5, 2018 at 8:53 PM Felix Cheung < > felixcheun...@hotmail.com> wrote: > > > >> > > > > > >> > > FYI. SparkR submission failed. It seems to detect Java 11 > correctly with vignettes but not skipping tests as would be expected. > > > >> > > > > > >> > > Error: processing vignette ¡¥sparkr-vignettes.Rmd¡Š failed with > diagnostics: > > > >> > > Java version 8 is required for this package; found version: > 11.0.1 > > > >> > > Execution halted > > > >> > > > > > >> > > * checking PDF version of manual ... OK > > > >> > > * DONE > > > >> > > Status: 1 WARNING, 1 NOTE > > > >> > > > > > >> > > Current CRAN status: ERROR: 1, OK: 1 > > > >> > > See: < > https://CRAN.R-project.org/web/checks/check_results_SparkR.html> > > > >> > > > > > >> > > Version: 2.3.0 > > > >> > > Check: tests, Result: ERROR > > > >> > > Running ¡¥run-all.R¡Š [8s/35s] > > > >> > > Running the tests in ¡¥tests/run-all.R¡Š failed. > > > >> > > Last 13 lines of output: > > > >> > > 4: > callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", > "fit", formula, > > > >> > > data@sdf, tolower(family$family), family$link, tol, > as.integer(maxIter), weightCol, > > > >> > > regParam, as.double(var.power), as.double(link.power), > stringIndexerOrderType, > > > >> > > offsetCol) > > > >> > > 5: invokeJava(isStatic = TRUE, className, methodName, ...) > > > >> > > 6: handleErrors(returnStatus, conn) > > > >> > > 7: stop(readString(conn)) > > > >> > > > > > >> > > ùùùù testthat results > ùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùùù > > > >> > > OK: 0 SKIPPED: 0 FAILED: 2 > > > >> > > 1. Error: create DataFrame from list or data.frame > (@test_basic.R#26) > > > >> > > 2. Error: spark.glm and predict (@test_basic.R#58) > > > >> > > > > > >> > > > > > >> > > > > > >> > > ---------- Forwarded message --------- > > > >> > > Date: Mon, Nov 5, 2018, 10:12 > > > >> > > Subject: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0 > > > >> > > > > > >> > > Dear maintainer, > > > >> > > > > > >> > > package SparkR_2.4.0.tar.gz does not pass the incoming checks > automatically, please see the following pre-tests: > > > >> > > Windows: < > https://win-builder.r-project.org/incoming_pretest/SparkR_2.4.0_20181105_165757/Windows/00check.log > > > > > >> > > Status: 1 NOTE > > > >> > > Debian: < > https://win-builder.r-project.org/incoming_pretest/SparkR_2.4.0_20181105_165757/Debian/00check.log > > > > > >> > > Status: 1 WARNING, 1 NOTE > > > >> > > > > > >> > > Last released version's CRAN status: ERROR: 1, OK: 1 > > > >> > > See: < > https://CRAN.R-project.org/web/checks/check_results_SparkR.html> > > > >> > > > > > >> > > CRAN Web: <https://cran.r-project.org/package=SparkR> > > > >> > > > > > >> > > Please fix all problems and resubmit a fixed version via the > webform. > > > >> > > If you are not sure how to fix the problems shown, please ask > for help on the R-package-devel mailing list: > > > >> > > <https://stat.ethz.ch/mailman/listinfo/r-package-devel> > > > >> > > If you are fairly certain the rejection is a false positive, > please reply-all to this message and explain. > > > >> > > > > > >> > > More details are given in the directory: > > > >> > > < > https://win-builder.r-project.org/incoming_pretest/SparkR_2.4.0_20181105_165757/ > > > > > >> > > The files will be removed after roughly 7 days. > > > >> > > > > > >> > > No strong reverse dependencies to be checked. > > > >> > > > > > >> > > Best regards, > > > >> > > CRAN teams' auto-check service > > > >> > > Flavor: r-devel-linux-x86_64-debian-gcc, > r-devel-windows-ix86+x86_64 > > > >> > > Check: CRAN incoming feasibility, Result: NOTE > > > >> > > Maintainer: 'Shivaram Venkataraman <shiva...@cs.berkeley.edu>' > > > >> > > > > > >> > > New submission > > > >> > > > > > >> > > Package was archived on CRAN > > > >> > > > > > >> > > Possibly mis-spelled words in DESCRIPTION: > > > >> > > Frontend (4:10, 5:28) > > > >> > > > > > >> > > CRAN repository db overrides: > > > >> > > X-CRAN-Comment: Archived on 2018-05-01 as check problems were > not > > > >> > > corrected despite reminders. > > > >> > > > > > >> > > Flavor: r-devel-linux-x86_64-debian-gcc > > > >> > > Check: re-building of vignette outputs, Result: WARNING > > > >> > > Error in re-building vignettes: > > > >> > > ... > > > >> > > > > > >> > > Attaching package: 'SparkR' > > > >> > > > > > >> > > The following objects are masked from 'package:stats': > > > >> > > > > > >> > > cov, filter, lag, na.omit, predict, sd, var, window > > > >> > > > > > >> > > The following objects are masked from 'package:base': > > > >> > > > > > >> > > as.data.frame, colnames, colnames<-, drop, endsWith, > > > >> > > intersect, rank, rbind, sample, startsWith, subset, summary, > > > >> > > transform, union > > > >> > > > > > >> > > trying URL ' > http://mirror.klaus-uwe.me/apache/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz > ' > > > >> > > Content type 'application/octet-stream' length 227893062 bytes > (217.3 MB) > > > >> > > ================================================== > > > >> > > downloaded 217.3 MB > > > >> > > > > > >> > > Quitting from lines 65-67 (sparkr-vignettes.Rmd) > > > >> > > Error: processing vignette 'sparkr-vignettes.Rmd' failed with > diagnostics: > > > >> > > Java version 8 is required for this package; found version: > 11.0.1 > > > >> > > Execution halted > > > >> >
