[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/16014 New PR in #16221 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/16014 Hmm looking more closely I dont think the directory being deleted / being in `dist/` is a real issue. This is because in `release-build.sh` we do `cd ..`[1] before trying to look for the file. I think just fixing the file naming thing should be sufficient. I'm going to try that first. [1] https://github.com/apache/spark/blob/202fcd21ce01393fa6dfaa1c2126e18e9b85ee96/dev/create-release/release-build.sh#L184 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/16014 I think release-tag.sh is only run while building RCs and final releases - I dont think we run it for nightly builds - So thats not getting run as a part of my test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16014 https://github.com/apache/spark/blob/202fcd21ce01393fa6dfaa1c2126e18e9b85ee96/dev/create-release/release-tag.sh I mean. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16014 For the 2nd point, I thought make-tag.sh should update the version in DESCRIPTION file. Was that not the case when we make preparations for 2.1.1? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/16014 No I found the problem - its two fold - The tgz is not copied into the `dist` directory in `make-distribution.sh` [1]. This is relatively easy to fix. It only works in Python because the entire python directory is copied into the dist directory. - The second problem is one of naming - Our jar file is called SparkR_2.1.0.tar.gz as that is the version in DESCRIPTION file in R. I think we need to do a copy at some point to create SparkR_2.1.1-SNAPSHOT.tar.gz [1] https://github.com/apache/spark/blob/202fcd21ce01393fa6dfaa1c2126e18e9b85ee96/dev/make-distribution.sh#L244 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16014 I'm guessing maybe the path needs to be a full path? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16014 This just means the source tgz file is not created? Copying and signing R source package cp: cannot stat `spark-2.1.1-SNAPSHOT-bin-hadoop2.6/R/SparkR_2.1.1-SNAPSHOT.tar.gz': No such file or directory --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/16014 Hmm - The problem is that the directory we want to get the file from gets deleted in the previous step ? I'm going to debug this locally now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/16014 Hmm still doesn't work ``` * DONE (SparkR) + popd + mkdir /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/dist/conf + cp /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/conf/docker.properties.template /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/conf/fairscheduler.xml.template /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/conf/log4j.properties.template /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/conf/metrics.properties.template /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/conf/slaves.template /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/conf/spark-defaults.conf.template /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/conf/spark-env.sh.template /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/dist/conf + cp /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/README.md /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/dist + cp -r /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/bin /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/dist + cp -r /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/python /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/dist + cp -r /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/sbin /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/dist + '[' -d /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/R/lib/SparkR ']' + mkdir -p /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/dist/R/lib + cp -r /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/R/lib/SparkR /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/dist/R/lib + cp /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/R/lib/sparkr.zip /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/dist/R/lib + '[' true == true ']' + TARDIR_NAME=spark-2.1.1-SNAPSHOT-bin-hadoop2.6 + TARDIR=/home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/spark-2.1.1-SNAPSHOT-bin-hadoop2.6 + rm -rf /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/spark-2.1.1-SNAPSHOT-bin-hadoop2.6 + cp -r /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/dist /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/spark-2.1.1-SNAPSHOT-bin-hadoop2.6 + tar czf spark-2.1.1-SNAPSHOT-bin-hadoop2.6.tgz -C /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6 spark-2.1.1-SNAPSHOT-bin-hadoop2.6 + rm -rf /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-hadoop2.6/spark-2.1.1-SNAPSHOT-bin-hadoop2.6 Copying and signing R source package cp: cannot stat `spark-2.1.1-SNAPSHOT-bin-hadoop2.6/R/SparkR_2.1.1-SNAPSHOT.tar.gz': No such file or directory ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/16014 @felixcheung I triggered a nightly build on branch-2.1 and this doesn't work correctly in the no-hadoop build. While building the vignettes we run into an error -- I'm going to change this to use another of the hadoop builds ``` ~/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-without-hadoop/R * checking for file '/home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-without-hadoop/R/pkg/DESCRIPTION' ... OK * preparing 'SparkR': * checking DESCRIPTION meta-information ... OK * installing the package to build vignettes * creating vignettes ... ERROR Attaching package: 'SparkR' The following objects are masked from 'package:stats': cov, filter, lag, na.omit, predict, sd, var, window The following objects are masked from 'package:base': as.data.frame, colnames, colnames<-, drop, intersect, rank, rbind, sample, subset, summary, transform, union Spark package found in SPARK_HOME: /home/jenkins/workspace/spark-branch-2.1-package/spark-2.1.1-SNAPSHOT-bin-without-hadoop Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:118) at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:118) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:118) at org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:104) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 7 more Quitting from lines 31-32 (sparkr-vignettes.Rmd) Error: processing vignette 'sparkr-vignettes.Rmd' failed with diagnostics: JVM is not ready after 10 seconds Execution halted ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/16014 LGTM. I took another look at `release-build.sh` and I think it looks fine. Merging this into master, branch-2.1. I'll also see if I can test this out somehow on jenkins --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16014 thanks, updated. @rxin could you please review release-build.sh? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16014 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69712/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16014 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16014 **[Test build #69712 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69712/consoleFull)** for PR 16014 at commit [`6ef26fe`](https://github.com/apache/spark/commit/6ef26fe3134880924fad03f39b4d6faa84aa05e0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16014 **[Test build #69712 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69712/consoleFull)** for PR 16014 at commit [`6ef26fe`](https://github.com/apache/spark/commit/6ef26fe3134880924fad03f39b4d6faa84aa05e0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16014 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16014 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69655/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16014 **[Test build #69655 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69655/consoleFull)** for PR 16014 at commit [`c17601b`](https://github.com/apache/spark/commit/c17601b5ef9d6f4f1ebdf965377c1f64c0645fbf). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/16014 Ah got it - I didn't notice that. BTW the way @rxin handled this for pip is a bit different in https://github.com/apache/spark/commit/37e52f8793bff306a7ae5a9aecc16f28333b70e3 Might be good to merge with upstream and handle pip, R similarly ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16014 **[Test build #69655 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69655/consoleFull)** for PR 16014 at commit [`c17601b`](https://github.com/apache/spark/commit/c17601b5ef9d6f4f1ebdf965377c1f64c0645fbf). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16014 As for release-build.sh - I had the change in there but I've changed it to make it more clear. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16014 @shivaram yes, that is correct and what I've found [here](https://github.com/apache/spark/pull/16014#issuecomment-263505953) Basically I don't think it is the R source package process to include the html files (they are still here getting created), and as explained and you pointed out, they are not actually used by R. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/16014 Sorry I was caught up with some other stuff today. Will take a final look tomm morning. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16014 Any more thought on this? Without this we don't really have a signed tarball in the official release to release to CRAN... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16014 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16014 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69315/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16014 **[Test build #69315 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69315/consoleFull)** for PR 16014 at commit [`c9c9802`](https://github.com/apache/spark/commit/c9c9802e67178e2283b1ebc9fa13f39db916773d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16014 **[Test build #69315 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69315/consoleFull)** for PR 16014 at commit [`c9c9802`](https://github.com/apache/spark/commit/c9c9802e67178e2283b1ebc9fa13f39db916773d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16014
@shivaram Cool, I did know about release-build but I didn't know it's
running on Jenkins. I *think* we should be ok but might want to check Jenkins
has "e1071" and "survival" which are optional for compatibility tests but `R
CMD check` is enforcing/requiring it. If you recall,
[this](https://github.com/apache/spark/pull/15790#issuecomment-259780799) is
this conversation that prompted this change.
@rxin This PR updates what goes into the Spark binary release to match what
we (intend to) release on CRAN for the R package
As for the diff, this is the delta between this PR and Spark 2.0.2 under
the R/lib/SparkR directory. It turns out `R CMD check` also depends on Rd file
generation in install-dev.sh (ie. `devtools::document(pkg="./pkg",
roclets=c("rd")) }`).. this is going to take more time to untangle this in a
follow up.
_what's additional_
```
SparkR/
-rw-r--r-- INDEX
drwxr-xr-x doc
SparkR/Meta/
-rw-r--r-- vignette.rds
SparkR/doc/
-rw-r--r-- sparkr-vignettes.Rmd
-rw-r--r-- sparkr-vignettes.R
-rw-r--r-- sparkr-vignettes.html
-rw-r--r-- index.html
```
_what's omitted_
```
SparkR/html/
-rw-r--r-- 1 root root 1319 Nov 29 08:05 R.css
-rw-r--r-- 1 root root 81153 Nov 29 08:05 00Index.html
```
What it used to have `year.html` `write.parquet.html`
`sparkR.session.html`, the html directory now only has 2 files. My
understanding is these knitr html output are actually not used at runtime. I
checked that `?sparkR.session` in the `sparkR` is still working correctly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16014 Why is this a blocker thing? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/16014 Just to clarify the Jenkins comment, we use Jenkins to test PRs and that doesn't run make-distribution.sh -- But the main release builds (say 2.0.2[2]) are also built on Jenkins by a separate job that calls just `release-build.sh package`[1] with the appropriate environment variables. [1] https://github.com/apache/spark/blob/master/dev/create-release/release-build.sh#L149 [2]https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Release/job/spark-release-package/57/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16014 I thought we don't run `make-distribution.sh` on Jenkins, so I have been testing it manually/locally. As per [here](https://github.com/apache/spark/pull/16014#discussion_r89912061) the output of install is different - I'll come back to list that in a bit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16014 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16014 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69175/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16014 **[Test build #69175 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69175/consoleFull)** for PR 16014 at commit [`7977139`](https://github.com/apache/spark/commit/79771392f7a8c7fe4ed90b20aec05e5e65304975). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16014 @shivaram --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16014 **[Test build #69175 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69175/consoleFull)** for PR 16014 at commit [`7977139`](https://github.com/apache/spark/commit/79771392f7a8c7fe4ed90b20aec05e5e65304975). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
