[GitHub] spark pull request #16248: [SPARK-18810][SPARKR] SparkR install.spark does n...

2016-12-12 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/16248


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16248: [SPARK-18810][SPARKR] SparkR install.spark does n...

2016-12-12 Thread shivaram 
Github user shivaram commented on a diff in the pull request:

https://github.com/apache/spark/pull/16248#discussion_r91997384
  
--- Diff: R/pkg/R/utils.R ---
@@ -851,3 +851,12 @@ rbindRaws <- function(inputData){
   out[!rawcolumns] <- lapply(out[!rawcolumns], unlist)
   out
 }
+
+# Get basename without extension from URL
+basenameSansExtFromUrl <- function(url) {
--- End diff --

Got it - yeah it sounds good to remove compression and just copy the two 
lines of code.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16248: [SPARK-18810][SPARKR] SparkR install.spark does n...

2016-12-11 Thread felixcheung 
Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/16248#discussion_r91891490
  
--- Diff: R/pkg/R/utils.R ---
@@ -851,3 +851,12 @@ rbindRaws <- function(inputData){
   out[!rawcolumns] <- lapply(out[!rawcolumns], unlist)
   out
 }
+
+# Get basename without extension from URL
+basenameSansExtFromUrl <- function(url) {
--- End diff --

My concern was to bring in another dependencies just for this (it's in the 
tools)

The regex was in fact copy-paste from file_path_sans_ext (hence the name) 
except for the compression part which is what you are referring to.
I could copy that over as well. Would you prefer `compression` be TRUE 
(default is FALSE) to remove `.gz`?

```
> library(tools)
> file_path_sans_ext
function (x, compression = FALSE)
{
if (compression)
x <- sub("[.](gz|bz2|xz)$", "", x)
sub("([^.]+)\\.[[:alnum:]]+$", "\\1", x)
}
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16248: [SPARK-18810][SPARKR] SparkR install.spark does n...

2016-12-11 Thread shivaram 
Github user shivaram commented on a diff in the pull request:

https://github.com/apache/spark/pull/16248#discussion_r91883327
  
--- Diff: R/pkg/R/utils.R ---
@@ -851,3 +851,12 @@ rbindRaws <- function(inputData){
   out[!rawcolumns] <- lapply(out[!rawcolumns], unlist)
   out
 }
+
+# Get basename without extension from URL
+basenameSansExtFromUrl <- function(url) {
--- End diff --

can we use file_path_sans_ext [1] for removing the extension ? I worry we 
might publish it as `.tar.gz` someday and then removing just the last `.` will 
be insufficient

[1] 
https://stat.ethz.ch/R-manual/R-patched/library/tools/html/fileutils.html


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16248: [SPARK-18810][SPARKR] SparkR install.spark does n...

2016-12-11 Thread shivaram 
Github user shivaram commented on a diff in the pull request:

https://github.com/apache/spark/pull/16248#discussion_r91883219
  
--- Diff: R/pkg/R/install.R ---
@@ -104,7 +113,12 @@ install.spark <- function(hadoopVersion = "2.7", 
mirrorUrl = NULL,
   if (tarExists && !overwrite) {
 message("tar file found.")
   } else {
-robustDownloadTar(mirrorUrl, version, hadoopVersion, packageName, 
packageLocalPath)
+if (releaseUrl != "") {
+  message("Downloading from alternate URL:\n- ", releaseUrl)
+  downloadUrl(releaseUrl, packageLocalPath, paste0("Fetch failed from 
", mirrorUrl))
--- End diff --

this should be `releaseUrl` instead of `mirrorUrl` in the `paste0` ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16248: [SPARK-18810][SPARKR] SparkR install.spark does n...

2016-12-11 Thread felixcheung 
Github user felixcheung closed the pull request at:

https://github.com/apache/spark/pull/16248


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16248: [SPARK-18810][SPARKR] SparkR install.spark does n...

2016-12-11 Thread felixcheung 
GitHub user felixcheung reopened a pull request:

https://github.com/apache/spark/pull/16248

[SPARK-18810][SPARKR] SparkR install.spark does not work for RCs, snapshots

## What changes were proposed in this pull request?

Support overriding the download url (include version directory) in an 
environment variable, `SPARKR_RELEASE_DOWNLOAD_URL`

## How was this patch tested?

unit test, manually testing
- snapshot build url
  - download when spark jar not cached
  - when spark jar is cached
- RC build url
  - download when spark jar not cached
  - when spark jar is cached
- multiple cached spark version


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark rinstallurl

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16248.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16248


commit 7c1d95b496784fd7db82e91e207323c071ccda4d
Author: Felix Cheung 
Date:   2016-12-10T00:07:25Z

changes to support env override url

commit 32a010afb611130cad1b12b5d13f50cde95ba74b
Author: Felix Cheung 
Date:   2016-12-11T07:08:46Z

change to setup path correctly

commit b5d58b1f4ce3dc9dd9ea2108679c5693235ba329
Author: Felix Cheung 
Date:   2016-12-11T07:38:51Z

missed in tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16248: [SPARK-18810][SPARKR] SparkR install.spark does n...

2016-12-11 Thread felixcheung 
GitHub user felixcheung reopened a pull request:

https://github.com/apache/spark/pull/16248

[SPARK-18810][SPARKR] SparkR install.spark does not work for RCs, snapshots

## What changes were proposed in this pull request?

Support overriding the download url (include version directory) in an 
environment variable, `SPARKR_RELEASE_DOWNLOAD_URL`

## How was this patch tested?

unit test, manually testing
- snapshot build url
  - download when spark jar not cached
  - when spark jar is cached
- RC build url
  - download when spark jar not cached
  - when spark jar is cached
- multiple cached spark version


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark rinstallurl

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16248.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16248


commit 7c1d95b496784fd7db82e91e207323c071ccda4d
Author: Felix Cheung 
Date:   2016-12-10T00:07:25Z

changes to support env override url

commit 32a010afb611130cad1b12b5d13f50cde95ba74b
Author: Felix Cheung 
Date:   2016-12-11T07:08:46Z

change to setup path correctly

commit b5d58b1f4ce3dc9dd9ea2108679c5693235ba329
Author: Felix Cheung 
Date:   2016-12-11T07:38:51Z

missed in tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16248: [SPARK-18810][SPARKR] SparkR install.spark does n...

2016-12-11 Thread felixcheung 
Github user felixcheung closed the pull request at:

https://github.com/apache/spark/pull/16248


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16248: [SPARK-18810][SPARKR] SparkR install.spark does n...

2016-12-10 Thread felixcheung 
GitHub user felixcheung opened a pull request:

https://github.com/apache/spark/pull/16248

[SPARK-18810][SPARKR] SparkR install.spark does not work for RCs, snapshots

## What changes were proposed in this pull request?

Support overriding the download url (include version directory) in an 
environment variable, `SPARKR_RELEASE_DOWNLOAD_URL`

## How was this patch tested?

unit test, manually testing
- snapshot build url
  - download when spark jar not cached
  - when spark jar is cached
- RC build url
  - download when spark jar not cached
  - when spark jar is cached
- multiple cached spark version


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark rinstallurl

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16248.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16248


commit 7c1d95b496784fd7db82e91e207323c071ccda4d
Author: Felix Cheung 
Date:   2016-12-10T00:07:25Z

changes to support env override url

commit 32a010afb611130cad1b12b5d13f50cde95ba74b
Author: Felix Cheung 
Date:   2016-12-11T07:08:46Z

change to setup path correctly




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org