[GitHub] spark pull request #16141: [MINOR][DOC] Use SparkR `TRUE` value and add defa...

[dongjoon-hyun]

GitHub user dongjoon-hyun opened a pull request:

https://github.com/apache/spark/pull/16141

[MINOR][DOC] Use SparkR `TRUE` value and add default values for 
`StructField` in SQL Guide.

## What changes were proposed in this pull request?

In `SQL Programming Guide`, this PR uses `TRUE` instead of `True` in SparkR 
and adds default values for `StructField`.

## How was this patch tested?

Manual.

```
cd docs
SKIP_API=1 jekyll build
open _site/index.html
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dongjoon-hyun/spark SPARK-SQL-GUIDE

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16141.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16141


commit aaab0d703357898d677cf8a880c1c9dd7f4a7d80
Author: Dongjoon Hyun 
Date:   2016-12-05T07:51:51Z

[MINOR][DOC] Use SparkR `TRUE` value and add default values for 
`StructField` in SQL Guide.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16030
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69659/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16030
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16030
  
**[Test build #69659 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69659/consoleFull)**
 for PR 16030 at commit 
[`87e3607`](https://github.com/apache/spark/commit/87e3607e70d8f6fb530213c1821b25064a4c80fa).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15664: [SPARK-18123][SQL] Use db column names instead of RDD co...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15664
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69658/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15664: [SPARK-18123][SQL] Use db column names instead of RDD co...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15664
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15664: [SPARK-18123][SQL] Use db column names instead of RDD co...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15664
  
**[Test build #69658 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69658/consoleFull)**
 for PR 15664 at commit 
[`528ccfe`](https://github.com/apache/spark/commit/528ccfe7c95f1a4c7be8ae4c71ab1c1bae2417bd).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16135: [SPARK-18700][SQL] Add ReadWriteLock for each table's re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16135
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69660/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16135: [SPARK-18700][SQL] Add ReadWriteLock for each table's re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16135
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16135: [SPARK-18700][SQL] Add ReadWriteLock for each table's re...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16135
  
**[Test build #69660 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69660/consoleFull)**
 for PR 16135 at commit 
[`95aabb8`](https://github.com/apache/spark/commit/95aabb8dd97419cc4532a90225381ccd7de8b19b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15664: [SPARK-18123][SQL] Use db column names instead of RDD co...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/15664
  
Yes, right! Thank you, @gatorsmile !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16139: [SPARK-18705][ML][DOC] Update user guide to reflect one ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16139
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16139: [SPARK-18705][ML][DOC] Update user guide to reflect one ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16139
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69661/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16139: [SPARK-18705][ML][DOC] Update user guide to reflect one ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16139
  
**[Test build #69661 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69661/consoleFull)**
 for PR 16139 at commit 
[`c39aa6d`](https://github.com/apache/spark/commit/c39aa6d54c7cd601e3dd2b499204d0c6f756606e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15664: [SPARK-18123][SQL] Use db column names instead of RDD co...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/15664
  
This is a bug fix, right? Will review this tomorrow. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16117: [SPARK-18686][SparkR][ML] Several cleanup and imp...

[wangmiao1981]

Github user wangmiao1981 commented on a diff in the pull request:

https://github.com/apache/spark/pull/16117#discussion_r90808362
  
--- Diff: R/pkg/R/mllib.R ---
@@ -817,44 +804,29 @@ setMethod("predict", signature(object = 
"LogisticRegressionModel"),
 #  Get the summary of an LogisticRegressionModel
 
 #' @param object an LogisticRegressionModel fitted by \code{spark.logit}
-#' @return \code{summary} returns the Binary Logistic regression results 
of a given model as list,
-#' including roc, areaUnderROC, pr, fMeasureByThreshold, 
precisionByThreshold,
-#' recallByThreshold, totalIterations, objectiveHistory. Note that 
Multinomial logistic
-#' regression summary is not available now.
+#' @return \code{summary} returns coefficients matrix of the fitted model
 #' @rdname spark.logit
 #' @aliases summary,LogisticRegressionModel-method
 #' @export
 #' @note summary(LogisticRegressionModel) since 2.1.0
 setMethod("summary", signature(object = "LogisticRegressionModel"),
   function(object) {
 jobj <- object@jobj
-is.loaded <- callJMethod(jobj, "isLoaded")
-
-if (is.loaded) {
-  stop("Loaded model doesn't have training summary.")
+features <- callJMethod(jobj, "rFeatures")
+labels <- callJMethod(jobj, "labels")
+coefficients <- callJMethod(jobj, "rCoefficients")
+nCol <- length(coefficients) / length(features)
--- End diff --

Sounds good.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16140: [SPARK-18714][SQL] Add a simple time function to SparkSe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16140
  
**[Test build #69662 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69662/consoleFull)**
 for PR 16140 at commit 
[`76d2f53`](https://github.com/apache/spark/commit/76d2f5319e5ec65bdbeb3ba0467b60f3e2733f76).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16140: [SPARK-18714][SQL] Add a simple time function to SparkSe...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/16140
  
cc @marmbrus 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16140: [SPARK-18714][SQL] Add a simple time function to ...

[rxin]

GitHub user rxin opened a pull request:

https://github.com/apache/spark/pull/16140

[SPARK-18714][SQL] Add a simple time function to SparkSession

## What changes were proposed in this pull request?
Many Spark developers often want to test the runtime of some function in 
interactive debugging and testing. This patch adds a simple time function to 
SparkSession:

```
scala> spark.time { spark.range(1000).count() }
Time taken: 77 ms   

res1: Long = 1000
```

## How was this patch tested?
I tested this interactively in spark-shell.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rxin/spark SPARK-18714

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16140.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16140


commit 76d2f5319e5ec65bdbeb3ba0467b60f3e2733f76
Author: Reynold Xin 
Date:   2016-12-05T06:10:51Z

[SPARK-18714][SQL] Add a simple time function to SparkSession




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16139: [SPARK-18705][ML][DOC] Update user guide to reflect one ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16139
  
**[Test build #69661 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69661/consoleFull)**
 for PR 16139 at commit 
[`c39aa6d`](https://github.com/apache/spark/commit/c39aa6d54c7cd601e3dd2b499204d0c6f756606e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16133: [SPARK-18702][SQL] input_file_block_start and inp...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/16133


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16139: [SPARK-18705][ML][DOC] Update user guide to refle...

[sethah]

GitHub user sethah opened a pull request:

https://github.com/apache/spark/pull/16139

[SPARK-18705][ML][DOC] Update user guide to reflect one pass solver for L1 
and elastic-net

## What changes were proposed in this pull request?

WeightedLeastSquares now supports L1 and elastic net penalties and has an 
additional solver option: QuasiNewton. The docs are updated to reflect this 
change.

## How was this patch tested?

Docs only. Generated documentation to make sure Latex looks ok.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sethah/spark SPARK-18705

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16139.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16139


commit c39aa6d54c7cd601e3dd2b499204d0c6f756606e
Author: sethah 
Date:   2016-12-05T05:46:41Z

update user guide




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16133: [SPARK-18702][SQL] input_file_block_start and input_file...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/16133
  
Thanks - going to merge this in master.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16135: [SPARK-18700][SQL] Add ReadWriteLock for each table's re...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16135
  
**[Test build #69660 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69660/consoleFull)**
 for PR 16135 at commit 
[`95aabb8`](https://github.com/apache/spark/commit/95aabb8dd97419cc4532a90225381ccd7de8b19b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16030
  
**[Test build #69659 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69659/consoleFull)**
 for PR 16030 at commit 
[`87e3607`](https://github.com/apache/spark/commit/87e3607e70d8f6fb530213c1821b25064a4c80fa).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14638: [SPARK-11374][SQL] Support `skip.header.line.count` opti...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/14638
  
Hi, @rxin , @srowen , @yhuai .
If there is something for me to do more for this issue, please let me know.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15664: [SPARK-18123][SQL] Use db column names instead of RDD co...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15664
  
**[Test build #69658 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69658/consoleFull)**
 for PR 15664 at commit 
[`528ccfe`](https://github.com/apache/spark/commit/528ccfe7c95f1a4c7be8ae4c71ab1c1bae2417bd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15664: [SPARK-18123][SQL] Use db column names instead of RDD co...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/15664
  
Thank you for review, @viirya .
I'll update like that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16014
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16014
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69655/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16014
  
**[Test build #69655 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69655/consoleFull)**
 for PR 16014 at commit 
[`c17601b`](https://github.com/apache/spark/commit/c17601b5ef9d6f4f1ebdf965377c1f64c0645fbf).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16077: [SPARK-18643][SPARKR] SparkR hangs at session sta...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/16077


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...

[shivaram]

Github user shivaram commented on the issue:

https://github.com/apache/spark/pull/16014
  
Ah got it - I didn't notice that. BTW the way @rxin handled this for pip is 
a bit different in 
https://github.com/apache/spark/commit/37e52f8793bff306a7ae5a9aecc16f28333b70e3

Might be good to merge with upstream and handle pip, R similarly ? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16077: [SPARK-18643][SPARKR] SparkR hangs at session start when...

[shivaram]

Github user shivaram commented on the issue:

https://github.com/apache/spark/pull/16077
  
LGTM. Merging this to master, branch-2.1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16030
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69656/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16030
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16030
  
**[Test build #69656 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69656/consoleFull)**
 for PR 16030 at commit 
[`9938de5`](https://github.com/apache/spark/commit/9938de57927756981f3ac672e0f62a67b28df9ee).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16059: [SPARK-18625][ML] OneVsRestModel should support setFeatu...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16059
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69657/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16059: [SPARK-18625][ML] OneVsRestModel should support setFeatu...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16059
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16059: [SPARK-18625][ML] OneVsRestModel should support setFeatu...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16059
  
**[Test build #69657 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69657/consoleFull)**
 for PR 16059 at commit 
[`b45f558`](https://github.com/apache/spark/commit/b45f558e4aa5799f2e00d87ab0490ec6755ecc73).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

[merlintang]

Github user merlintang commented on the issue:

https://github.com/apache/spark/pull/15819
  
@cloud-fan  this is related to this PR in the 2.0.x
https://github.com/apache/spark/pull/12770



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15998: [SPARK-18572][SQL] Add a method `listPartitionNames` to ...

[mallman]

Github user mallman commented on the issue:

https://github.com/apache/spark/pull/15998
  
@cloud-fan That's unfortunate if it's going to block this PR. How do we 
proceed?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/15819
  
ok @merlintang can you find out which PR adds these codes to 2.0? Then 
other people can know what we are backporting in this PR


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15998: [SPARK-18572][SQL] Add a method `listPartitionNames` to ...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/15998
  
@mallman ah it's a mistake, check 
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala#L228
 , we need to move this to place before 
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala#L198,
 so that the static partitions are normalized.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15664: [SPARK-18123][SQL] Use db column names instead of...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/15664#discussion_r90799712
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
 ---
@@ -112,7 +112,25 @@ object JdbcUtils extends Logging {
*/
   def insertStatement(conn: Connection, table: String, rddSchema: 
StructType, dialect: JdbcDialect)
   : PreparedStatement = {
-val columns = rddSchema.fields.map(x => 
dialect.quoteIdentifier(x.name)).mkString(",")
+// Use database column names instead of RDD schema column names
+val tableSchemaQuery = 
conn.prepareStatement(dialect.getSchemaQuery(table))
+var columns: String = ""
+try {
+  val tableSchema = getSchema(tableSchemaQuery.executeQuery(), dialect)
+  val nameMap = tableSchema.fields.map(f => f.name -> f.name).toMap
+  val lowercaseNameMap = tableSchema.fields.map(f => 
f.name.toLowerCase -> f.name).toMap
+  columns = rddSchema.fields.map { x =>
+if (nameMap.isDefinedAt(x.name)) {
+  dialect.quoteIdentifier(x.name)
+} else if (lowercaseNameMap.isDefinedAt(x.name.toLowerCase)) {
+  dialect.quoteIdentifier(nameMap(x.name.toLowerCase))
--- End diff --

nameMap -> lowercaseNameMap.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15994: [SPARK-18555][SQL]DataFrameNaFunctions.fill miss up orig...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15994
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69653/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15994: [SPARK-18555][SQL]DataFrameNaFunctions.fill miss up orig...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15994
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15994: [SPARK-18555][SQL]DataFrameNaFunctions.fill miss up orig...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15994
  
**[Test build #69653 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69653/consoleFull)**
 for PR 15994 at commit 
[`431038f`](https://github.com/apache/spark/commit/431038fe23cfca6813f2952f4d6c94c914b483f1).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16059: [SPARK-18625][ML] OneVsRestModel should support setFeatu...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16059
  
**[Test build #69657 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69657/consoleFull)**
 for PR 16059 at commit 
[`b45f558`](https://github.com/apache/spark/commit/b45f558e4aa5799f2e00d87ab0490ec6755ecc73).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16059: [SPARK-18625][ML] OneVsRestModel should support setFeatu...

[zhengruifeng]

Github user zhengruifeng commented on the issue:

https://github.com/apache/spark/pull/16059
  
@yanboliang Updated. Thanks for reviewing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15987: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION sho...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/15987#discussion_r90797605
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 ---
@@ -661,6 +661,36 @@ class Analyzer(
   // rule: ResolveDeserializer.
   case plan if containsDeserializer(plan.expressions) => plan
 
+  // Some commands (AlterTableDropPartitionCommand) use expressions, 
so we need to resolve.
+  case c: CommandWithExpression =>
+val catalogTable = catalog.getTableMetadata(c.getTableName)
+
+// We should not do this here in order to support general 
CommandWithExpression
+if (catalogTable.tableType == CatalogTableType.VIEW) {
+  throw new AnalysisException(
+"Cannot alter a view with ALTER TABLE. Please use ALTER VIEW 
instead")
--- End diff --

Except `AlterTableDropPartitionCommand`, do we have other running commands 
which need this resolving? If not, I think we may not need to add this to 
`Analyzer` and create new abstraction `CommandWithExpression`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16077: [SPARK-18643][SPARKR] SparkR hangs at session start when...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16077
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16077: [SPARK-18643][SPARKR] SparkR hangs at session start when...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16077
  
**[Test build #69654 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69654/consoleFull)**
 for PR 16077 at commit 
[`98f3250`](https://github.com/apache/spark/commit/98f32502c4a9f649f4c859568e6be0e3b0cb5697).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16077: [SPARK-18643][SPARKR] SparkR hangs at session start when...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16077
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69654/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16030
  
**[Test build #69656 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69656/consoleFull)**
 for PR 16030 at commit 
[`9938de5`](https://github.com/apache/spark/commit/9938de57927756981f3ac672e0f62a67b28df9ee).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

[merlintang]

Github user merlintang commented on the issue:

https://github.com/apache/spark/pull/15819
  
Ok.

On Sun, Dec 4, 2016 at 6:25 PM, Reynold Xin 
wrote:

> We have stopped making new releases for 1.5 so it makes no sense to
> backport.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/15819
  
We have stopped making new releases for 1.5 so it makes no sense to 
backport.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

[merlintang]

Github user merlintang commented on the issue:

https://github.com/apache/spark/pull/15819
  
this bug is related to 1.5.x as well as 1.6.x.  please backport to 1.5.x as
well.

On Sun, Dec 4, 2016 at 6:20 PM, Reynold Xin 
wrote:

> If it is a bug fix and low risk, sure.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/15819
  
If it is a bug fix and low risk, sure.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

[maropu]

Github user maropu commented on the issue:

https://github.com/apache/spark/pull/16030
  
okay, I'll do


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16014
  
**[Test build #69655 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69655/consoleFull)**
 for PR 16014 at commit 
[`c17601b`](https://github.com/apache/spark/commit/c17601b5ef9d6f4f1ebdf965377c1f64c0645fbf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15998: [SPARK-18572][SQL] Add a method `listPartitionNames` to ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15998
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15998: [SPARK-18572][SQL] Add a method `listPartitionNames` to ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15998
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69652/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16133: [SPARK-18702][SQL] input_file_block_start and inp...

[ueshin]

Github user ueshin commented on a diff in the pull request:

https://github.com/apache/spark/pull/16133#discussion_r90794591
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala ---
@@ -533,31 +533,54 @@ class ColumnExpressionSuite extends QueryTest with 
SharedSQLContext {
 )
   }
 
-  test("input_file_name - FileScanRDD") {
+  test("input_file_name, input_file_block_start, input_file_block_length - 
FileScanRDD") {
 withTempPath { dir =>
   val data = sparkContext.parallelize(0 to 10).toDF("id")
   data.write.parquet(dir.getCanonicalPath)
-  val answer = 
spark.read.parquet(dir.getCanonicalPath).select(input_file_name())
-.head.getString(0)
-  assert(answer.contains(dir.getCanonicalPath))
 
-  checkAnswer(data.select(input_file_name()).limit(1), Row(""))
+  // Test the 3 expressions when reading from files
+  val q = spark.read.parquet(dir.getCanonicalPath).select(
+input_file_name(), expr("input_file_block_start()"), 
expr("input_file_block_length()"))
--- End diff --

I see.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15998: [SPARK-18572][SQL] Add a method `listPartitionNames` to ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15998
  
**[Test build #69652 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69652/consoleFull)**
 for PR 15998 at commit 
[`095cdd5`](https://github.com/apache/spark/commit/095cdd51c386e1747cc8645851b0cdffa83a5a70).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/16014
  
As for release-build.sh - I had the change in there but I've changed it to 
make it more clear.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16077: [SPARK-18643][SPARKR] SparkR hangs at session start when...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16077
  
**[Test build #69654 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69654/consoleFull)**
 for PR 16077 at commit 
[`98f3250`](https://github.com/apache/spark/commit/98f32502c4a9f649f4c859568e6be0e3b0cb5697).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/16014
  
@shivaram yes, that is correct and what I've found 
[here](https://github.com/apache/spark/pull/16014#issuecomment-263505953)

Basically I don't think it is the R source package process to include the 
html files (they are still here getting created), and as explained and you 
pointed out, they are not actually used by R.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15994: [SPARK-18555][SQL]DataFrameNaFunctions.fill miss up orig...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15994
  
**[Test build #69653 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69653/consoleFull)**
 for PR 15994 at commit 
[`431038f`](https://github.com/apache/spark/commit/431038fe23cfca6813f2952f4d6c94c914b483f1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16077: [SPARK-18643][SPARKR] SparkR hangs at session start when...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/16077
  
Right, I was just reviewing possible code paths for this in the last few 
days and I'm pretty confident that this change will not run install.spark in 
cluster modes (which would have Spark/JVM already running).

Also you are right, I just found that we didn't really talk about how 
install.spark would be called in sparkR.session() - I'll add that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16014: [SPARK-18590][SPARKR] build R source package when...

[shivaram]

Github user shivaram commented on a diff in the pull request:

https://github.com/apache/spark/pull/16014#discussion_r90792118
  
--- Diff: R/check-cran.sh ---
@@ -82,4 +83,20 @@ else
   # This will run tests and/or build vignettes, and require SPARK_HOME
   SPARK_HOME="${SPARK_HOME}" "$R_SCRIPT_PATH/"R CMD check 
$CRAN_CHECK_OPTIONS SparkR_"$VERSION".tar.gz
 fi
+
+# Install source package to get it to generate vignettes rds files, etc.
+if [ -n "$CLEAN_INSTALL" ]
--- End diff --

So I did the diff. Here are the new files in the output of 
`make-distribution` in master branch with this change vs. 2.0.0 
Files Added: 

```
- R/lib/SparkR/Meta/vignette.rds
- /R/lib/SparkR/doc/
- /R/lib/SparkR/doc/index.html
- /R/lib/SparkR/doc/sparkr-vignettes.R
- /R/lib/SparkR/doc/sparkr-vignettes.Rmd
- /R/lib/SparkR/doc/sparkr-vignettes.html
```

Files removed: A bunch of HTML files starting from
```
/R/lib/SparkR/html/AFTSurvivalRegressionModel-class.html
...
/R/lib/SparkR/html/year.html
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16014: [SPARK-18590][SPARKR] build R source package when...

[shivaram]

Github user shivaram commented on a diff in the pull request:

https://github.com/apache/spark/pull/16014#discussion_r90790181
  
--- Diff: dev/make-distribution.sh ---
@@ -208,11 +212,24 @@ cp -r "$SPARK_HOME/data" "$DISTDIR"
 # Make pip package
 if [ "$MAKE_PIP" == "true" ]; then
   echo "Building python distribution package"
-  cd $SPARK_HOME/python
+  pushd "$SPARK_HOME/python" > /dev/null
   python setup.py sdist
-  cd ..
+  popd > /dev/null
+else
+  echo "Skipping building python distribution package"
+fi
+
+# Make R package - this is used for both CRAN release and packing R layout 
into distribution
+if [ "$MAKE_R" == "true" ]; then
+  echo "Building R source package"
+  pushd "$SPARK_HOME/R" > /dev/null
+  # Build source package and run full checks
+  # Install source package to get it to generate vignettes, etc.
+  # Do not source the check-cran.sh - it should be run from where it is 
for it to set SPARK_HOME
+  NO_TESTS=1 CLEAN_INSTALL=1 "$SPARK_HOME/"R/check-cran.sh
--- End diff --

Yeah longer term that sounds like a good idea.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16014: [SPARK-18590][SPARKR] build R source package when...

[shivaram]

Github user shivaram commented on a diff in the pull request:

https://github.com/apache/spark/pull/16014#discussion_r90792416
  
--- Diff: R/check-cran.sh ---
@@ -82,4 +83,20 @@ else
   # This will run tests and/or build vignettes, and require SPARK_HOME
   SPARK_HOME="${SPARK_HOME}" "$R_SCRIPT_PATH/"R CMD check 
$CRAN_CHECK_OPTIONS SparkR_"$VERSION".tar.gz
 fi
+
+# Install source package to get it to generate vignettes rds files, etc.
+if [ -n "$CLEAN_INSTALL" ]
--- End diff --

So it looks like we lost the knitted HTML files in the SparkR package with 
this change. FWIW this may not be bad as the html files are not usually used 
locally and only used for the website and I think the docs creation part of the 
build should pick that up. (Verifying that now)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16014: [SPARK-18590][SPARKR] build R source package when...

[shivaram]

Github user shivaram commented on a diff in the pull request:

https://github.com/apache/spark/pull/16014#discussion_r90792485
  
--- Diff: dev/make-distribution.sh ---
@@ -71,6 +72,9 @@ while (( "$#" )); do
 --pip)
   MAKE_PIP=true
   ;;
+--r)
+  MAKE_R=true
--- End diff --

FWIW if you want this to get picked up by the official release building 
procedure we also need to edit release-build.sh [1]. Can you coordinate this 
with @rxin ?

[1] 
https://github.com/apache/spark/blob/d9eb4c7215f26dd05527c0b9980af35087ab9d64/dev/create-release/release-build.sh#L220


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15998: [SPARK-18572][SQL] Add a method `listPartitionNames` to ...

[mallman]

Github user mallman commented on the issue:

https://github.com/apache/spark/pull/15998
  
> @mallman do you know which tests fail the partition spec checking? It 
looks to me that before we call partition related API in SessionCatalog, the 
partition column names should be normalized already.


https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69581/testReport/junit/org.apache.spark.sql.hive/PartitionProviderCompatibilitySuite/SPARK_18659_insert_overwrite_table_with_lowercase___partition_management_true/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

[brkyvz]

Github user brkyvz commented on the issue:

https://github.com/apache/spark/pull/16030
  
@maropu There seems to be a merge conflict. Should be pretty trivial to 
resolve. Mind taking a look when you can?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15998: [SPARK-18572][SQL] Add a method `listPartitionNames` to ...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/15998
  
@mallman do you know which tests fail the partition spec checking? It looks 
to me that before we call partition related API in `SessionCatalog`, the 
partition column names should be normalized already.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/15819
  
OK so the problem becomes, do we want to backport this to 1.6? cc @rxin 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with format ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16138
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69651/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with format ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16138
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with format ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16138
  
**[Test build #69651 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69651/consoleFull)**
 for PR 16138 at commit 
[`5bf5002`](https://github.com/apache/spark/commit/5bf5002553e3c27632e9050efa36eb382bf8d76d).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with format ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16138
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with format ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16138
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69650/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with format ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16138
  
**[Test build #69650 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69650/consoleFull)**
 for PR 16138 at commit 
[`c17c0bd`](https://github.com/apache/spark/commit/c17c0bdcf91c6154678ca47152f890b71a984eb3).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with format ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16138
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with format ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16138
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69649/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with format ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16138
  
**[Test build #69649 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69649/consoleFull)**
 for PR 16138 at commit 
[`89dccbd`](https://github.com/apache/spark/commit/89dccbd5f4805cb07de29b504669fc15339f4f5e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class ToDate(child: Expression) extends UnaryExpression with 
ImplicitCastInputTypes `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15998: [SPARK-18572][SQL] Add a method `listPartitionNames` to ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15998
  
**[Test build #69652 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69652/consoleFull)**
 for PR 15998 at commit 
[`095cdd5`](https://github.com/apache/spark/commit/095cdd51c386e1747cc8645851b0cdffa83a5a70).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15998: [SPARK-18572][SQL] Add a method `listPartitionNames` to ...

[mallman]

Github user mallman commented on the issue:

https://github.com/apache/spark/pull/15998
  
I've modified the behavior of the partition spec checking methods in 
`SessionCatalog` to test for case-sensitive analysis.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16077: [SPARK-18643][SPARKR] SparkR hangs at session start when...

[shivaram]

Github user shivaram commented on the issue:

https://github.com/apache/spark/pull/16077
  
Just to clarify this limits the auto-install feature to SparkR running on 
an interactive shell -- Is that right ? I think thats mostly fine as its the 
use-case we were targeting but it might good to check our documentation and 
update it appropriately.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16094: [SPARK-18541][Python]Add metadata parameter to pyspark.s...

[holdenk]

Github user holdenk commented on the issue:

https://github.com/apache/spark/pull/16094
  
Lets see if maybe @marmbrus or @davies has some time to look at this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with ...

[anabranch]

Github user anabranch commented on a diff in the pull request:

https://github.com/apache/spark/pull/16138#discussion_r90788929
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala ---
@@ -351,34 +351,81 @@ class DateFunctionsSuite extends QueryTest with 
SharedSQLContext {
   Seq(Row(Date.valueOf("2015-07-30")), 
Row(Date.valueOf("2015-07-30"
   }
 
+
+//  test("function to_timestamp") {
+//
+//  }
+
   test("function to_date") {
 val d1 = Date.valueOf("2015-07-22")
 val d2 = Date.valueOf("2015-07-01")
+val d3 = Date.valueOf("2014-12-31")
 val t1 = Timestamp.valueOf("2015-07-22 10:00:00")
 val t2 = Timestamp.valueOf("2014-12-31 23:59:59")
+val t3 = Timestamp.valueOf("2014-12-31 23:59:59")
 val s1 = "2015-07-22 10:00:00"
 val s2 = "2014-12-31"
-val df = Seq((d1, t1, s1), (d2, t2, s2)).toDF("d", "t", "s")
+val s3 = "2014-31-12"
+val df = Seq((d1, t1, s1), (d2, t2, s2), (d3, t3, s3)).toDF("d", "t", 
"s")
 
 checkAnswer(
   df.select(to_date(col("t"))),
-  Seq(Row(Date.valueOf("2015-07-22")), 
Row(Date.valueOf("2014-12-31"
+  Seq(Row(Date.valueOf("2015-07-22")), Row(Date.valueOf("2014-12-31")),
+Row(Date.valueOf("2014-12-31"
 checkAnswer(
   df.select(to_date(col("d"))),
-  Seq(Row(Date.valueOf("2015-07-22")), 
Row(Date.valueOf("2015-07-01"
+  Seq(Row(Date.valueOf("2015-07-22")), Row(Date.valueOf("2015-07-01")),
+Row(Date.valueOf("2014-12-31"
 checkAnswer(
   df.select(to_date(col("s"))),
-  Seq(Row(Date.valueOf("2015-07-22")), 
Row(Date.valueOf("2014-12-31"
+  Seq(Row(Date.valueOf("2015-07-22")), 
Row(Date.valueOf("2014-12-31")), Row(null)))
 
 checkAnswer(
   df.selectExpr("to_date(t)"),
-  Seq(Row(Date.valueOf("2015-07-22")), 
Row(Date.valueOf("2014-12-31"
+  Seq(Row(Date.valueOf("2015-07-22")), Row(Date.valueOf("2014-12-31")),
+Row(Date.valueOf("2014-12-31"
 checkAnswer(
   df.selectExpr("to_date(d)"),
-  Seq(Row(Date.valueOf("2015-07-22")), 
Row(Date.valueOf("2015-07-01"
+  Seq(Row(Date.valueOf("2015-07-22")), Row(Date.valueOf("2015-07-01")),
+Row(Date.valueOf("2014-12-31"
 checkAnswer(
   df.selectExpr("to_date(s)"),
-  Seq(Row(Date.valueOf("2015-07-22")), 
Row(Date.valueOf("2014-12-31"
+  Seq(Row(Date.valueOf("2015-07-22")), 
Row(Date.valueOf("2014-12-31")), Row(null)))
+
+  // Now with format
+checkAnswer(
+  df.select(to_date(col("t"), "-MM-dd")),
+  Seq(Row(Date.valueOf("2015-07-22")), Row(Date.valueOf("2014-12-31")),
+Row(Date.valueOf("2014-12-31"
+checkAnswer(
+  df.select(to_date(col("d"), "-MM-dd")),
+  Seq(Row(Date.valueOf("2015-07-22")), Row(Date.valueOf("2015-07-01")),
+Row(Date.valueOf("2014-12-31"
+checkAnswer(
+  df.select(to_date(col("s"), "-MM-dd")),
+  Seq(Row(Date.valueOf("2015-07-22")), 
Row(Date.valueOf("2014-12-31")), Row(null)))
--- End diff --

```
scala> val format = new java.text.SimpleDateFormat("-MM-dd")
format: java.text.SimpleDateFormat = java.text.SimpleDateFormat@f67a0200

scala> format.parse("2014-31-12")
res1: java.util.Date = Tue Jul 12 00:00:00 PDT 2016
```

This is strange to me but is expected behavior.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

[merlintang]

Github user merlintang commented on the issue:

https://github.com/apache/spark/pull/15819
  
it is updated.

On Sun, Dec 4, 2016 at 11:23 AM, Xiao Li  wrote:

> @merlintang  Could you please add
> [Branch-1.6] in your PR title?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with format ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16138
  
**[Test build #69651 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69651/consoleFull)**
 for PR 16138 at commit 
[`5bf5002`](https://github.com/apache/spark/commit/5bf5002553e3c27632e9050efa36eb382bf8d76d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with format ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16138
  
**[Test build #69650 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69650/consoleFull)**
 for PR 16138 at commit 
[`c17c0bd`](https://github.com/apache/spark/commit/c17c0bdcf91c6154678ca47152f890b71a984eb3).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with format ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16138
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69648/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with format ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16138
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with format ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16138
  
**[Test build #69648 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69648/consoleFull)**
 for PR 16138 at commit 
[`ecde7b7`](https://github.com/apache/spark/commit/ecde7b7dd5dbcc9985cd0424ddba55de8e95009e).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15998: [SPARK-18572][SQL] Add a method `listPartitionNam...

[mallman]

Github user mallman commented on a diff in the pull request:

https://github.com/apache/spark/pull/15998#discussion_r90788135
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala 
---
@@ -519,6 +519,26 @@ private[hive] class HiveClientImpl(
 client.alterPartitions(table, newParts.map { p => toHivePartition(p, 
hiveTable) }.asJava)
   }
 
+  /**
+   * Returns the partition names for the given table that match the 
supplied partition spec.
+   * If no partition spec is specified, all partitions are returned.
+   *
+   * The returned sequence is sorted as strings.
--- End diff --

The main use of this API is to answer `SHOW PARTITIONS` queries. Sorted 
results are simply easier to read than unsorted results.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16138: [WIP][SPARK-16609] Add to_date/to_timestamp with format ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16138
  
**[Test build #69649 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69649/consoleFull)**
 for PR 16138 at commit 
[`89dccbd`](https://github.com/apache/spark/commit/89dccbd5f4805cb07de29b504669fc15339f4f5e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org