[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14116
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14116
  
**[Test build #62043 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62043/consoleFull)**
 for PR 14116 at commit 
[`c770315`](https://github.com/apache/spark/commit/c7703155904a727c0f77fbf9d600646429f4f3a4).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14116
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62043/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14118: [SPARK-16462][SPARK-16460][SQL] Make CSV cast null value...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/14118
  
I just wonder why string should be ignored in the case above. I mean, you 
just said "we don't need to handle type-specific null values" and it seems 
strings are okay to handle together.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13993: [SPARK-16144][SPARKR] update R API doc for mllib

[yanboliang]

Github user yanboliang commented on a diff in the pull request:

https://github.com/apache/spark/pull/13993#discussion_r70176040
  
--- Diff: R/pkg/R/mllib.R ---
@@ -53,26 +53,27 @@ setClass("AFTSurvivalRegressionModel", 
representation(jobj = "jobj"))
 #' @note KMeansModel since 2.0.0
 setClass("KMeansModel", representation(jobj = "jobj"))
 
-#' Saves the machine learning model to the input path
+#' Saves the MLlib model to the input path
--- End diff --

The ```write.ml``` can only be used for saving MLlib models, it can not 
save other machine learning model produced by native R functions. So I think 
the current description is accurate enough.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14121: [MINOR][ML] update comment where is inconsistent with co...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14121
  
**[Test build #62042 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62042/consoleFull)**
 for PR 14121 at commit 
[`ab7452c`](https://github.com/apache/spark/commit/ab7452c2a48884571c9db6444c739b8cb0c7c603).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14121: [MINOR][ML] update comment where is inconsistent with co...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14121
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14121: [MINOR][ML] update comment where is inconsistent with co...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14121
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62042/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14011: [SparkR] [Doc] SparkR ML user guides update for 2.0

[yanboliang]

Github user yanboliang commented on the issue:

https://github.com/apache/spark/pull/14011
  
@shivaram I think it's good to go, I only did user guides sync and reorg. 
Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14118: [SPARK-16462][SPARK-16460][SQL] Make CSV cast null value...

[lw-lin]

Github user lw-lin commented on the issue:

https://github.com/apache/spark/pull/14118
  
FYI, before 
[SPARK-14143](https://issues.apache.org/jira/browse/SPARK-14143), null values 
had been handled this way: :
```scala
if (datum == options.nullValue && nullable && 
(!castType.isInstanceOf[StringType]))
```
Then in [SPARK-14143](https://issues.apache.org/jira/browse/SPARK-14143), 
it was first broken down into numeric data types in 
https://github.com/apache/spark/pull/11947/commits/93ac6bb3eb63efb775b48af090a37a6cbe4f30c4
 to handle byte-specific null value, short-specific null value, int-specific 
null value, ... :
```scala
case _: ByteType => if (datum == params.byteNullValue && nullable) null 
else datum.toByte
case _: ShortType => if (datum == params.shortNullValue && nullable) null 
else datum.toShort
case _: IntegerType => if (datum == params.integerNullValue && nullable) 
null else datum.toInt
...
```
then in 
https://github.com/apache/spark/pull/11947/commits/698b4b41baa1ebd5d66ea6242bcb39bcd0887f8b
 byte-specific null value, short-specific null value, int-specific null value, 
... were reduced back to one single null value:
```scala
case _: ByteType => if (datum == params.nullValue && nullable) null else 
datum.toByte
case _: ShortType => if (datum == params.nullValue && nullable) null else 
datum.toShort
case _: IntegerType => if (datum == params.nullValue && nullable) null else 
datum.toInt
```

Along with that change, we had introduced regression handling non-numeric 
data types like `BooleanType` etc. Since we don't need to handle type-specific 
null values, this patch switchs back to the way we handled null values in the 
1.6 days (and thus fixes the regression):
 ```scala
if (datum == options.nullValue && nullable && 
(!castType.isInstanceOf[StringType]))
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14121: [MINOR][ML] update comment where is inconsistent with co...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14121
  
**[Test build #62042 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62042/consoleFull)**
 for PR 14121 at commit 
[`ab7452c`](https://github.com/apache/spark/commit/ab7452c2a48884571c9db6444c739b8cb0c7c603).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14116
  
**[Test build #62043 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62043/consoleFull)**
 for PR 14116 at commit 
[`c770315`](https://github.com/apache/spark/commit/c7703155904a727c0f77fbf9d600646429f4f3a4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14121: [MINOR][ML] update comment where is inconsistent ...

[WeichenXu123]

GitHub user WeichenXu123 opened a pull request:

https://github.com/apache/spark/pull/14121

[MINOR][ML] update comment where is inconsistent with code in 
ml.regression.LinearRegression

## What changes were proposed in this pull request?

In `train` method of `ml.regression.LinearRegression` when handling 
situation `std(label) == 0`
the code replace `std(label)` with `mean(label)` but the relative comment 
is inconsistent, I update it. 

## How was this patch tested?

N/A

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/WeichenXu123/spark update_lr_comment

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14121.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14121


commit ab7452c2a48884571c9db6444c739b8cb0c7c603
Author: WeichenXu 
Date:   2016-07-10T04:12:42Z

update_lr_comment




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14118: [SPARK-16462][SPARK-16460][SQL] Make CSV cast null value...

[lw-lin]

Github user lw-lin commented on the issue:

https://github.com/apache/spark/pull/14118
  
The diff that github shows is a mess. The actual diff (which is quite 
small) is:


![diff](https://cloud.githubusercontent.com/assets/15843379/16711624/db6faf94-4697-11e6-8c56-53f10711aea5.png)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14116
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62041/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14116
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14116
  
**[Test build #62041 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62041/consoleFull)**
 for PR 14116 at commit 
[`a55da04`](https://github.com/apache/spark/commit/a55da040607c40be48e1908fa25891bc04b4a26f).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14116
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14116
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62040/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14116
  
**[Test build #62040 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62040/consoleFull)**
 for PR 14116 at commit 
[`8cb4956`](https://github.com/apache/spark/commit/8cb4956363f1470fbab91abe83bce60237c88a82).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14114: [SPARK-16458][SQL] SessionCatalog should support `listCo...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/14114
  
Hi, @rxin .
Now, it's ready for review again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14114: [SPARK-16458][SQL] SessionCatalog should support `listCo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14114
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62039/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14114: [SPARK-16458][SQL] SessionCatalog should support `listCo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14114
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14114: [SPARK-16458][SQL] SessionCatalog should support `listCo...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14114
  
**[Test build #62039 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62039/consoleFull)**
 for PR 14114 at commit 
[`bb1204d`](https://github.com/apache/spark/commit/bb1204d412c54c5f0e3b2cb024df84f086b057e4).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14116
  
**[Test build #62041 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62041/consoleFull)**
 for PR 14116 at commit 
[`a55da04`](https://github.com/apache/spark/commit/a55da040607c40be48e1908fa25891bc04b4a26f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14116
  
**[Test build #62040 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62040/consoleFull)**
 for PR 14116 at commit 
[`8cb4956`](https://github.com/apache/spark/commit/8cb4956363f1470fbab91abe83bce60237c88a82).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14118: [SPARK-16462][SPARK-16460][SQL] Make CSV cast null value...

[shivaram]

Github user shivaram commented on the issue:

https://github.com/apache/spark/pull/14118
  
No - I just noticed a JIRA that said it was a regression, so I wanted to 
make sure you caught this in the RC cycle


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14116
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62038/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14116
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14116
  
**[Test build #62038 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62038/consoleFull)**
 for PR 14116 at commit 
[`b89039d`](https://github.com/apache/spark/commit/b89039deb19c3b9fe4e0775eeed4b2ab0262edfa).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class DatabasesRelationProvider extends SchemaRelationProvider `
  * `class TablesRelationProvider extends SchemaRelationProvider `
  * `class ViewsRelationProvider extends SchemaRelationProvider `
  * `class ColumnsRelationProvider extends SchemaRelationProvider `
  * `class SessionVariablesRelationProvider extends SchemaRelationProvider `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14114: [SPARK-16458][SQL] SessionCatalog should support `listCo...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14114
  
**[Test build #62039 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62039/consoleFull)**
 for PR 14114 at commit 
[`bb1204d`](https://github.com/apache/spark/commit/bb1204d412c54c5f0e3b2cb024df84f086b057e4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14115: [SPARK-16459][SQL] Prevent dropping current database

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14115
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14115: [SPARK-16459][SQL] Prevent dropping current database

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14115
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62037/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14115: [SPARK-16459][SQL] Prevent dropping current database

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14115
  
**[Test build #62037 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62037/consoleFull)**
 for PR 14115 at commit 
[`805b2f7`](https://github.com/apache/spark/commit/805b2f7f99ddaf15b18887c1ef2d5940442bc167).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14116
  
**[Test build #62038 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62038/consoleFull)**
 for PR 14116 at commit 
[`b89039d`](https://github.com/apache/spark/commit/b89039deb19c3b9fe4e0775eeed4b2ab0262edfa).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14114: [SPARK-16458][SQL] SessionCatalog should support `listCo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14114
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14114: [SPARK-16458][SQL] SessionCatalog should support `listCo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14114
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62036/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14114: [SPARK-16458][SQL] SessionCatalog should support `listCo...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14114
  
**[Test build #62036 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62036/consoleFull)**
 for PR 14114 at commit 
[`be0e69a`](https://github.com/apache/spark/commit/be0e69a8a6e0e482f2b0fb2fae219b1ff934dad2).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14114: [SPARK-16458][SQL] SessionCatalog should support `listCo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14114
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62035/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14114: [SPARK-16458][SQL] SessionCatalog should support `listCo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14114
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14114: [SPARK-16458][SQL] SessionCatalog should support `listCo...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14114
  
**[Test build #62035 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62035/consoleFull)**
 for PR 14114 at commit 
[`9134a47`](https://github.com/apache/spark/commit/9134a47820e26b4ab60b23d7ec2e228514396826).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA ...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/14116#discussion_r70173703
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/systemcatalog/InformationSchema.scala
 ---
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.systemcatalog
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.catalog.CatalogDatabase
+import org.apache.spark.sql.catalyst.dsl.plans._
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.sources._
+import org.apache.spark.sql.types._
+
+object InformationSchema extends Logging {
+  def register(sparkSession: SparkSession, relation: BaseRelation, names: 
Seq[String]) {
+val plan = LogicalRelation(relation).analyze
+val projectList = plan.output.zip(plan.schema).map {
+  case (attr, col) => Alias(attr, col.name)()
+}
+sparkSession.sessionState.executePlan(Project(projectList, plan))
+for (name <- names)
+  sparkSession.sessionState.catalog.createTempView(name, plan, 
overrideIfExists = true)
+  }
+
+  def build(sparkSession: SparkSession) {
--- End diff --

Here, `registerInformationSchema`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA ...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/14116#discussion_r70173700
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/systemcatalog/InformationSchema.scala
 ---
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.systemcatalog
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.catalog.CatalogDatabase
+import org.apache.spark.sql.catalyst.dsl.plans._
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.sources._
+import org.apache.spark.sql.types._
+
+object InformationSchema extends Logging {
+  def register(sparkSession: SparkSession, relation: BaseRelation, names: 
Seq[String]) {
--- End diff --

I used `registerTable`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14115: [SPARK-16459][SQL] Prevent dropping current database

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14115
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14115: [SPARK-16459][SQL] Prevent dropping current database

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14115
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62034/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14115: [SPARK-16459][SQL] Prevent dropping current database

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14115
  
**[Test build #62034 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62034/consoleFull)**
 for PR 14115 at commit 
[`8aa1c2d`](https://github.com/apache/spark/commit/8aa1c2d8589b62657d004cf97cceaec2ae60a8cd).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #11317: [SPARK-12639] [SQL] Mark Filters Fully Handled By Source...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/11317
  
**[Test build #3176 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3176/consoleFull)**
 for PR 11317 at commit 
[`4c23cf1`](https://github.com/apache/spark/commit/4c23cf111abf00c5a9049afddbb745fd130e0b30).
 * This patch passes all tests.
 * This patch **does not merge cleanly**.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13969: [SPARK-16284][SQL] Implement reflect SQL function

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13969
  
**[Test build #3174 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3174/consoleFull)**
 for PR 13969 at commit 
[`d24ca97`](https://github.com/apache/spark/commit/d24ca97a0ac85b54ea64b017343fe9eb51ecc02e).
 * This patch passes all tests.
 * This patch **does not merge cleanly**.
 * This patch adds the following public classes _(experimental)_:
  * `case class ParseUrl(children: Seq[Expression])`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13991: [SPARK-16318][SQL] Implement all remaining xpath functio...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13991
  
**[Test build #3175 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3175/consoleFull)**
 for PR 13991 at commit 
[`0c60d87`](https://github.com/apache/spark/commit/0c60d87c0dd1b7e78fd77c2f01b67a2ae8a0151e).
 * This patch passes all tests.
 * This patch **does not merge cleanly**.
 * This patch adds the following public classes _(experimental)_:
  * `case class ParseUrl(children: Seq[Expression])`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14071: [SPARK-16397][SQL] make CatalogTable more general...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/14071#discussion_r70173359
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
 ---
@@ -45,35 +45,28 @@ case class CatalogFunction(
  */
 case class CatalogStorageFormat(
 locationUri: Option[String],
-inputFormat: Option[String],
-outputFormat: Option[String],
-serde: Option[String],
-compressed: Boolean,
-serdeProperties: Map[String, String]) {
+provider: Option[String],
--- End diff --

To remove `CreateHiveTableAsSelectLogicalPlan`, we can add a `CatalogTable` 
member into the signature of `CreateTableUsingAsSelect`. After code changes, 
the signature of `CreateTableUsingAsSelect` becomes
```
tableDesc: CatalogTable,
provider: String,
child: LogicalPlan,
mode: SaveMode
```

After this PR is merged, we also can remove `provider` from the signature!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA ...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/14116#discussion_r7017
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/systemcatalog/InformationSchema.scala
 ---
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.systemcatalog
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.catalog.CatalogDatabase
+import org.apache.spark.sql.catalyst.dsl.plans._
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.sources._
+import org.apache.spark.sql.types._
+
+object InformationSchema extends Logging {
+  def register(sparkSession: SparkSession, relation: BaseRelation, names: 
Seq[String]) {
+val plan = LogicalRelation(relation).analyze
+val projectList = plan.output.zip(plan.schema).map {
+  case (attr, col) => Alias(attr, col.name)()
+}
+sparkSession.sessionState.executePlan(Project(projectList, plan))
+for (name <- names)
+  sparkSession.sessionState.catalog.createTempView(name, plan, 
overrideIfExists = true)
+  }
+
+  def build(sparkSession: SparkSession) {
+// Create database
+val informationSchemaDb = new CatalogDatabase(
+  "information_schema",
+  "Spark information schema",
+  locationUri = "/tmp/",
+  Map())
+sparkSession.sessionState.catalog.createDatabase(informationSchemaDb, 
ignoreIfExists = true)
+
+// Create tables
+register(sparkSession, new DatabasesRelation(sparkSession), 
Seq("schemata", "databases"))
+register(sparkSession, new TablesRelation(sparkSession), Seq("tables"))
+register(sparkSession, new ViewsRelation(sparkSession), Seq("views"))
+register(sparkSession, new ColumnsRelation(sparkSession), 
Seq("columns"))
+register(sparkSession, new SessionVariablesRelation(sparkSession), 
Seq("session_variables"))
+  }
+}
+
+class DatabasesSource extends SchemaRelationProvider {
+  override def createRelation(
+  sqlContext: SQLContext,
+  parameters: Map[String, String],
+  schema: StructType): BaseRelation = {
+DatabasesRelation(sqlContext.sparkSession)
+  }
+}
+
+case class DatabasesRelation(@transient sparkSession: SparkSession)
+  extends BaseRelation with PrunedFilteredScan {
+
+  override def sqlContext: SQLContext = sparkSession.sqlContext
+
+  override def schema: StructType = StructType(Seq(
+StructField("CATALOG_NAME", StringType, false),
+StructField("SCHEMA_NAME", StringType, false)
+  ))
+
+  override def buildScan(requiredColumns: Array[String], filters: 
Array[Filter]): RDD[Row] = {
+sparkSession.catalog.listDatabases()
+  .selectExpr(
+"'default' as CATALOG_NAME",
+"name as SCHEMA_NAME")
+  .select(requiredColumns.head, requiredColumns.tail: _*).rdd
+  }
+}
+
+class TablesSource extends SchemaRelationProvider {
+  override def createRelation(
+  sqlContext: SQLContext,
+  parameters: Map[String, String],
+  schema: StructType): BaseRelation = {
+TablesRelation(sqlContext.sparkSession)
+  }
+}
+
+case class TablesRelation(@transient sparkSession: SparkSession)
+  extends BaseRelation with PrunedFilteredScan {
+
+  override def sqlContext: SQLContext = sparkSession.sqlContext
+
+  override def schema: StructType = StructType(Seq(
+StructField("TABLE_CATALOG", StringType, false),
+StructField("TABLE_SCHEMA", StringType, false),
+StructField("TABLE_NAME", StringType, false),
+StructField("TABLE_TYPE", StringType, 

[GitHub] spark issue #14115: [SPARK-16459][SQL] Prevent dropping current database

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14115
  
**[Test build #62037 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62037/consoleFull)**
 for PR 14115 at commit 
[`805b2f7`](https://github.com/apache/spark/commit/805b2f7f99ddaf15b18887c1ef2d5940442bc167).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14114: [SPARK-16458][SQL] SessionCatalog should support `listCo...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/14114
  
Now, `getTableMetadata` is improved.
Thank you for that advice, @rxin .


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14115: [SPARK-16459][SQL] Prevent dropping current datab...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/14115#discussion_r70173212
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -146,8 +148,10 @@ class SessionCatalog(
 
   def dropDatabase(db: String, ignoreIfNotExists: Boolean, cascade: 
Boolean): Unit = {
 val dbName = formatDatabaseName(db)
-if (dbName == "default") {
+if (dbName == DEFAULT_DATABASE) {
   throw new AnalysisException(s"Can not drop default database")
+} else if (dbName == getCurrentDatabase) {
--- End diff --

Oh, I was confused. For case sensitive case, we don't need to handle that.
Dropping with different name will fails due to lookup failure.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14114: [SPARK-16458][SQL] SessionCatalog should support `listCo...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14114
  
**[Test build #62036 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62036/consoleFull)**
 for PR 14114 at commit 
[`be0e69a`](https://github.com/apache/spark/commit/be0e69a8a6e0e482f2b0fb2fae219b1ff934dad2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14119: [SPARK-16303][DOCS][EXAMPLES][WIP] Updated SQL pr...

[aokolnychyi]

Github user aokolnychyi commented on a diff in the pull request:

https://github.com/apache/spark/pull/14119#discussion_r70173180
  
--- Diff: 
examples/src/main/java/org/apache/spark/examples/sql/JavaSparkSqlExample.java 
---
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.examples.sql;
+
+// $example on:programmatic_schema$
+import java.util.ArrayList;
+import java.util.List;
+// $example off:programmatic_schema$
+// $example on:create_ds$
+import java.util.Arrays;
--- End diff --

Here the imports do not follow the alphabetical order to avoid too many 
imports groups in the documentation (there would be a blank line between each 
"example on/off" block).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14114: [SPARK-16458][SQL] SessionCatalog should support `listCo...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14114
  
**[Test build #62035 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62035/consoleFull)**
 for PR 14114 at commit 
[`9134a47`](https://github.com/apache/spark/commit/9134a47820e26b4ab60b23d7ec2e228514396826).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14115: [SPARK-16459][SQL] Prevent dropping current database

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14115
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14115: [SPARK-16459][SQL] Prevent dropping current database

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14115
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62032/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14115: [SPARK-16459][SQL] Prevent dropping current database

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14115
  
**[Test build #62032 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62032/consoleFull)**
 for PR 14115 at commit 
[`ac5f3ea`](https://github.com/apache/spark/commit/ac5f3ea8397f6ebeed42905c0fed133e45c6b5dd).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14115: [SPARK-16459][SQL] Prevent dropping current datab...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/14115#discussion_r70173134
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -146,8 +148,10 @@ class SessionCatalog(
 
   def dropDatabase(db: String, ignoreIfNotExists: Boolean, cascade: 
Boolean): Unit = {
 val dbName = formatDatabaseName(db)
-if (dbName == "default") {
+if (dbName == DEFAULT_DATABASE) {
   throw new AnalysisException(s"Can not drop default database")
+} else if (dbName == getCurrentDatabase) {
--- End diff --

I'll add two testcases for both and fix this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14119: [SPARK-16303][DOCS][EXAMPLES][WIP] Updated SQL pr...

[aokolnychyi]

Github user aokolnychyi commented on a diff in the pull request:

https://github.com/apache/spark/pull/14119#discussion_r70173131
  
--- Diff: 
examples/src/main/scala/org/apache/spark/examples/sql/SqlDataSourceExample.scala
 ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.examples.sql
+
+import org.apache.spark.sql.SparkSession
+
+object SqlDataSourceExample {
+
+  case class Person(name: String, age: Long)
+
+  def main(args: Array[String]) {
+val spark = SparkSession
+.builder()
+.appName("Spark SQL Data Soures Example")
+.config("spark.some.config.option", "some-value")
+.getOrCreate()
+
+runBasicDataSourceExample(spark)
+runBasicParquetExample(spark)
+runParquetSchemaMergingExample(spark)
+runJsonDatasetExample(spark)
+
+spark.stop()
+  }
+
+  private def runBasicDataSourceExample(spark: SparkSession): Unit = {
+// $example on:generic_load_save_functions$
+val usersDF = 
spark.read.load("examples/src/main/resources/users.parquet")
+usersDF.select("name", 
"favorite_color").write.save("namesAndFavColors.parquet")
+// $example off:generic_load_save_functions$
+// $example on:manual_load_options$
+val peopleDF = 
spark.read.format("json").load("examples/src/main/resources/people.json")
+peopleDF.select("name", 
"age").write.format("parquet").save("namesAndAges.parquet")
+// $example off:manual_load_options$
+// $example on:direct_sql$
+val sqlDF = spark.sql("SELECT * FROM 
parquet.`examples/src/main/resources/users.parquet`")
--- End diff --

Here the line length slightly exceeds the limit to make the look of the 
documentation better. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14115: [SPARK-16459][SQL] Prevent dropping current datab...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/14115#discussion_r70173113
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -146,8 +148,10 @@ class SessionCatalog(
 
   def dropDatabase(db: String, ignoreIfNotExists: Boolean, cascade: 
Boolean): Unit = {
 val dbName = formatDatabaseName(db)
-if (dbName == "default") {
+if (dbName == DEFAULT_DATABASE) {
   throw new AnalysisException(s"Can not drop default database")
+} else if (dbName == getCurrentDatabase) {
--- End diff --

Oops. `formatDatabaseName` returns the raw string for case sensitive case.
Hm. That is the same situation with "default" database, isn't?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14119: [SPARK-16303][DOCS][EXAMPLES][WIP] Updated SQL pr...

[aokolnychyi]

Github user aokolnychyi commented on a diff in the pull request:

https://github.com/apache/spark/pull/14119#discussion_r70173058
  
--- Diff: 
examples/src/main/scala/org/apache/spark/examples/sql/hive/SparkHiveExample.scala
 ---
@@ -41,43 +35,47 @@ object HiveFromSpark {
 // in the current directory and creates a directory configured by 
`spark.sql.warehouse.dir`,
 // which defaults to the directory `spark-warehouse` in the current 
directory that the spark
 // application is started.
-val spark = SparkSession.builder
-  .appName("HiveFromSpark")
-  .enableHiveSupport()
-  .getOrCreate()
+
+// $example on:spark_hive$
+// warehouseLocation points to the default location for managed 
databases and tables
+val warehouseLocation = "file:${system:user.dir}/spark-warehouse"
+
+val spark = SparkSession
+.builder()
+.appName("Spark Hive Example")
+.config("spark.sql.warehouse.dir", warehouseLocation)
+.enableHiveSupport()
+.getOrCreate()
 
 import spark.implicits._
 import spark.sql
 
 sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
-sql(s"LOAD DATA LOCAL INPATH '${kv1File.getAbsolutePath}' INTO TABLE 
src")
+sql("LOAD DATA LOCAL INPATH 'examples/src/main/resources/kv1.txt' INTO 
TABLE src")
 
 // Queries are expressed in HiveQL
-println("Result of 'SELECT *': ")
-sql("SELECT * FROM src").collect().foreach(println)
+sql("SELECT * FROM src").show()
--- End diff --

I replaced collect().foreach(println) with show() in all examples. Is it OK?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14119: [SPARK-16303][DOCS][EXAMPLES][WIP] Updated SQL pr...

[aokolnychyi]

Github user aokolnychyi commented on a diff in the pull request:

https://github.com/apache/spark/pull/14119#discussion_r70173035
  
--- Diff: docs/sql-programming-guide.md ---
@@ -1380,17 +949,17 @@ metadata.
 
 
 {% highlight scala %}
-// spark is an existing HiveContext
-spark.refreshTable("my_table")
+// spark is an existing SparkSession
+spark.catalog.refreshTable("my_table")
--- End diff --

Is it the correct way to refresh?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14112: [SPARK-16240][ML] Model loading backward compatibility f...

[hhbyyh]

Github user hhbyyh commented on the issue:

https://github.com/apache/spark/pull/14112
  
@jkbradley I find it not easy to add a unit test to cover the logic. 
Appreciate your thoughts.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14115: [SPARK-16459][SQL] Prevent dropping current datab...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/14115#discussion_r70172847
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -146,8 +148,10 @@ class SessionCatalog(
 
   def dropDatabase(db: String, ignoreIfNotExists: Boolean, cascade: 
Boolean): Unit = {
 val dbName = formatDatabaseName(db)
-if (dbName == "default") {
+if (dbName == DEFAULT_DATABASE) {
   throw new AnalysisException(s"Can not drop default database")
+} else if (dbName == getCurrentDatabase) {
--- End diff --

do we need to check case sensitivity?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14112: [SPARK-16240][ML] Model loading backward compatibility f...

[hhbyyh]

Github user hhbyyh commented on the issue:

https://github.com/apache/spark/pull/14112
  
PR created. Sorry I got something else that I need to turn to. Ideally, the 
overriding `getAndSetParams` should be in LDAParams, thus it can be reused by 
LDA and LDA Local/Distributed Model. Please help move it there (perhaps a new 
Function in LDAParams)

Let me know if you have any question. I'll revisit ASAP.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14114: [SPARK-16458][SQL] SessionCatalog should support ...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/14114#discussion_r70172672
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -442,6 +442,10 @@ class SessionCatalog(
 name.database.isEmpty && 
tempTables.contains(formatTableName(name.table))
   }
 
+  def listTemporaryTableOutput(name: String): Seq[Attribute] = {
--- End diff --

Ah. I remember that why I do this in this way. Basically, there are two 
barriers to reach `getTableMetadata`. Before making change, let me describe 
here.

1. Redirecting: `listColumns(table)`  -> `listColumns(currentDatabase, 
tableName)`
2. Table existence failure: `requireTableExists(dbName, tableName)` in 
`listColumns(currentDatabase, tableName)`.

Anyway, I'm trying to change the above barriers.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14115: [SPARK-16459][SQL] Prevent dropping current database

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14115
  
**[Test build #62034 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62034/consoleFull)**
 for PR 14115 at commit 
[`8aa1c2d`](https://github.com/apache/spark/commit/8aa1c2d8589b62657d004cf97cceaec2ae60a8cd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14115: [SPARK-16459][SQL] Prevent dropping current database

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/14115
  
Now, it preserves the previous behavior for dropping default db, and throws 
new exception message for new cases only.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14116
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62031/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14116
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14116
  
**[Test build #62031 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62031/consoleFull)**
 for PR 14116 at commit 
[`d9d9344`](https://github.com/apache/spark/commit/d9d93449f29711f1786a76947ad6bc72855a0917).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14116
  
**[Test build #62030 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62030/consoleFull)**
 for PR 14116 at commit 
[`d9d9344`](https://github.com/apache/spark/commit/d9d93449f29711f1786a76947ad6bc72855a0917).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14116
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62030/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14116
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14120: [SPARK-16199][SQL] Add a method to list the referenced c...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14120
  
**[Test build #62033 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62033/consoleFull)**
 for PR 14120 at commit 
[`795b927`](https://github.com/apache/spark/commit/795b92722e992b0eff6069f1763766cc0fdd70f1).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class FiltersSuite extends SparkFunSuite `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14120: [SPARK-16199][SQL] Add a method to list the referenced c...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14120
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62033/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14120: [SPARK-16199][SQL] Add a method to list the referenced c...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14120
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14115: [SPARK-16459][SQL] Prevent dropping current datab...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/14115#discussion_r70172456
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -146,8 +148,8 @@ class SessionCatalog(
 
   def dropDatabase(db: String, ignoreIfNotExists: Boolean, cascade: 
Boolean): Unit = {
 val dbName = formatDatabaseName(db)
-if (dbName == "default") {
-  throw new AnalysisException(s"Can not drop default database")
+if (dbName == DEFAULT_DATABASE || dbName == getCurrentDatabase) {
+  throw new AnalysisException(s"Can not drop `${DEFAULT_DATABASE}` or 
current database")
--- End diff --

I see. No problem.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14115: [SPARK-16459][SQL] Prevent dropping current datab...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/14115#discussion_r70172359
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala 
---
@@ -1270,21 +1270,28 @@ class DDLSuite extends QueryTest with 
SharedSQLContext with BeforeAndAfterEach {
   "WITH SERDEPROPERTIES ('spark.sql.sources.me'='anything')")
   }
 
-  test("drop default database") {
+  test("drop default or current database") {
--- End diff --

break it into two test cases


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14115: [SPARK-16459][SQL] Prevent dropping current datab...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/14115#discussion_r70172357
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -146,8 +148,8 @@ class SessionCatalog(
 
   def dropDatabase(db: String, ignoreIfNotExists: Boolean, cascade: 
Boolean): Unit = {
 val dbName = formatDatabaseName(db)
-if (dbName == "default") {
-  throw new AnalysisException(s"Can not drop default database")
+if (dbName == DEFAULT_DATABASE || dbName == getCurrentDatabase) {
+  throw new AnalysisException(s"Can not drop `${DEFAULT_DATABASE}` or 
current database")
--- End diff --

hm i think we should break this into two messages rather than a single one.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14120: [SPARK-16199][SQL] Add a method to list the referenced c...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14120
  
**[Test build #62033 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62033/consoleFull)**
 for PR 14120 at commit 
[`795b927`](https://github.com/apache/spark/commit/795b92722e992b0eff6069f1763766cc0fdd70f1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14120: [SPARK-16199][SQL] Add a method to list the refer...

[petermaxlee]

GitHub user petermaxlee opened a pull request:

https://github.com/apache/spark/pull/14120

[SPARK-16199][SQL] Add a method to list the referenced columns in data 
source Filter

## What changes were proposed in this pull request?
It would be useful to support listing the columns that are referenced by a 
filter. This can help simplify data source planning, because with this we would 
be able to implement unhandledFilters method in HadoopFsRelation.

This is based on @rxin's patch (#13901) and adds unit tests.

## How was this patch tested?
Added a new suite FiltersSuite.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/petermaxlee/spark SPARK-16199

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14120.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14120


commit 846c0e4ddec35352b07c9a4f72769bcceecfa25e
Author: Reynold Xin 
Date:   2016-06-24T22:55:35Z

[SPARK-16199][SQL] Add a method to list the referenced columns in data 
source Filter

commit 5c8978ebf77e60f0ab890f7e7756ed5930fba128
Author: petermaxlee 
Date:   2016-07-09T21:22:50Z

Merge pull request #13901 from rxin/SPARK-16199

[SPARK-16199][SQL] Add a method to list the referenced columns in data 
source Filter

commit 795b92722e992b0eff6069f1763766cc0fdd70f1
Author: petermaxlee 
Date:   2016-07-09T21:34:25Z

Added unit tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13901: [SPARK-16199][SQL] Add a method to list the referenced c...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/13901
  
@rxin I added tests to this pull request and submitted 
https://github.com/apache/spark/pull/14120


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/14116
  
Sure. I'll update the PR and proceed in this way.
Thank you, @rxin .


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA ...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/14116#discussion_r70172331
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/systemcatalog/InformationSchemaSuite.scala
 ---
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.systemcatalog
+
+import org.apache.spark.sql.{QueryTest, Row}
+import org.apache.spark.sql.test.SharedSQLContext
+
+/**
+ * Information Schema Suite
+ */
+class InformationSchemaSuite extends QueryTest with SharedSQLContext {
+  override def beforeAll(): Unit = {
+super.beforeAll()
+InformationSchema.build(spark)
+sql("CREATE TABLE s(a int, b double)")
+sql("CREATE TABLE t(c int, d double)")
+  }
+
+  override def afterAll(): Unit = {
+try {
+  sql("DROP TABLE IF EXISTS t")
+  sql("DROP TABLE IF EXISTS s")
+} finally {
+  super.afterAll()
+}
+  }
+
+  test("databases/schemata scan") {
+checkAnswer(
+  sql("select * from databases"),
+  Row("default", "default") :: Row("default", "information_schema") :: 
Nil)
+
+checkAnswer(
+  sql("select * from schemata"),
+  Row("default", "default") :: Row("default", "information_schema") :: 
Nil)
+  }
+
+  test("databases/schemata project") {
+checkAnswer(
+  sql("select SCHEMA_NAME from databases"),
+  Row("default") :: Row("information_schema") :: Nil)
+
+checkAnswer(
+  sql("select SCHEMA_NAME from schemata"),
+  Row("default") :: Row("information_schema") :: Nil)
+  }
+
+  test("tables scan") {
+checkAnswer(
+  sql("select * from tables"),
+  Row("default", "default", "columns", "VIEW") ::
--- End diff --

Yep.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14115: [SPARK-16459][SQL] Prevent dropping current database

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14115
  
**[Test build #62032 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62032/consoleFull)**
 for PR 14115 at commit 
[`ac5f3ea`](https://github.com/apache/spark/commit/ac5f3ea8397f6ebeed42905c0fed133e45c6b5dd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14112: [SPARK-16240][ML] Model loading backward compatibility f...

[hhbyyh]

Github user hhbyyh commented on the issue:

https://github.com/apache/spark/pull/14112
  
Thanks @GayathriMurali for the PR. I think we'll need to override the 
default behavior of getAndSetParams. Meanwhile, we need to invoke both 
convertVectorColumnsToML and convertMatrixColumnsToML. 

I'll send a PR to your repository for reference.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14119: [SPARK-16303][DOCS][EXAMPLES][WIP] Updated SQL programmi...

[aokolnychyi]

Github user aokolnychyi commented on the issue:

https://github.com/apache/spark/pull/14119
  
@liancheng could you, please, review this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14090: [SPARK-16112][SparkR] Programming guide for gappl...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/14090#discussion_r70172206
  
--- Diff: docs/sparkr.md ---
@@ -306,6 +306,64 @@ head(ldf, 3)
 {% endhighlight %}
 
 
+ Run a given function on a large dataset grouping by input column(s) 
and using `gapply` or `gapplyCollect`
+
+# gapply
+Apply a function to each group of a `SparkDataFrame`. The function is to 
be applied to each group of the `SparkDataFrame` and should have only two 
parameters: grouping key and R `data.frame` corresponding to
+that key. The groups are chosen from `SparkDataFrame`s column(s).
+The output of function should be a `data.frame`. Schema specifies the row 
format of the resulting
+`SparkDataFrame`. It must match the R function's output.
--- End diff --

I think gapply and dapply are the first important use cases where we 
require strict mapping Spark JVM types to R atomic types. It might be 
worthwhile to add a section in the programming guide to illustrate and explain 
that further.

To be more concrete, what should be the column type of the UDF output R 
data.frame if the SparkDataFrame has a column of double? It would be good to 
have a table on that.

That could be a separate PR though.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14119: [SPARK-16303][DOCS][EXAMPLES][WIP] Updated SQL programmi...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14119
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14117: [SPARK-16461][SQL] Support partition batch pruning with ...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/14117
  
Thank you! Could you take a look @davies and @liancheng ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #11317: [SPARK-12639] [SQL] Mark Filters Fully Handled By Source...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/11317
  
**[Test build #3176 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3176/consoleFull)**
 for PR 11317 at commit 
[`4c23cf1`](https://github.com/apache/spark/commit/4c23cf111abf00c5a9049afddbb745fd130e0b30).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13969: [SPARK-16284][SQL] Implement reflect SQL function

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13969
  
**[Test build #3174 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3174/consoleFull)**
 for PR 13969 at commit 
[`d24ca97`](https://github.com/apache/spark/commit/d24ca97a0ac85b54ea64b017343fe9eb51ecc02e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13991: [SPARK-16318][SQL] Implement all remaining xpath functio...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13991
  
**[Test build #3175 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3175/consoleFull)**
 for PR 13991 at commit 
[`0c60d87`](https://github.com/apache/spark/commit/0c60d87c0dd1b7e78fd77c2f01b67a2ae8a0151e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14119: [SPARK-16303][DOCS][EXAMPLES][WIP] Updated SQL pr...

[aokolnychyi]

GitHub user aokolnychyi opened a pull request:

https://github.com/apache/spark/pull/14119

[SPARK-16303][DOCS][EXAMPLES][WIP] Updated SQL programming guide and 
examples

## What changes were proposed in this pull request?

- Hard-coded Spark SQL sample snippets were moved into source files under 
examples sub-project.
- Removed the inconsistency between Scala and Java Spark SQL examples
- Scala and Java Spark SQL examples were updated

## How was this patch tested?

The work is still in progress. All involved examples were tested manually. 
An additional round of testing will be done after the code review.




![image](https://cloud.githubusercontent.com/assets/6235869/16710314/51851606-462a-11e6-9fbe-0818daef65e4.png)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aokolnychyi/spark spark_16303

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14119.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14119


commit 95f0f41fa12e1c6f0fb8ce6cd4222fb63842b495
Author: aokolnychyi 
Date:   2016-07-09T20:56:47Z

[SPARK-16303][DOCS][EXAMPLES] Updated SQL programming guide and examples




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA support

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/14116
  
This looks pretty good. Can you add more comments explaining what each 
class/method does, and how the whole thing works?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14116: [SPARK-16452][SQL][WIP] Basic INFORMATION_SCHEMA ...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/14116#discussion_r70172130
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/systemcatalog/InformationSchema.scala
 ---
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.systemcatalog
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.catalog.CatalogDatabase
+import org.apache.spark.sql.catalyst.dsl.plans._
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.execution.datasources._
+import org.apache.spark.sql.sources._
+import org.apache.spark.sql.types._
+
+object InformationSchema extends Logging {
+  def register(sparkSession: SparkSession, relation: BaseRelation, names: 
Seq[String]) {
--- End diff --

and setupTable or registerTable


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org