[GitHub] spark issue #15637: [SPARK-18000] [SQL] Aggregation function for computing e...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15637 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15637: [SPARK-18000] [SQL] Aggregation function for computing e...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15637 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67555/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15637: [SPARK-18000] [SQL] Aggregation function for computing e...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15637 **[Test build #67555 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67555/consoleFull)** for PR 15637 at commit [`32478d1`](https://github.com/apache/spark/commit/32478d160356aec3cc07579a657b3e8fbd20e2bd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11228: [SPARK-13356][Streaming]WebUI missing input informations...
Github user jeanlyn commented on the issue: https://github.com/apache/spark/pull/11228 @tdas OK, i will try to add unit test these day. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15516: [SPARK-17961][SparkR][SQL] Add storageLevel to DataFrame...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15516 **[Test build #67565 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67565/consoleFull)** for PR 15516 at commit [`1977591`](https://github.com/apache/spark/commit/1977591400208672c4987d7b51a4a3a70710a6d6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15599: [SPARK-18022][SQL] java.lang.NullPointerException...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/15599#discussion_r85057571
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
---
@@ -607,7 +607,7 @@ object JdbcUtils extends Logging {
} catch {
case e: SQLException =>
val cause = e.getNextException
-if (e.getCause != cause) {
+if (cause != null && e.getCause != cause) {
--- End diff --
This looks correct as `addSuppressed(null)` will throw NPE. However, it
might be hard to create a test for it...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15596: [SPARK-18089][SQL] Remove CollectLimitExec
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15596 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15596: [SPARK-18089][SQL] Remove CollectLimitExec
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15596 **[Test build #67556 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67556/consoleFull)** for PR 15596 at commit [`110a3e4`](https://github.com/apache/spark/commit/110a3e44f983ec90e7dbfafbfc9ce1932885c903). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15596: [SPARK-18089][SQL] Remove CollectLimitExec
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15596 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67556/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15538: [SPARK-17993][SQL] Fix Parquet log output redirection
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15538 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15538: [SPARK-17993][SQL] Fix Parquet log output redirection
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15538 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67552/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15538: [SPARK-17993][SQL] Fix Parquet log output redirection
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15538 **[Test build #67552 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67552/consoleFull)** for PR 15538 at commit [`02df8c2`](https://github.com/apache/spark/commit/02df8c273ffac794bfb5bff6f3cc0ab9532264f9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15615: [SPARK-17693] [SQL] Fixed Insert Failure To Data Source ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15615 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15516: [SPARK-17961][SparkR][SQL] Add storageLevel to DataFrame...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15516 **[Test build #67563 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67563/consoleFull)** for PR 15516 at commit [`aa56467`](https://github.com/apache/spark/commit/aa56467a523ccaee17e224415378db794ddd7f8b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15516: [SPARK-17961][SparkR][SQL] Add storageLevel to DataFrame...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15516 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15615: [SPARK-17693] [SQL] Fixed Insert Failure To Data Source ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15615 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67553/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14957 **[Test build #67564 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67564/consoleFull)** for PR 14957 at commit [`d9aa397`](https://github.com/apache/spark/commit/d9aa397683afc4b936529d6983f9b48dd4d2ee15). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15516: [SPARK-17961][SparkR][SQL] Add storageLevel to DataFrame...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15516 **[Test build #67563 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67563/consoleFull)** for PR 15516 at commit [`aa56467`](https://github.com/apache/spark/commit/aa56467a523ccaee17e224415378db794ddd7f8b). * This patch **fails some tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15516: [SPARK-17961][SparkR][SQL] Add storageLevel to DataFrame...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15516 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67563/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15615: [SPARK-17693] [SQL] Fixed Insert Failure To Data Source ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15615 **[Test build #67553 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67553/consoleFull)** for PR 15615 at commit [`bf81bb6`](https://github.com/apache/spark/commit/bf81bb66c136184facc67008df2929144234cb5a). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class SingularMatrixException(message: String, cause: Throwable)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15596: [SPARK-18089][SQL] Remove CollectLimitExec
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15596 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67557/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15596: [SPARK-18089][SQL] Remove CollectLimitExec
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15596 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15596: [SPARK-18089][SQL] Remove CollectLimitExec
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15596 **[Test build #67557 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67557/consoleFull)** for PR 15596 at commit [`e919f4a`](https://github.com/apache/spark/commit/e919f4a9d3c55cfe7b28b9fd89709cc747e736e6). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15516: [SPARK-17961][SparkR][SQL] Add storageLevel to DataFrame...
Github user WeichenXu123 commented on the issue: https://github.com/apache/spark/pull/15516 @felixcheung update rdname, `unpersited-method` also updated by the way. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15513: [SPARK-17963][SQL][Documentation] Add examples (extend) ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15513 This PR adds a new section about function arguments, which do not exist before. That is why I think we should not merge anything that is not accurate. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15639: [Spark-Core]add defensive check for zipWithIndex
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15639 **[Test build #67562 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67562/consoleFull)** for PR 15639 at commit [`6390cd8`](https://github.com/apache/spark/commit/6390cd80bb5776b1170c5eb57ff9860691d89627). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15639: [Spark-Core]add defensive check for zipWithIndex
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15639#discussion_r85053706
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1765,6 +1765,7 @@ private[spark] object Utils extends Logging {
*/
def getIteratorZipWithIndex[T](iterator: Iterator[T], startIndex: Long):
Iterator[(T, Long)] = {
new Iterator[(T, Long)] {
+ require(startIndex > 0, "startIndex should be > 0.")
--- End diff --
OK. I will update the check to >= 0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15639: [Spark-Core]add defensive check for zipWithIndex
Github user WeichenXu123 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15639#discussion_r85053450
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1765,6 +1765,7 @@ private[spark] object Utils extends Logging {
*/
def getIteratorZipWithIndex[T](iterator: Iterator[T], startIndex: Long):
Iterator[(T, Long)] = {
new Iterator[(T, Long)] {
+ require(startIndex > 0, "startIndex should be > 0.")
--- End diff --
yeah, this case inital value = -1, but fisrt generated index is 0, because
there is a `index += 1` clause running first.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15639: [Spark-Core]add defensive check for zipWithIndex
Github user wangmiao1981 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15639#discussion_r85053102
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1765,6 +1765,7 @@ private[spark] object Utils extends Logging {
*/
def getIteratorZipWithIndex[T](iterator: Iterator[T], startIndex: Long):
Iterator[(T, Long)] = {
new Iterator[(T, Long)] {
+ require(startIndex > 0, "startIndex should be > 0.")
--- End diff --
In the following line, you do `var index: Long = startIndex - 1L`. If it is
== 0, then this line is -1L.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15639: [Spark-Core]add defensive check for zipWithIndex
Github user WeichenXu123 commented on a diff in the pull request:
https://github.com/apache/spark/pull/15639#discussion_r85052950
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1765,6 +1765,7 @@ private[spark] object Utils extends Logging {
*/
def getIteratorZipWithIndex[T](iterator: Iterator[T], startIndex: Long):
Iterator[(T, Long)] = {
new Iterator[(T, Long)] {
+ require(startIndex > 0, "startIndex should be > 0.")
--- End diff --
It seems to be `startIndex >= 0` ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Github user wangmiao1981 commented on the issue: https://github.com/apache/spark/pull/15365 Sure. I will do it. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15640: [SPARK-18106][SQL] ANALYZE TABLE should raise a ParseExc...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15640 **[Test build #67560 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67560/consoleFull)** for PR 15640 at commit [`4819dd1`](https://github.com/apache/spark/commit/4819dd147114ce50b388a1385ebb7119097c9beb). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15024: [SPARK-17470][SQL] unify path for data source table and ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15024 **[Test build #67561 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67561/consoleFull)** for PR 15024 at commit [`0fd8d1c`](https://github.com/apache/spark/commit/0fd8d1ccaf6c74799e81a9fc404c5b6c1c329aee). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15640: [SPARK-18106][SQL] ANALYZE TABLE should raise a P...
GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/15640
[SPARK-18106][SQL] ANALYZE TABLE should raise a ParseException for invalid
option
## What changes were proposed in this pull request?
Currently, `ANALYZE TABLE` command accepts `identifier` for option
`NOSCAN`. This PR raises a ParseException for unknown option.
**Before**
```scala
scala> sql("create table test(a int)")
res0: org.apache.spark.sql.DataFrame = []
scala> sql("analyze table test compute statistics blah")
res1: org.apache.spark.sql.DataFrame = []
```
**After**
```scala
scala> sql("create table test(a int)")
res0: org.apache.spark.sql.DataFrame = []
scala> sql("analyze table test compute statistics blah")
org.apache.spark.sql.catalyst.parser.ParseException:
Expected `NOSCAN` instead of `blah`(line 1, pos 0)
```
## How was this patch tested?
Pass the Jenkins test with a new test case.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-18106
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15640.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15640
commit 4819dd147114ce50b388a1385ebb7119097c9beb
Author: Dongjoon Hyun
Date: 2016-10-26T04:54:34Z
[SPARK-18106][SQL] ANALYZE TABLE should raise a ParseException for invalid
option
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15638: [SPARK-18110][PYTHON][ML] add missing parameter in Pytho...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15638 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67558/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15638: [SPARK-18110][PYTHON][ML] add missing parameter in Pytho...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15638 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15638: [SPARK-18110][PYTHON][ML] add missing parameter in Pytho...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15638 **[Test build #67558 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67558/consoleFull)** for PR 15638 at commit [`e734e01`](https://github.com/apache/spark/commit/e734e01034b1fa4d3e3b3e48e7b233cd4008a40e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15638: [SPARK-18110][PYTHON][ML] add missing parameter i...
Github user wangmiao1981 commented on a diff in the pull request: https://github.com/apache/spark/pull/15638#discussion_r85051191 --- Diff: python/pyspark/ml/classification.py --- @@ -758,20 +758,21 @@ def __init__(self, featuresCol="features", labelCol="label", predictionCol="pred probabilityCol="probability", rawPredictionCol="rawPrediction", maxDepth=5, maxBins=32, minInstancesPerNode=1, minInfoGain=0.0, maxMemoryInMB=256, cacheNodeIds=False, checkpointInterval=10, impurity="gini", - numTrees=20, featureSubsetStrategy="auto", seed=None): + numTrees=20, featureSubsetStrategy="auto", seed=None, subsamplingRate=1.0): --- End diff -- Add some doc string tests? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15638: [SPARK-18110][PYTHON][ML] add missing parameter i...
Github user wangmiao1981 commented on a diff in the pull request: https://github.com/apache/spark/pull/15638#discussion_r85051149 --- Diff: python/pyspark/ml/regression.py --- @@ -828,7 +828,7 @@ def featureImportances(self): @inherit_doc class RandomForestRegressor(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol, HasSeed, RandomForestParams, TreeRegressorParams, HasCheckpointInterval, -JavaMLWritable, JavaMLReadable): +JavaMLWritable, JavaMLReadable, HasVarianceCol): --- End diff -- Would you like to group all the `Has*` parameters? Just a minor comment on the style. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15639: [Spark-Core]add defensive check for zipWithIndex
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15639 **[Test build #67559 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67559/consoleFull)** for PR 15639 at commit [`1d3d4fe`](https://github.com/apache/spark/commit/1d3d4fec775b05bb4b4d8de225cf500ac661a2cd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15639: [Core]add defensive check for zipWithIndex
Github user wangmiao1981 commented on the issue: https://github.com/apache/spark/pull/15639 @WeichenXu123 Can you take a look? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15639: [Core]add defensive check for zipWithIndex
GitHub user wangmiao1981 opened a pull request: https://github.com/apache/spark/pull/15639 [Core]add defensive check for zipWithIndex ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) `Utils.getIteratorZipWithIndex` was added to deal with number of records > 2147483647 in one partition. method `getIteratorZipWithIndex` accepts `startIndex` <=0, which leads to negative index. This PR just adds a defensive check on `startIndex` to make sure it is > 0. ## How was this patch tested? Add a new unit test. You can merge this pull request into a Git repository by running: $ git pull https://github.com/wangmiao1981/spark zip Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/15639.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #15639 commit 1d3d4fec775b05bb4b4d8de225cf500ac661a2cd Author: Miao Wang Date: 2016-10-26T05:16:04Z add defensive check for zipWithIndex --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15428: [SPARK-17219][ML] enhanced NaN value handling in Bucketi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15428 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67547/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15428: [SPARK-17219][ML] enhanced NaN value handling in Bucketi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15428 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15428: [SPARK-17219][ML] enhanced NaN value handling in Bucketi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15428 **[Test build #67547 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67547/consoleFull)** for PR 15428 at commit [`2f98d31`](https://github.com/apache/spark/commit/2f98d31118413e61e1aa0431da402c41aa1ca5a6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15448: [SPARK-17108][SQL]: Fix BIGINT and INT comparison failur...
Github user weiqingy commented on the issue: https://github.com/apache/spark/pull/15448 Hi, @hvanhovell Could you review this PR again? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15638: [SPARK-18110][PYTHON] add missing parameter in Python fo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15638 **[Test build #67558 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67558/consoleFull)** for PR 15638 at commit [`e734e01`](https://github.com/apache/spark/commit/e734e01034b1fa4d3e3b3e48e7b233cd4008a40e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15638: [SPARK-18110][PYTHON] add missing parameter in Py...
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/15638 [SPARK-18110][PYTHON] add missing parameter in Python for RandomForest regression and classification ## What changes were proposed in this pull request? Add subsmaplingRate to randomForestClassifier Add varianceCol to randomForestRegressor In Python ## How was this patch tested? manual tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/felixcheung/spark pyrandomforest Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/15638.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #15638 commit e734e01034b1fa4d3e3b3e48e7b233cd4008a40e Author: Felix Cheung Date: 2016-10-26T05:02:51Z add parameters for randomforest --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15417: [SPARK-17851][SQL][TESTS] Make sure all test sqls...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15417#discussion_r85049260
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
---
@@ -1016,6 +1016,8 @@ class FilterPushdownSuite extends PlanTest {
val correctAnswer = x.where("x.a".attr === 5).join(y.where("y.a".attr
=== 5),
condition = Some("x.a".attr === Rand(10) && "y.b".attr === 5))
-comparePlans(Optimize.execute(originalQuery.analyze),
correctAnswer.analyze)
--- End diff --
Sorry, we are unable to merge this PR until you fix the above issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15596: [SPARK-18089][SQL] Remove CollectLimitExec
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15596 **[Test build #67557 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67557/consoleFull)** for PR 15596 at commit [`e919f4a`](https://github.com/apache/spark/commit/e919f4a9d3c55cfe7b28b9fd89709cc747e736e6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15417: [SPARK-17851][SQL][TESTS] Make sure all test sqls...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15417#discussion_r85049175
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/BooleanSimplificationSuite.scala
---
@@ -40,12 +40,16 @@ class BooleanSimplificationSuite extends PlanTest with
PredicateHelper {
PruneFilters) :: Nil
}
- val testRelation = LocalRelation('a.int, 'b.int, 'c.int, 'd.string)
-
- private def checkCondition(input: Expression, expected: Expression):
Unit = {
-val plan = testRelation.where(input).analyze
+ val testRelation = LocalRelation(
+'a.int, 'b.int, 'c.int, 'd.string, 'e.boolean, 'f.boolean, 'g.boolean,
'h.boolean)
+
+ private def checkCondition(
+ input: Expression,
+ expected: Expression,
+ relation: LocalRelation = testRelation): Unit = {
--- End diff --
You do not need to change the function interface, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15596: [SPARK-18089][SQL] Remove CollectLimitExec
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/15596#discussion_r85049115
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
---
@@ -298,6 +298,12 @@ case class WholeStageCodegenExec(child: SparkPlan)
extends UnaryExecNode with Co
override def outputPartitioning: Partitioning = child.outputPartitioning
override def outputOrdering: Seq[SortOrder] = child.outputOrdering
+ override def executeCollect(): Array[InternalRow] = child match {
+// This happens when the user is collecting results back to the
driver, we could skip
+// the shuffling and scan increasingly the RDD to get the limited
items.
+case g: GlobalLimitExec => g.executeCollect()
--- End diff --
Still think this is confusing as you said. Removed it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15417: [SPARK-17851][SQL][TESTS] Make sure all test sqls...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15417#discussion_r85049030
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisTest.scala
---
@@ -51,6 +51,14 @@ trait AnalysisTest extends PlanTest {
comparePlans(actualPlan, expectedPlan)
}
+ protected override def comparePlans(
+ plan1: LogicalPlan,
+ plan2: LogicalPlan,
+ checkAnalysis: Boolean = false): Unit = {
+// Analysis tests may have not been fully resolved, so skip
checkAnalysis.
+super.comparePlans(plan1, plan2, checkAnalysis = false)
--- End diff --
`super.comparePlans(plan1, plan2, checkAnalysis)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15596: [SPARK-18089][SQL] Remove CollectLimitExec
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15596 **[Test build #67556 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67556/consoleFull)** for PR 15596 at commit [`110a3e4`](https://github.com/apache/spark/commit/110a3e44f983ec90e7dbfafbfc9ce1932885c903). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15552: [SPARK-18007][SparkR][ML] update SparkR MLP - add inital...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/15552 merged to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15552: [SPARK-18007][SparkR][ML] update SparkR MLP - add...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15552 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15417: [SPARK-17851][SQL][TESTS] Make sure all test sqls in cat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15417 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12195: [Spark-14300][Docs][MLLIB]Scala MLlib examples code merg...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12195 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67554/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15417: [SPARK-17851][SQL][TESTS] Make sure all test sqls in cat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15417 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67546/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12195: [Spark-14300][Docs][MLLIB]Scala MLlib examples code merg...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12195 **[Test build #67554 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67554/consoleFull)** for PR 12195 at commit [`7bb5d9f`](https://github.com/apache/spark/commit/7bb5d9f5ab40e03e7e01bf44199d1860628138c9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12195: [Spark-14300][Docs][MLLIB]Scala MLlib examples code merg...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12195 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15417: [SPARK-17851][SQL][TESTS] Make sure all test sqls in cat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15417 **[Test build #67546 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67546/consoleFull)** for PR 15417 at commit [`87ed4da`](https://github.com/apache/spark/commit/87ed4da5468cbaa546fa43c110022de67d18cf3c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15637: [SPARK-18000] [SQL] Aggregation function for computing e...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15637 **[Test build #67555 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67555/consoleFull)** for PR 15637 at commit [`32478d1`](https://github.com/apache/spark/commit/32478d160356aec3cc07579a657b3e8fbd20e2bd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15552: [SPARK-18007][SparkR][ML] update SparkR MLP - add inital...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/15552 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15365: [SPARK-17157][SPARKR]: Add multiclass logistic regressio...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/15365 LGTM. Let's see if anyone has any other comments. Could you open a JIRA on Vector/SparseVector/DenseVector? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15636: [SPARK-18109][ML] Add instrumentation to GMM
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15636 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15636: [SPARK-18109][ML] Add instrumentation to GMM
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15636 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67549/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15636: [SPARK-18109][ML] Add instrumentation to GMM
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15636 **[Test build #67549 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67549/consoleFull)** for PR 15636 at commit [`069f377`](https://github.com/apache/spark/commit/069f377dcb925e9b2b54368f7b7932f3d276c504). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12195: [Spark-14300][Docs][MLLIB]Scala MLlib examples code merg...
Github user keypointt commented on the issue: https://github.com/apache/spark/pull/12195 I've also updated the description of this PR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15552: [SPARK-18007][SparkR][ML] update SparkR MLP - add inital...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15552 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67548/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15552: [SPARK-18007][SparkR][ML] update SparkR MLP - add inital...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15552 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12195: [Spark-14300][Docs][MLLIB]Scala MLlib examples code merg...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12195 **[Test build #67554 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67554/consoleFull)** for PR 12195 at commit [`7bb5d9f`](https://github.com/apache/spark/commit/7bb5d9f5ab40e03e7e01bf44199d1860628138c9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15552: [SPARK-18007][SparkR][ML] update SparkR MLP - add inital...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15552 **[Test build #67548 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67548/consoleFull)** for PR 15552 at commit [`4524c86`](https://github.com/apache/spark/commit/4524c863f2109d310af557c0c08886924e7b5a18). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15632: [SPARK-18105] fix buffer overflow in LZ4
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/15632#discussion_r85047248
--- Diff: core/src/main/java/org/apache/spark/io/LZ4BlockInputStream.java
---
@@ -197,7 +197,7 @@ private void refill() throws IOException {
readFully(buffer, originalLen);
break;
case COMPRESSION_METHOD_LZ4:
-if (compressedBuffer.length < originalLen) {
+if (compressedBuffer.length < compressedLen) {
--- End diff --
Does this possibly happen?
I go to check
https://github.com/jpountz/lz4-java/blob/master/src/java/net/jpountz/lz4/LZ4BlockOutputStream.java#L192
If the compressed lengh is more the original length, it will choose
`COMPRESSION_METHOD_RAW` as compress method, instead of
`COMPRESSION_METHOD_LZ4`. In other words, the compressed length is never more
then original length under `COMPRESSION_METHOD_LZ4`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15172: [SPARK-13331] AES support for over-the-wire encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15172 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67545/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15172: [SPARK-13331] AES support for over-the-wire encryption
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15172 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15172: [SPARK-13331] AES support for over-the-wire encryption
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15172 **[Test build #67545 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67545/consoleFull)** for PR 15172 at commit [`f3e2518`](https://github.com/apache/spark/commit/f3e2518dbe3b0297360925300ad86a3991760ff1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15615: [SPARK-17693] [SQL] Fixed Insert Failure To Data Source ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15615 **[Test build #67553 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67553/consoleFull)** for PR 15615 at commit [`bf81bb6`](https://github.com/apache/spark/commit/bf81bb66c136184facc67008df2929144234cb5a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15024: [SPARK-17470][SQL] unify path for data source tab...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15024#discussion_r85046747
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -741,16 +762,20 @@ object HiveExternalCatalog {
val STATISTICS_NUM_ROWS = STATISTICS_PREFIX + "numRows"
val STATISTICS_COL_STATS_PREFIX = STATISTICS_PREFIX + "colStats."
- def removeStatsProperties(metadata: CatalogTable): Map[String, String] =
{
-metadata.properties.filterNot { case (key, _) =>
key.startsWith(STATISTICS_PREFIX) }
+ // Ideally we should use `spark.sql.sources.location` to store the table
location, but as we have
+ // already used `path` to store it, we should keep it for backward
compatibility.
+ val TABLE_LOCATION = "path"
--- End diff --
the `path` option has special meaning(table location) only when it's used
to create data source tables. Other places like the streaming code path may not
have this semantic.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15615: [SPARK-17693] [SQL] Fixed Insert Failure To Data Source ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15615 Will merge it when the test can pass. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15629: [SQL][DOC] updating doc for JSON source to link to jsonl...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/15629 thanks, streaming are good ones. I'm not sure about changing the deprecated methods in SQLContext though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15635: Branch 1.6
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15635 @lklong Could you please close this? I guess you can have a better answer from user mailing list. Please check out http://spark.apache.org/community.html (This leaves a failure mark on each commit log in branch-1.6. Please see https://github.com/apache/spark/commits/branch-1.6) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15538: [SPARK-17993][SQL] Fix Parquet log output redirection
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15538 **[Test build #67552 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67552/consoleFull)** for PR 15538 at commit [`02df8c2`](https://github.com/apache/spark/commit/02df8c273ffac794bfb5bff6f3cc0ab9532264f9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15516: [SPARK-17961][SparkR][SQL] Add storageLevel to Da...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15516#discussion_r85046280
--- Diff: R/pkg/R/DataFrame.R ---
@@ -654,6 +654,33 @@ setMethod("unpersist",
x
})
+#' StorageLevel
+#'
+#' Get storage level of this SparkDataFrame.
+#'
+#' @param x the SparkDataFrame to get the storage level.
+#'
+#' @family SparkDataFrame functions
+#' @rdname storageLevel-methods
--- End diff --
this should be
`@rdname storageLevel` instead of
`@rdname storageLevel-methods`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15538: [SPARK-17993][SQL] Fix Parquet log output redirection
Github user mallman commented on the issue: https://github.com/apache/spark/pull/15538 So raising the log threshold looks like it didn't do anything for Jenkins, but when I run the tests locally it does just the trick. \*sigh\* Anyway, might as well push a rebase and see what happens. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15618: [SPARK-14914][CORE] Fix Resource not closed after...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/15618#discussion_r85046075
--- Diff:
core/src/main/scala/org/apache/spark/rdd/ReliableCheckpointRDD.scala ---
@@ -239,7 +239,14 @@ private[spark] object ReliableCheckpointRDD extends
Logging {
val fs = partitionerFilePath.getFileSystem(sc.hadoopConfiguration)
val fileInputStream = fs.open(partitionerFilePath, bufferSize)
val serializer = SparkEnv.get.serializer.newInstance()
- val deserializeStream = serializer.deserializeStream(fileInputStream)
+ val deserializeStream = try {
+serializer.deserializeStream(fileInputStream)
+ } catch {
+case e : Throwable =>
+ fileInputStream.close()
--- End diff --
I don't mean having the finally here on this line...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15618: [SPARK-14914][CORE] Fix Resource not closed after using,...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/15618 @HyukjinKwon So the idea is that you acquire resources required and dont need to track it by wrapping them in Utils.tryWithResource (similar to memory management in jvm). As an example: main/scala/org/apache/spark/rdd/ReliableCheckpointRDD.scala change will simply acquire the fileInputStream in the try and release it in the finally automatically - without needing to manage it via catch/rethrow, etc (ex: what if close() throws exception ?). Even core/src/test/scala/org/apache/spark/FileSuite.scala, core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala, etc change can be modelled the same way. You get the idea :-) This is essentially analogous to try-with-resources in java. Which is not to say it applies every where ofcourse : drawback is that unlike in java, you need to explicitly specify the finally action, which can be pita imo compared to java's idiom. Since you are anyway going through the pain of making all these changes to fix up code, might be a good idea to change it such that future tests will follow the same pattern. Thoughts ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15590: [SPARK-17949][SQL] A JVM object based aggregate operator
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/15590 lgtm1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15634: [SPARK-18103] [SQL] Rename *FileCatalog to *FileProvider
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15634 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15634: [SPARK-18103] [SQL] Rename *FileCatalog to *FileProvider
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15634 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67544/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15634: [SPARK-18103] [SQL] Rename *FileCatalog to *FileProvider
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15634 **[Test build #67544 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67544/consoleFull)** for PR 15634 at commit [`0776537`](https://github.com/apache/spark/commit/0776537cdb13863c22b948980bcc1e54c2221ddc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14957 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14957 **[Test build #67550 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67550/consoleFull)** for PR 14957 at commit [`5697911`](https://github.com/apache/spark/commit/56979118bfee1f2de3ac22c52280e8b36a14fc38). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14957 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67550/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15637: [SPARK-18000] [SQL] Aggregation function for computing e...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15637 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67551/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15637: [SPARK-18000] [SQL] Aggregation function for computing e...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15637 **[Test build #67551 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67551/consoleFull)** for PR 15637 at commit [`15eb372`](https://github.com/apache/spark/commit/15eb3721f56ac27bd90933ef7e66f3453eae4a75). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15637: [SPARK-18000] [SQL] Aggregation function for computing e...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15637 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15443: [SPARK-17881] [SQL] Aggregation function for generating ...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/15443 This pr is included in [a new pr](https://github.com/apache/spark/pull/15637), so I'll close this one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15443: [SPARK-17881] [SQL] Aggregation function for gene...
Github user wzhfy closed the pull request at: https://github.com/apache/spark/pull/15443 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15637: [SPARK-18000] [SQL] Aggregation function for computing e...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15637 **[Test build #67551 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67551/consoleFull)** for PR 15637 at commit [`15eb372`](https://github.com/apache/spark/commit/15eb3721f56ac27bd90933ef7e66f3453eae4a75). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
