[GitHub] spark issue #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests by cha...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20199 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85960/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20218: [SPARK-23000] [TEST-HADOOP2.6] Fix Flaky test suite Data...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20218 **[Test build #85959 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85959/testReport)** for PR 20218 at commit [`7924e28`](https://github.com/apache/spark/commit/7924e28d26623a0ba0a7a67cb6994e9ee0220677). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests by cha...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20199 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests by cha...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20199 **[Test build #85960 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85960/testReport)** for PR 20199 at commit [`c3d93c3`](https://github.com/apache/spark/commit/c3d93c3d5c14f080ed5dbf668c0b1241bd82f2fc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20234: [SPARK-19732] [Follow-up] Document behavior changes made...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20234 **[Test build #85963 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85963/testReport)** for PR 20234 at commit [`ff30553`](https://github.com/apache/spark/commit/ff30553092a7bfe8d9aac3fc1f89b99ff679a2aa). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20234: [SPARK-19732] [Follow-up] Document behavior changes made...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20234 cc @rberenguel @ueshin @HyukjinKwon --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20234: [SPARK-19732] [Follow-up] Document behavior chang...
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/20234 [SPARK-19732] [Follow-up] Document behavior changes made in na.fill and fillna ## What changes were proposed in this pull request? https://github.com/apache/spark/pull/18164 introduces the behavior changes. We need to document it. ## How was this patch tested? N/A You can merge this pull request into a Git repository by running: $ git pull https://github.com/gatorsmile/spark docBehaviorChange Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20234.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20234 commit 89f00867f319cddf5ff49442b9ab38f1cefe837e Author: gatorsmile Date: 2018-01-11T03:10:11Z fix commit 09193499f522dee74d99262347e254e028e9d636 Author: gatorsmile Date: 2018-01-11T03:10:35Z clean commit ff30553092a7bfe8d9aac3fc1f89b99ff679a2aa Author: gatorsmile Date: 2018-01-11T03:11:56Z fix --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20222 @shaneknapp Could we bump the job timeout? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20222 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85955/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20222 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20222 **[Test build #85955 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85955/testReport)** for PR 20222 at commit [`5eded03`](https://github.com/apache/spark/commit/5eded033a0b352e7a799c7890131d8075475c8ff). * This patch **fails from timeout after a configured wait of \`250m\`**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20013: [SPARK-20657][core] Speed up rendering of the sta...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20013 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20013 thanks, merging to master/2.3! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20232: [SPARK-23042][ML] Use OneHotEncoderModel to encode label...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20232 **[Test build #85962 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85962/testReport)** for PR 20232 at commit [`39afee1`](https://github.com/apache/spark/commit/39afee1c22a23692196993ca446dba532591a9cf). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20233: [SPARK-23043][BUILD] Upgrade json4s to 3.5.3
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20233 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20216: [SPARK-23024][WEB-UI]Spark ui about the contents of the ...
Github user guoxiaolongzte commented on the issue: https://github.com/apache/spark/pull/20216 No, just hide the table, in fact, the data is already on the page, but we can not see. When we refresh the page, it will re-show all the data. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20233: [SPARK-23043][BUILD] Upgrade json4s to 3.5.3
GitHub user shimamoto opened a pull request: https://github.com/apache/spark/pull/20233 [SPARK-23043][BUILD] Upgrade json4s to 3.5.3 ## What changes were proposed in this pull request? Spark still use a few years old version 3.2.11. This change is to upgrade json4s to 3.5.3. Note that this change does not include the Jackson update because the Jackson version referenced in json4s 3.5.3 is 2.8.4, which has a security vulnerability ([see](https://issues.apache.org/jira/browse/SPARK-20433)). ## How was this patch tested? Existing unit tests and build. You can merge this pull request into a Git repository by running: $ git pull https://github.com/shimamoto/spark upgrade-json4s Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20233.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20233 commit e9cf6061a883abbee77ce50f5aa15416f7b86028 Author: shimamoto Date: 2018-01-10T12:51:40Z Upgrade json4s to 3.5.3 version --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20216: [SPARK-23024][WEB-UI]Spark ui about the contents of the ...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/20216 But @guoxiaolongzte your change doesn't cause it to load less data, right? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20231: [SPARK-23000][TEST-HADOOP2.6] Fix Flaky test suite DataS...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20231 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85954/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20231: [SPARK-23000][TEST-HADOOP2.6] Fix Flaky test suite DataS...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20231 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/20230#discussion_r160926186 --- Diff: external/docker/spark-test/base/Dockerfile --- @@ -15,14 +15,14 @@ # limitations under the License. # -FROM ubuntu:precise +FROM ubuntu:xenial --- End diff -- This is probably a good change, but is it required in order to get Java 8? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20231: [SPARK-23000][TEST-HADOOP2.6] Fix Flaky test suite DataS...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20231 **[Test build #85954 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85954/testReport)** for PR 20231 at commit [`39df77f`](https://github.com/apache/spark/commit/39df77f77838bb59a35ceb4919b6903559e1b270). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20232: [SPARK-23042][ML] Use OneHotEncoderModel to encode label...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20232 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20232: [SPARK-23042][ML] Use OneHotEncoderModel to encode label...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20232 **[Test build #85961 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85961/testReport)** for PR 20232 at commit [`de4fec9`](https://github.com/apache/spark/commit/de4fec91c04ae904ac1bc41e644b7687a6ca69fd). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20232: [SPARK-23042][ML] Use OneHotEncoderModel to encode label...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20232 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85961/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20229: [SPARK-23037][ML] Update RFormula to use VectorSizeHint ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20229 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20229: [SPARK-23037][ML] Update RFormula to use VectorSizeHint ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20229 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85958/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20229: [SPARK-23037][ML] Update RFormula to use VectorSizeHint ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20229 **[Test build #85958 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85958/testReport)** for PR 20229 at commit [`2f64147`](https://github.com/apache/spark/commit/2f641474d4d701e439c50b1f05986230d9804b44). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20163: [SPARK-22966][PySpark] Spark SQL should handle Python UD...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/20163 +1 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20229: [SPARK-23037][ML] Update RFormula to use VectorSizeHint ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20229 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20229: [SPARK-23037][ML] Update RFormula to use VectorSizeHint ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20229 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85957/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20229: [SPARK-23037][ML] Update RFormula to use VectorSizeHint ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20229 **[Test build #85957 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85957/testReport)** for PR 20229 at commit [`f67ce68`](https://github.com/apache/spark/commit/f67ce68627c28697319e44d22ef084cef93cc999). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20232: [SPARK-23042][ML] Use OneHotEncoderModel to encode label...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20232 **[Test build #85961 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85961/testReport)** for PR 20232 at commit [`de4fec9`](https://github.com/apache/spark/commit/de4fec91c04ae904ac1bc41e644b7687a6ca69fd). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20214: [SPARK-23023][SQL] Cast field data to strings in showStr...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20214 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85956/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20214: [SPARK-23023][SQL] Cast field data to strings in showStr...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20214 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20214: [SPARK-23023][SQL] Cast field data to strings in showStr...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20214 **[Test build #85956 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85956/testReport)** for PR 20214 at commit [`9cf9954`](https://github.com/apache/spark/commit/9cf995461990e46c007405344481cd802a0d6501). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20232: [SPARK-23042][ML] Use OneHotEncoderModel to encod...
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/20232 [SPARK-23042][ML] Use OneHotEncoderModel to encode labels in MultilayerPerceptronClassifier ## What changes were proposed in this pull request? In MultilayerPerceptronClassifier, we use RDD operation to encode labels for now. I think we should use ML's OneHotEncoderEstimator/Model to do the encoding. ## How was this patch tested? Existing tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/viirya/spark-1 SPARK-23042 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20232.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20232 commit de4fec91c04ae904ac1bc41e644b7687a6ca69fd Author: Liang-Chi Hsieh Date: 2018-01-11T10:45:17Z Use OneHotEncoderModel to encode labels. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20163: [SPARK-22966][PySpark] Spark SQL should handle Python UD...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20163 One more SGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20211: [SPARK-23011][PYTHON][SQL] Prepend missing groupi...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20211#discussion_r160918997 --- Diff: python/pyspark/sql/group.py --- @@ -233,6 +233,27 @@ def apply(self, udf): | 2| 1.1094003924504583| +---+---+ +Notes on grouping column: --- End diff -- From a SQL background, I think we should add grouping keys to the input of UDF. Sometimes users do need to read the grouping keys when aggregating, and we should give users a way to do it. BTW this is also consistent with Dataset, see `KeyValueGroupedDataset.mapGroups`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20163: [SPARK-22966][PySpark] Spark SQL should handle Python UD...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20163 SGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests by cha...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20199 **[Test build #85960 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85960/testReport)** for PR 20199 at commit [`c3d93c3`](https://github.com/apache/spark/commit/c3d93c3d5c14f080ed5dbf668c0b1241bd82f2fc). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18991: [SPARK-21783][SQL] Turn on ORC filter push-down by defau...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18991 Yea let's add some, I'm curious to see how well PPD works in ORC, since for parquet PPD doesn't work well and we disable record level filtering for parquet. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20215: [SPARK-23001] [SQL] Fix NullPointerException when...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20215 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20215: [SPARK-23001] [SQL] Fix NullPointerException when DESC a...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20215 thanks, merging to master/2.3! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20218: [SPARK-23000] [TEST-HADOOP2.6] Fix Flaky test suite Data...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20218 **[Test build #85959 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85959/testReport)** for PR 20218 at commit [`7924e28`](https://github.com/apache/spark/commit/7924e28d26623a0ba0a7a67cb6994e9ee0220677). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20218: [SPARK-23000] [TEST-HADOOP2.6] Fix Flaky test suite Data...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20218 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20069: [SPARK-22895] [SQL] Push down the deterministic p...
Github user jinxing64 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20069#discussion_r160910495
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -851,7 +851,7 @@ object PushDownPredicate extends Rule[LogicalPlan] with
PredicateHelper {
case filter @ Filter(condition, union: Union) =>
// Union could change the rows, so non-deterministic predicate can't
be pushed down
- val (pushDown, stayUp) =
splitConjunctivePredicates(condition).span(_.deterministic)
+ val (pushDown, stayUp) =
splitConjunctivePredicates(condition).partition(_.deterministic)
--- End diff --
@viirya
IIUC, with `span`, the deterministic predicates after the first
non-deterministic will not be pushed down, but with `partition`, all
deterministic predicates will be pushed down.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20229: [SPARK-23037][ML] Update RFormula to use VectorSizeHint ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20229 **[Test build #85958 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85958/testReport)** for PR 20229 at commit [`2f64147`](https://github.com/apache/spark/commit/2f641474d4d701e439c50b1f05986230d9804b44). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20229: [SPARK-23037][ML] Update RFormula to use VectorSizeHint ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20229 **[Test build #85957 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85957/testReport)** for PR 20229 at commit [`f67ce68`](https://github.com/apache/spark/commit/f67ce68627c28697319e44d22ef084cef93cc999). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16592: [SPARK-19235] [SQL] [TESTS] Enable Test Cases in ...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16592#discussion_r160889706
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
---
@@ -1082,24 +1173,21 @@ class DDLSuite extends QueryTest with
SharedSQLContext with BeforeAndAfterEach {
convertToDatasourceTable(catalog, tableIdent)
}
assert(catalog.getTableMetadata(tableIdent).storage.locationUri.isDefined)
-assert(catalog.getTableMetadata(tableIdent).storage.properties.isEmpty)
+
assert(normalizeSerdeProp(catalog.getTableMetadata(tableIdent).storage.properties).isEmpty)
assert(catalog.getPartition(tableIdent,
partSpec).storage.locationUri.isDefined)
-assert(catalog.getPartition(tableIdent,
partSpec).storage.properties.isEmpty)
+assert(
+ normalizeSerdeProp(catalog.getPartition(tableIdent,
partSpec).storage.properties).isEmpty)
+
// Verify that the location is set to the expected string
def verifyLocation(expected: URI, spec: Option[TablePartitionSpec] =
None): Unit = {
val storageFormat = spec
.map { s => catalog.getPartition(tableIdent, s).storage }
.getOrElse { catalog.getTableMetadata(tableIdent).storage }
- if (isDatasourceTable) {
-if (spec.isDefined) {
- assert(storageFormat.properties.isEmpty)
- assert(storageFormat.locationUri === Some(expected))
-} else {
- assert(storageFormat.locationUri === Some(expected))
-}
- } else {
-assert(storageFormat.locationUri === Some(expected))
- }
+ // TODO(gatorsmile): fix the bug in alter table set location.
+ // if (isUsingHiveMetastore) {
+ // assert(storageFormat.properties.get("path") === expected)
--- End diff --
Do we need to fix this bug and satify this test case?
When porting these test cases, a bug of SET LOCATION is found. path is not
set when the location is changed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20231: [SPARK-23000][TEST-HADOOP2.6] Fix Flaky test suite DataS...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20231 **[Test build #85954 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85954/testReport)** for PR 20231 at commit [`39df77f`](https://github.com/apache/spark/commit/39df77f77838bb59a35ceb4919b6903559e1b270). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20214: [SPARK-23023][SQL] Cast field data to strings in showStr...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20214 **[Test build #85956 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85956/testReport)** for PR 20214 at commit [`9cf9954`](https://github.com/apache/spark/commit/9cf995461990e46c007405344481cd802a0d6501). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20222 **[Test build #85955 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85955/testReport)** for PR 20222 at commit [`5eded03`](https://github.com/apache/spark/commit/5eded033a0b352e7a799c7890131d8075475c8ff). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20214: [SPARK-23023][SQL] Cast field data to strings in showStr...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/20214 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20231: [SPARK-23000][TEST-HADOOP2.6] Fix Flaky test suit...
GitHub user sameeragarwal opened a pull request: https://github.com/apache/spark/pull/20231 [SPARK-23000][TEST-HADOOP2.6] Fix Flaky test suite DataSourceWithHiveMetastoreCatalogSuite in Spark 2.3 [WIP] ## What changes were proposed in this pull request? Another attempt at reproducing the underlying failure (sbt/hadoop 2.6 against 2.3) by running all tests. ## How was this patch tested? N/A You can merge this pull request into a Git repository by running: $ git pull https://github.com/sameeragarwal/spark bugfix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20231.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20231 commit 39df77f77838bb59a35ceb4919b6903559e1b270 Author: Sameer Agarwal Date: 2018-01-11T08:01:38Z Run All Tests --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/20222 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20214: [SPARK-23023][SQL] Cast field data to strings in showStr...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20214 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20222 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20230 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85953/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20214: [SPARK-23023][SQL] Cast field data to strings in showStr...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20214 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85951/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20230 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20230 **[Test build #85953 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85953/testReport)** for PR 20230 at commit [`cc3321c`](https://github.com/apache/spark/commit/cc3321c20fd0dc2ef75a8740b5c0292beef98beb). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20222 **[Test build #85952 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85952/testReport)** for PR 20222 at commit [`5eded03`](https://github.com/apache/spark/commit/5eded033a0b352e7a799c7890131d8075475c8ff). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20222 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85952/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20214: [SPARK-23023][SQL] Cast field data to strings in showStr...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20214 **[Test build #85951 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85951/testReport)** for PR 20214 at commit [`9cf9954`](https://github.com/apache/spark/commit/9cf995461990e46c007405344481cd802a0d6501). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
