[GitHub] spark issue #20665: [SPARK-23499][MESOS] Support for priority queues in Meso...
Github user pgillet commented on the issue: https://github.com/apache/spark/pull/20665 @skonto I attached a screenshot in the JIRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20679: [SPARK-23514] Use SessionState.newHadoopConf() to propag...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20679 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20693: [SPARK-23523] [SQL] [FOLLOWUP] Minor refactor of Optimiz...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20693 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20692: [SPARK-23531][SQL] Show attribute type in explain
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20692 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87781/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20692: [SPARK-23531][SQL] Show attribute type in explain
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20692 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20692: [SPARK-23531][SQL] Show attribute type in explain
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20692 **[Test build #87781 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87781/testReport)** for PR 20692 at commit [`e076732`](https://github.com/apache/spark/commit/e07673284027b3cf9d13dc9fc8527f7d7c7d31c2). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20694: [SPARK-23173][SQL] Avoid creating corrupt parquet files ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20694 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20694: [SPARK-23173][SQL] Avoid creating corrupt parquet files ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20694 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87776/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20694: [SPARK-23173][SQL] Avoid creating corrupt parquet files ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20694 **[Test build #87776 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87776/testReport)** for PR 20694 at commit [`1cd1919`](https://github.com/apache/spark/commit/1cd19196cf46e15aaf1636240d053996e623370b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20684: [SPARK-23523] [SQL] Fix the incorrect result caused by t...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20684 I see. Thank you for confirmation, @gatorsmile ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20679: [SPARK-23514] Use SessionState.newHadoopConf() to propag...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20679 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87774/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20679: [SPARK-23514] Use SessionState.newHadoopConf() to propag...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20679 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20679: [SPARK-23514] Use SessionState.newHadoopConf() to propag...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20679 **[Test build #87774 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87774/testReport)** for PR 20679 at commit [`b37f24f`](https://github.com/apache/spark/commit/b37f24f372bb45ff9b8380222e0eb7e6d8819e58). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20343: [SPARK-23167][SQL] Add TPCDS queries v2.7 in TPCDSQueryS...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/20343 ping @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17671: [SPARK-20368][PYSPARK] Provide optional support for Sent...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17671 ð --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17671: [SPARK-20368][PYSPARK] Provide optional support for Sent...
Github user kxepal commented on the issue: https://github.com/apache/spark/pull/17671 @holdenk mmm...sweet! That may work and even makes integration process more flexible. Sentry integration wrapper would be trivial with this feature. Thanks! For the future reference: https://github.com/apache/spark/commit/afae8f2bc82597593595af68d1aa2d802210ea8b --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17671: [SPARK-20368][PYSPARK] Provide optional support f...
Github user kxepal closed the pull request at: https://github.com/apache/spark/pull/17671 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20693: [SPARK-23523] [SQL] [FOLLOWUP] Minor refactor of Optimiz...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20693 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1156/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19425: [SPARK-22196][Core] Combine multiple input splits into a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19425 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20693: [SPARK-23523] [SQL] [FOLLOWUP] Minor refactor of Optimiz...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20693 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20695: [SPARK-21741][ML][PySpark] Python API for DataFrame-base...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20695 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20695: [SPARK-21741][ML][PySpark] Python API for DataFrame-base...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20695 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87784/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20695: [SPARK-21741][ML][PySpark] Python API for DataFrame-base...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20695 **[Test build #87784 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87784/testReport)** for PR 20695 at commit [`001ff46`](https://github.com/apache/spark/commit/001ff4685b7b40e14e678a004eee2a88ee6e1356). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20686: [SPARK-22915][MLlib] Streaming tests for spark.ml.featur...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20686 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20686: [SPARK-22915][MLlib] Streaming tests for spark.ml.featur...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20686 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87780/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20693: [SPARK-23523] [SQL] [FOLLOWUP] Minor refactor of Optimiz...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20693 **[Test build #87787 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87787/testReport)** for PR 20693 at commit [`0a3d84a`](https://github.com/apache/spark/commit/0a3d84af6cec90df72ebaf3d16f231cdb5c627aa). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20686: [SPARK-22915][MLlib] Streaming tests for spark.ml.featur...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20686 **[Test build #87780 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87780/testReport)** for PR 20686 at commit [`836a173`](https://github.com/apache/spark/commit/836a1730ecaa8ff5232ed5ec7cad70925f1da0f6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20680: [SPARK-23517][PYTHON] Make `pyspark.util._excepti...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20680 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20680: [SPARK-23517][PYTHON] Make `pyspark.util._exception_mess...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20680 Will take an action for this PR in branch-2.3 if something goes wrong with 2.3 release somehow. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20680: [SPARK-23517][PYTHON] Make `pyspark.util._exception_mess...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20680 Merged to master and branch-2.3. Thanks for reviewing this, @ueshin, @viirya, @BryanCutler, and @holdenk. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20696 **[Test build #87786 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87786/testReport)** for PR 20696 at commit [`92b2f31`](https://github.com/apache/spark/commit/92b2f317aca1c8fe96e4448c8d6dcedce02c9429). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20696 cc @skliarpawlo @gatorsmile @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20696 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN CO...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20696 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1155/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE CO...
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20696#discussion_r171285332
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
---
@@ -1597,6 +1597,7 @@ abstract class DDLSuite extends QueryTest with
SQLTestUtils {
// Ensure that change column will preserve other metadata fields.
sql("ALTER TABLE dbx.tab1 CHANGE COLUMN col1 col1 INT COMMENT 'this is
col1'")
assert(getMetadata("col1").getString("key") == "value")
+assert(getMetadata("col1").getString("comment") == "this is col1")
--- End diff --
I didn't verify the comment is correctly set to the column metadata, and
that caused the issue. :(
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20696: [SPARK-23525] [SQL] Support ALTER TABLE CHANGE CO...
GitHub user jiangxb1987 opened a pull request: https://github.com/apache/spark/pull/20696 [SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN COMMENT for external hive table ## What changes were proposed in this pull request? The following query doesn't work as expected: ``` CREATE EXTERNAL TABLE ext_table(a STRING, b INT, c STRING) PARTITIONED BY (d STRING) LOCATION 'sql/core/spark-warehouse/ext_table'; ALTER TABLE ext_table CHANGE a a STRING COMMENT "new comment"; DESC ext_table; ``` The comment of column `a` is not updated, that's because `HiveExternalCatalog.doAlterTable` ignores table schema changes. To fix the issue, we should call `doAlterTableDataSchema` instead of `doAlterTable`. ## How was this patch tested? Updated `DDLSuite.testChangeColumn`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jiangxb1987/spark alterColumnComment Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20696.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20696 commit 92b2f317aca1c8fe96e4448c8d6dcedce02c9429 Author: Xingbo Jiang Date: 2018-02-28T15:29:19Z support change column comment for external hive table --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20678: [SPARK-23380][PYTHON] Adds a conf for Arrow fallback in ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20678 **[Test build #87785 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87785/testReport)** for PR 20678 at commit [`ed30c20`](https://github.com/apache/spark/commit/ed30c205d95a6555475a06376f0d88e53e2f3da3). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/19222#discussion_r171283123
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/vectorized/ArrowColumnVector.java
---
@@ -377,8 +379,9 @@ final UTF8String getUTF8String(int rowId) {
if (stringResult.isSet == 0) {
return null;
} else {
-return UTF8String.fromAddress(null,
- stringResult.buffer.memoryAddress() + stringResult.start,
+mb.setAddressAndSize(stringResult.buffer.memoryAddress(),
stringResult.buffer.capacity());
--- End diff --
Good catch, done
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20678: [SPARK-23380][PYTHON] Adds a conf for Arrow fallback in ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20678 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20678: [SPARK-23380][PYTHON] Adds a conf for Arrow fallback in ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20678 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1154/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/19222#discussion_r171282943
--- Diff:
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java ---
@@ -195,15 +205,15 @@ private static int numBytesForFirstByte(final byte b)
{
* Returns the number of bytes
*/
public int numBytes() {
-return numBytes;
+return (int)base.size();
--- End diff --
Finally, I removed assertion due to performance reason.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20667: [SPARK-23508][CORE] Fix BlockmanagerId in case blockMana...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/20667 Thanks @cloud-fan @jiangxb1987 @kiszk @Ngone51 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20665: [SPARK-23499][MESOS] Support for priority queues in Meso...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/20665 @pgillet since you are modifying the ui could you add a screenshot in the description? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20670: [SPARK-23405] Generate additional constraints for Join's...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20670 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20670: [SPARK-23405] Generate additional constraints for Join's...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20670 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87772/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20670: [SPARK-23405] Generate additional constraints for Join's...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20670 **[Test build #87772 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87772/testReport)** for PR 20670 at commit [`ed5c170`](https://github.com/apache/spark/commit/ed5c170c35d8786df241921ac19d95520ace3836). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20695: [SPARK-21741][ML][PySpark] Python API for DataFrame-base...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20695 **[Test build #87784 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87784/testReport)** for PR 20695 at commit [`001ff46`](https://github.com/apache/spark/commit/001ff4685b7b40e14e678a004eee2a88ee6e1356). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20695: [SPARK-21741][ML][PySpark] Python API for DataFrame-base...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20695 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1153/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20695: [SPARK-21741][ML][PySpark] Python API for DataFrame-base...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20695 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20576: [SPARK-23389][CORE]When the shuffle dependency sp...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20576#discussion_r171279296
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala ---
@@ -188,9 +188,8 @@ private[spark] object SortShuffleManager extends
Logging {
log.debug(s"Can't use serialized shuffle for shuffle $shufId because
the serializer, " +
s"${dependency.serializer.getClass.getName}, does not support
object relocation")
false
-} else if (dependency.aggregator.isDefined) {
- log.debug(
-s"Can't use serialized shuffle for shuffle $shufId because an
aggregator is defined")
+} else if (dependency.mapSideCombine) {
+ require(dependency.aggregator.isDefined, "Map-side combine without
Aggregator specified!")
--- End diff --
can we move this `require` to the constructor of `ShuffleDependency`? It
appears many times in the codebase.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20667: [SPARK-23508][CORE] Fix BlockmanagerId in case bl...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20667 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20576: [SPARK-23389][CORE]When the shuffle dependency sp...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20576#discussion_r171278411
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala ---
@@ -188,9 +188,8 @@ private[spark] object SortShuffleManager extends
Logging {
log.debug(s"Can't use serialized shuffle for shuffle $shufId because
the serializer, " +
s"${dependency.serializer.getClass.getName}, does not support
object relocation")
false
-} else if (dependency.aggregator.isDefined) {
- log.debug(
-s"Can't use serialized shuffle for shuffle $shufId because an
aggregator is defined")
--- End diff --
can we keep the log with a little update?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20667: [SPARK-23508][CORE] Fix BlockmanagerId in case blockMana...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20667 thanks, merging to master/2.3/2.2! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20670: [SPARK-23405] Generate additional constraints for...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20670#discussion_r171276798
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/InferFiltersFromConstraintsSuite.scala
---
@@ -192,4 +192,17 @@ class InferFiltersFromConstraintsSuite extends
PlanTest {
comparePlans(Optimize.execute(original.analyze), correct.analyze)
}
+
+ test("SPARK-23405: left-semi equal-join should filter out null join keys
on both sides") {
+val x = testRelation.subquery('x)
+val y = testRelation.subquery('y)
+val condition = Some("x.a".attr === "y.a".attr)
+val originalQuery = x.join(y, LeftSemi, condition).analyze
+val left = x.where(IsNotNull('a))
+val right = y.where(IsNotNull('a))
+val correctAnswer = left.join(right, LeftSemi, condition)
+.analyze
--- End diff --
this doesn't need to be in a new line
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20693: [SPARK-23523] [SQL] [FOLLOWUP] Minor refactor of Optimiz...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20693 **[Test build #87783 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87783/testReport)** for PR 20693 at commit [`a3cf3ca`](https://github.com/apache/spark/commit/a3cf3ca98298ec43a7163e53e851a32e483f4c42). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20695: [SPARK-21741][ML][PySpark] Python API for DataFrame-base...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20695 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20695: [SPARK-21741][ML][PySpark] Python API for DataFrame-base...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20695 **[Test build #87782 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87782/testReport)** for PR 20695 at commit [`7d3cb1b`](https://github.com/apache/spark/commit/7d3cb1b131a4a459d78b7d50479e241680f2e544). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20695: [SPARK-21741][ML][PySpark] Python API for DataFrame-base...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20695 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87782/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20693: [SPARK-23523] [SQL] [FOLLOWUP] Minor refactor of Optimiz...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20693 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20693: [SPARK-23523] [SQL] [FOLLOWUP] Minor refactor of Optimiz...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20693 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1152/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20693: [SPARK-23523] [SQL] [FOLLOWUP] Minor refactor of ...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20693#discussion_r171275557
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LocalRelation.scala
---
@@ -43,11 +43,10 @@ object LocalRelation {
}
}
-case class LocalRelation(
-output: Seq[Attribute],
-data: Seq[InternalRow] = Nil,
-// Indicates whether this relation has data from a streaming source.
-override val isStreaming: Boolean = false)
+case class LocalRelation(output: Seq[Attribute],
--- End diff --
although we should not include this style change in the original commit,
since it's already there, let's not bother about reverting it back.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20693: [SPARK-23523] [SQL] [FOLLOWUP] Minor refactor of Optimiz...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20693 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20695: [SPARK-21741][ML][PySpark] Python API for DataFrame-base...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20695 **[Test build #87782 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87782/testReport)** for PR 20695 at commit [`7d3cb1b`](https://github.com/apache/spark/commit/7d3cb1b131a4a459d78b7d50479e241680f2e544). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20695: [SPARK-21741][ML][PySpark] Python API for DataFrame-base...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20695 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1151/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20695: [SPARK-21741][ML][PySpark] Python API for DataFrame-base...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20695 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171268197
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
---
@@ -22,7 +22,7 @@ import scala.collection.mutable.ArrayBuffer
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext,
ExprCode}
+import org.apache.spark.sql.catalyst.expressions.codegen._
--- End diff --
ditto
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20686: [SPARK-22915][MLlib] Streaming tests for spark.ml.featur...
Github user attilapiros commented on the issue: https://github.com/apache/spark/pull/20686 Ignored tests where issues found during streaming: - OneHotEncoderSuite / "input column without ML attribute" - RFormulaSuite / "label column already exists but is not numeric type" - VectorAssemblerSuite / "VectorAssembler" - VectorAssemblerSuite / "ML attributes" From this problems new jira issues can be created when my PR is accepted. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20692: [SPARK-23531][SQL] Show attribute type in explain
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20692 **[Test build #87781 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87781/testReport)** for PR 20692 at commit [`e076732`](https://github.com/apache/spark/commit/e07673284027b3cf9d13dc9fc8527f7d7c7d31c2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20692: [SPARK-23531][SQL] Show attribute type in explain
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20692 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20692: [SPARK-23531][SQL] Show attribute type in explain
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20692 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1150/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20686: [SPARK-22915][MLlib] Streaming tests for spark.ml.featur...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20686 **[Test build #87780 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87780/testReport)** for PR 20686 at commit [`836a173`](https://github.com/apache/spark/commit/836a1730ecaa8ff5232ed5ec7cad70925f1da0f6). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171264271
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+ val javaType: ExprType
+
+ // Whether we can directly access the evaluation value anywhere.
+ // For example, a variable created outside a method can not be accessed
inside the method.
+ // For such cases, we may need to pass the evaluation as parameter.
+ val canDirectAccess: Boolean
+}
+
+object ExprValue {
+ implicit def exprValueToString(exprValue: ExprValue): String =
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends
ExprValue {
+ override def toString: String = value
+ override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+ def apply(value: String, javaType: ExprType): LiteralValue = new
LiteralValue(value, javaType)
+ def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
--- End diff --
why isn't this fixed like for `GlobalValue`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171264038
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -31,7 +31,7 @@ import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.ScalaReflection.universe.TermName
import org.apache.spark.sql.catalyst.encoders.RowEncoder
import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext,
ExprCode}
+import org.apache.spark.sql.catalyst.expressions.codegen._
--- End diff --
can we list the needed classes instead?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171263916
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/ExprValue.scala
---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.codegen
+
+import scala.language.implicitConversions
+
+import org.apache.spark.sql.types.DataType
+
+// An abstraction that represents the evaluation result of [[ExprCode]].
+abstract class ExprValue {
+
+ val javaType: ExprType
+
+ // Whether we can directly access the evaluation value anywhere.
+ // For example, a variable created outside a method can not be accessed
inside the method.
+ // For such cases, we may need to pass the evaluation as parameter.
+ val canDirectAccess: Boolean
+}
+
+object ExprValue {
+ implicit def exprValueToString(exprValue: ExprValue): String =
exprValue.toString
+}
+
+// A literal evaluation of [[ExprCode]].
+class LiteralValue(val value: String, val javaType: ExprType) extends
ExprValue {
+ override def toString: String = value
+ override val canDirectAccess: Boolean = true
+}
+
+object LiteralValue {
+ def apply(value: String, javaType: ExprType): LiteralValue = new
LiteralValue(value, javaType)
+ def unapply(literal: LiteralValue): Option[(String, ExprType)] =
+Some((literal.value, literal.javaType))
+}
+
+// A variable evaluation of [[ExprCode]].
+case class VariableValue(
+val variableName: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
+ override def toString: String = variableName
+}
+
+// A statement evaluation of [[ExprCode]].
+case class StatementValue(
+val statement: String,
+val javaType: ExprType,
+val canDirectAccess: Boolean = false) extends ExprValue {
+ override def toString: String = statement
+}
+
+// A global variable evaluation of [[ExprCode]].
+case class GlobalValue(val value: String, val javaType: ExprType) extends
ExprValue {
+ override def toString: String = value
+ override val canDirectAccess: Boolean = true
+}
+
+case object TrueLiteral extends LiteralValue("true", ExprType("boolean",
true))
+case object FalseLiteral extends LiteralValue("false", ExprType("boolean",
true))
+
+// Represents the java type of an evaluation.
+case class ExprType(val typeName: String, val isPrimitive: Boolean)
--- End diff --
why is this `isPrimitive` needed? If I am not wrong, we have somewhere a
method to check whether a type is primitive or not. I think we can get rid of
this and use that method when needed, or at least store this using that method
instead of passing it every time.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20695: [SPARK-21741][ML][PySpark] Python API for DataFrame-base...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20695 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87778/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20695: [SPARK-21741][ML][PySpark] Python API for DataFrame-base...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20695 **[Test build #87778 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87778/testReport)** for PR 20695 at commit [`488d45a`](https://github.com/apache/spark/commit/488d45aa9cb9c4de96453d1f2c06f83db2b1ef77). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class Summarizer(object):` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20695: [SPARK-21741][ML][PySpark] Python API for DataFrame-base...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20695 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20693: [SPARK-23523] [SQL] [FOLLOWUP] Minor refactor of Optimiz...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20693 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20693: [SPARK-23523] [SQL] [FOLLOWUP] Minor refactor of Optimiz...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20693 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87775/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20693: [SPARK-23523] [SQL] [FOLLOWUP] Minor refactor of Optimiz...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20693 **[Test build #87775 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87775/testReport)** for PR 20693 at commit [`a3cf3ca`](https://github.com/apache/spark/commit/a3cf3ca98298ec43a7163e53e851a32e483f4c42). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class LocalRelation(output: Seq[Attribute],` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20208: [SPARK-23007][SQL][TEST] Add schema evolution test suite...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20208 **[Test build #87779 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87779/testReport)** for PR 20208 at commit [`6ae471c`](https://github.com/apache/spark/commit/6ae471c8ecaae3eb3888eecaac1c4e7552bedcc6). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20208: [SPARK-23007][SQL][TEST] Add schema evolution test suite...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20208 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20208: [SPARK-23007][SQL][TEST] Add schema evolution test suite...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20208 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1149/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20043: [SPARK-22856][SQL] Add wrappers for codegen outpu...
Github user mgaido91 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20043#discussion_r171262081
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -323,7 +323,8 @@ class CodegenContext {
case _: StructType | _: ArrayType | _: MapType => s"$value =
$initCode.copy();"
case _ => s"$value = $initCode;"
}
-ExprCode(code, "false", value)
+ExprCode(code, FalseLiteral,
+ GlobalValue(value, ExprType(this, dataType)))
--- End diff --
nit: this can go on one line
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20208: [SPARK-23007][SQL][TEST] Add schema evolution test suite...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20208 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20692: [SPARK-23531][SQL] Show attribute type in explain
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20692 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20692: [SPARK-23531][SQL] Show attribute type in explain
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20692 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87773/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20692: [SPARK-23531][SQL] Show attribute type in explain
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20692 **[Test build #87773 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87773/testReport)** for PR 20692 at commit [`0cbd0c5`](https://github.com/apache/spark/commit/0cbd0c5368115c2ac06928e163dcbc3ccaa10b83). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20695: [SPARK-21741][ML][PySpark] Python API for DataFrame-base...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20695 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20695: [SPARK-21741][ML][PySpark] Python API for DataFrame-base...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20695 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1148/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20664 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87771/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20695: [SPARK-21741][ML][PySpark] Python API for DataFrame-base...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20695 **[Test build #87778 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87778/testReport)** for PR 20695 at commit [`488d45a`](https://github.com/apache/spark/commit/488d45aa9cb9c4de96453d1f2c06f83db2b1ef77). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20664 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20695: [SPARK-21741][ML][PySpark] Python API for DataFra...
GitHub user WeichenXu123 opened a pull request: https://github.com/apache/spark/pull/20695 [SPARK-21741][ML][PySpark] Python API for DataFrame-based multivariate summarizer ## What changes were proposed in this pull request? Python API for DataFrame-based multivariate summarizer. ## How was this patch tested? doctest added. You can merge this pull request into a Git repository by running: $ git pull https://github.com/WeichenXu123/spark py_summarizer Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20695.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20695 commit 488d45aa9cb9c4de96453d1f2c06f83db2b1ef77 Author: WeichenXu Date: 2018-02-28T14:17:12Z init pr --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20664 **[Test build #87771 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87771/testReport)** for PR 20664 at commit [`0512736`](https://github.com/apache/spark/commit/051273651cd65b9eca568b37c79b50342a7f69c2). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20691: [SPARK-18161] [Python] Allow pickle to serialize >4 GB o...
Github user inpefess commented on the issue: https://github.com/apache/spark/pull/20691 Well, actually I just wanted to simply merge an older seemingly straightforward PR https://github.com/apache/spark/pull/15670 :) And @holdenk warned me that "it _should_ just be fixing the merge conflicts". So now I will fix this unit-tests failure and add a (disabled by default) test that @hvanhovell suggested. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20685: [SPARK-23524] Big local shuffle blocks should not be che...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20685 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87770/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20685: [SPARK-23524] Big local shuffle blocks should not be che...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20685 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20685: [SPARK-23524] Big local shuffle blocks should not be che...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20685 **[Test build #87770 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87770/testReport)** for PR 20685 at commit [`110c851`](https://github.com/apache/spark/commit/110c8510dcc6c2abaf4ca416b95854daf129b0a5). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20691: [SPARK-18161] [Python] Allow pickle to serialize >4 GB o...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20691 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
