[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21635 **[Test build #93293 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93293/testReport)** for PR 21635 at commit [`7958525`](https://github.com/apache/spark/commit/79585251d46b6d94e3a2d95e1e635e2801108cee). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21739: [SPARK-22187][SS] Update unsaferow format for saved stat...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/21739 LGTM again --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21813: [SPARK-24424][SQL] Support ANSI-SQL compliant syn...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21813#discussion_r203841560
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -442,17 +442,32 @@ class Analyzer(
child: LogicalPlan): LogicalPlan = {
val gid = AttributeReference(VirtualColumn.groupingIdName,
IntegerType, false)()
+ val finalGroupByExpressions = if (groupByExprs == Nil) {
--- End diff --
Shouldn't we do this in the branch of `case x: GroupingSets if
x.expressions.forall(_.resolved) =>`? I think this `constructAggregate` method
is also used by other clauses like `Cube` and `Rollup`.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21813: [SPARK-24424][SQL] Support ANSI-SQL compliant syn...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21813#discussion_r203842450
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveGroupingAnalyticsSuite.scala
---
@@ -91,6 +91,40 @@ class ResolveGroupingAnalyticsSuite extends AnalysisTest
{
assertAnalysisError(originalPlan3, Seq("doesn't show up in the GROUP
BY list"))
}
+ test("grouping sets with no explicit group by expressions") {
+val originalPlan = GroupingSets(Seq(Seq(), Seq(unresolved_a),
Seq(unresolved_a, unresolved_b)),
+ Nil, r1,
+ Seq(unresolved_a, unresolved_b,
UnresolvedAlias(count(unresolved_c
+val expected = Aggregate(Seq(a, b, gid), Seq(a, b,
count(c).as("count(c)")),
+ Expand(
+Seq(Seq(a, b, c, nulInt, nulStr, 3), Seq(a, b, c, a, nulStr, 1),
Seq(a, b, c, a, b, 0)),
+Seq(a, b, c, a, b, gid),
+Project(Seq(a, b, c, a.as("a"), b.as("b")), r1)))
+checkAnalysis(originalPlan, expected)
+
+val originalPlan2 = GroupingSets(Seq(Seq(), Seq(unresolved_a),
Seq(unresolved_a, unresolved_b)),
+ Nil, r1,
+ Seq(unresolved_a, unresolved_b,
UnresolvedAlias(count(unresolved_c
--- End diff --
hmm, I think `originalPlan2` looks the same as `originalPlan`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21584 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1138/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21635 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93293/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21635 **[Test build #93293 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93293/testReport)** for PR 21635 at commit [`7958525`](https://github.com/apache/spark/commit/79585251d46b6d94e3a2d95e1e635e2801108cee). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21584 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93294/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21635 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21584 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21584 **[Test build #93294 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93294/testReport)** for PR 21584 at commit [`d809f67`](https://github.com/apache/spark/commit/d809f67357afb955c5f398a9ada6523ed43c7f99). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21811: [SPARK-24801][CORE] Avoid memory waste by empty byte[] a...
Github user squito commented on the issue: https://github.com/apache/spark/pull/21811 Ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21811: [SPARK-24801][CORE] Avoid memory waste by empty byte[] a...
Github user squito commented on the issue: https://github.com/apache/spark/pull/21811 can you add something in the PR description about how this is important because sometimes many of these messages queue up in netty's ChannelOutboundBuffer before transferTo() is called? its discussed in the jira, but good to have here too --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21584 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21584 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1138/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21584: [SPARK-24433][K8S] Initial R Bindings for SparkR on K8s
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21584 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1138/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21811: [SPARK-24801][CORE] Avoid memory waste by empty byte[] a...
Github user squito commented on the issue: https://github.com/apache/spark/pull/21811 @zsxwing @jerryshao @Victsm you might be interested in this --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21818: [SPARK-24860][SQL] Support setting of partitionOverWrite...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21818 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21818: [SPARK-24860][SQL] Support setting of partitionOverWrite...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21818 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1139/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21818: [SPARK-24860][SQL] Support setting of partitionOverWrite...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21818 **[Test build #93295 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93295/testReport)** for PR 21818 at commit [`7d0752b`](https://github.com/apache/spark/commit/7d0752bc0f7e53542dec3bbc01a2f4e00e051f42). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21813: [SPARK-24424][SQL] Support ANSI-SQL compliant syntax for...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21813 **[Test build #93291 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93291/testReport)** for PR 21813 at commit [`ac8f04f`](https://github.com/apache/spark/commit/ac8f04fe26ad48bbd51754bf257da7e52866d87a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21813: [SPARK-24424][SQL] Support ANSI-SQL compliant syntax for...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21813 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93291/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21813: [SPARK-24424][SQL] Support ANSI-SQL compliant syntax for...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21813 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21739: [SPARK-22187][SS] Update unsaferow format for sav...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21739 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21804: [SPARK-24268][SQL] Use datatype.catalogString in error m...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21804 **[Test build #93285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93285/testReport)** for PR 21804 at commit [`20ee744`](https://github.com/apache/spark/commit/20ee744eae5f4d1291db34a4cbe17434585a5878). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21813: [SPARK-24424][SQL] Support ANSI-SQL compliant syn...
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21813#discussion_r203859940
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -442,17 +442,32 @@ class Analyzer(
child: LogicalPlan): LogicalPlan = {
val gid = AttributeReference(VirtualColumn.groupingIdName,
IntegerType, false)()
+ val finalGroupByExpressions = if (groupByExprs == Nil) {
--- End diff --
@viirya Yeah.. so for cube and rollup, we will always have groupByExprs
setup right ? So i felt its better to keep the code consolidated here in this
function. What do u think ?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21817: [SPARK-24861][SS][test] create corrected temp directorie...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21817 **[Test build #93289 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93289/testReport)** for PR 21817 at commit [`10c5ecf`](https://github.com/apache/spark/commit/10c5ecf7443866c822d9af60f7aca7306e582d7e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21804: [SPARK-24268][SQL] Use datatype.catalogString in error m...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21804 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21804: [SPARK-24268][SQL] Use datatype.catalogString in error m...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21804 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93285/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21817: [SPARK-24861][SS][test] create corrected temp directorie...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21817 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93289/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21817: [SPARK-24861][SS][test] create corrected temp directorie...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21817 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21819: [SPARK-24863][SS] Report Kafka offset lag as a cu...
GitHub user arunmahadevan opened a pull request: https://github.com/apache/spark/pull/21819 [SPARK-24863][SS] Report Kafka offset lag as a custom metrics ## What changes were proposed in this pull request? This builds on top of SPARK-24748 to report 'offset lag' as a custom metrics for Kafka structured streaming source. This lag is the difference between the latest offsets in Kafka the time the metrics is reported (just after a micro-batch completes) and the latest offset Spark has processed. It can be 0 (or close to 0) if spark keeps up with the rate at which messages are ingested into Kafka topics in steady state. This measures how far behind the spark source has fallen behind (per partition) and can aid in tuning the application. ## How was this patch tested? Existing and new unit tests Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/arunmahadevan/spark SPARK-24863 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21819.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21819 commit 29919fe07191cf75f5a7651f8ac9434dc79c119d Author: Arun Mahadevan Date: 2018-07-06T01:51:50Z [SPARK-24748][SS] Support for reporting custom metrics via Streaming Query Progress commit 43190e9112c3d87e482d81ac8c56097c5c513012 Author: Arun Mahadevan Date: 2018-07-06T18:07:28Z Add error reporting API for custom metrics and address review comments commit 6d4165efc9c49f73141292b6c0f318f6a3cafb23 Author: Arun Mahadevan Date: 2018-07-11T17:42:17Z Added support for custom metrics in Sink and use MemorySinkV2 as an example commit bca054f978406b257bfa4c4010e7655144fc820f Author: Arun Mahadevan Date: 2018-07-11T17:59:54Z remove kafka source metrics outside the scope of this PR commit 5e732cba85a5c2e3ed3f0487c70c1ebe4c20b75d Author: Arun Mahadevan Date: 2018-07-11T18:48:41Z Fix scala style issues Change-Id: I831719f1e9ef1437d9df2b3529bf0a288ef5d0fa commit c1fc3ca1ec2e2698d1d83ca2bd3ecbecd4da76a6 Author: Arun Mahadevan Date: 2018-07-19T20:14:40Z [SPARK-24863][SS] Report Kafka offset lag as a custom metrics --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21635 **[Test build #93297 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93297/testReport)** for PR 21635 at commit [`6761098`](https://github.com/apache/spark/commit/6761098499f786e981f67520a10bc3a9dd890561). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21819: [SPARK-24863][SS] Report Kafka offset lag as a custom me...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21819 **[Test build #93296 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93296/testReport)** for PR 21819 at commit [`c1fc3ca`](https://github.com/apache/spark/commit/c1fc3ca1ec2e2698d1d83ca2bd3ecbecd4da76a6). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21813: [SPARK-24424][SQL] Support ANSI-SQL compliant syn...
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21813#discussion_r203861013
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveGroupingAnalyticsSuite.scala
---
@@ -91,6 +91,40 @@ class ResolveGroupingAnalyticsSuite extends AnalysisTest
{
assertAnalysisError(originalPlan3, Seq("doesn't show up in the GROUP
BY list"))
}
+ test("grouping sets with no explicit group by expressions") {
+val originalPlan = GroupingSets(Seq(Seq(), Seq(unresolved_a),
Seq(unresolved_a, unresolved_b)),
+ Nil, r1,
+ Seq(unresolved_a, unresolved_b,
UnresolvedAlias(count(unresolved_c
+val expected = Aggregate(Seq(a, b, gid), Seq(a, b,
count(c).as("count(c)")),
+ Expand(
+Seq(Seq(a, b, c, nulInt, nulStr, 3), Seq(a, b, c, a, nulStr, 1),
Seq(a, b, c, a, b, 0)),
+Seq(a, b, c, a, b, gid),
+Project(Seq(a, b, c, a.as("a"), b.as("b")), r1)))
+checkAnalysis(originalPlan, expected)
+
+val originalPlan2 = GroupingSets(Seq(Seq(), Seq(unresolved_a),
Seq(unresolved_a, unresolved_b)),
+ Nil, r1,
+ Seq(unresolved_a, unresolved_b,
UnresolvedAlias(count(unresolved_c
--- End diff --
@viirya Thanks.. u right. I will remove it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21819: [SPARK-24863][SS] Report Kafka offset lag as a custom me...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21819 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21652: [SPARK-24551][K8S] Add integration tests for secrets
Github user skonto commented on the issue: https://github.com/apache/spark/pull/21652 @foxish @srowen I rebased again could you review and merge pls. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21819: [SPARK-24863][SS] Report Kafka offset lag as a custom me...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21819 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21817: [SPARK-24861][SS][test] create corrected temp directorie...
Github user jose-torres commented on the issue: https://github.com/apache/spark/pull/21817 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21652: [SPARK-24551][K8S] Add integration tests for secrets
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21652 **[Test build #93298 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93298/testReport)** for PR 21652 at commit [`67df340`](https://github.com/apache/spark/commit/67df340d943d38afd1ea4c12c02b417b5434970f). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21819: [SPARK-24863][SS] Report Kafka offset lag as a custom me...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21819 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93296/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21819: [SPARK-24863][SS] Report Kafka offset lag as a custom me...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21819 **[Test build #93296 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93296/testReport)** for PR 21819 at commit [`c1fc3ca`](https://github.com/apache/spark/commit/c1fc3ca1ec2e2698d1d83ca2bd3ecbecd4da76a6). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21819: [SPARK-24863][SS] Report Kafka offset lag as a custom me...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21819 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21652: [SPARK-24551][K8S] Add integration tests for secrets
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21652 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1140/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21774: [SPARK-24811][SQL]Avro: add new function from_avro and t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21774 **[Test build #93288 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93288/testReport)** for PR 21774 at commit [`833dccc`](https://github.com/apache/spark/commit/833d99d3f411e30235f5c0b7df529918996f). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class CatalystDataToAvro(child: Expression) extends UnaryExpression ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21774: [SPARK-24811][SQL]Avro: add new function from_avro and t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21774 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93288/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21774: [SPARK-24811][SQL]Avro: add new function from_avro and t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21774 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21635 **[Test build #93297 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93297/testReport)** for PR 21635 at commit [`6761098`](https://github.com/apache/spark/commit/6761098499f786e981f67520a10bc3a9dd890561). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21635 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21635: [SPARK-24594][YARN] Introducing metrics for YARN
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21635 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93297/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21652: [SPARK-24551][K8S] Add integration tests for secrets
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21652 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1140/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21652: [SPARK-24551][K8S] Add integration tests for secrets
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21652 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/1140/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21652: [SPARK-24551][K8S] Add integration tests for secrets
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21652 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21813: [SPARK-24424][SQL] Support ANSI-SQL compliant syntax for...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21813 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21813: [SPARK-24424][SQL] Support ANSI-SQL compliant syntax for...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21813 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1141/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21813: [SPARK-24424][SQL] Support ANSI-SQL compliant syntax for...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21813 **[Test build #93299 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93299/testReport)** for PR 21813 at commit [`7cf187d`](https://github.com/apache/spark/commit/7cf187db02a54bcfd3b44e0710d95462b273ea97). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21067: [SPARK-23980][K8S] Resilient Spark driver on Kubernetes
Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/21067 +1 on what @foxish said. If using a Job is the right way to go ultimately, it's good to open discussion with sig-apps on adding an option to the Job API & controller to use deterministic pod names as well as to offer the exactly-once semantic. Spark probably is not the only use case needing such a semantic guarantee. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21652: [SPARK-24551][K8S] Add integration tests for secr...
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/21652#discussion_r203871861 --- Diff: bin/docker-image-tool.sh --- @@ -135,7 +135,7 @@ BASEDOCKERFILE= PYDOCKERFILE= NOCACHEARG= BUILD_PARAMS= -while getopts f:mr:t:n:b: option +while getopts f:p:mr:t:n:b: option --- End diff -- @debasishg spotted this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21806: [SPARK-24846][SQL] Made hashCode ExprId independent of j...
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/21806 Merging to master. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21806: [SPARK-24846][SQL] Made hashCode ExprId independe...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21806 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21813: [SPARK-24424][SQL] Support ANSI-SQL compliant syn...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21813#discussion_r203882021
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -442,17 +442,32 @@ class Analyzer(
child: LogicalPlan): LogicalPlan = {
val gid = AttributeReference(VirtualColumn.groupingIdName,
IntegerType, false)()
+ val finalGroupByExpressions = if (groupByExprs == Nil) {
--- End diff --
Ok. Mind to add a comment on this like `SPARK-24424: this only happens for
ANSI-SQL compliant syntax for GROUPING SET`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21813: [SPARK-24424][SQL] Support ANSI-SQL compliant syn...
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/21813#discussion_r203882365
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -442,17 +442,32 @@ class Analyzer(
child: LogicalPlan): LogicalPlan = {
val gid = AttributeReference(VirtualColumn.groupingIdName,
IntegerType, false)()
+ val finalGroupByExpressions = if (groupByExprs == Nil) {
--- End diff --
@viirya Sure will do.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21818: [SPARK-24860][SQL] Support setting of partitionOverWrite...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21818 **[Test build #93290 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93290/testReport)** for PR 21818 at commit [`7dd2eab`](https://github.com/apache/spark/commit/7dd2eabfd4f5ca18354df85f5bf5285e3e23359d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21818: [SPARK-24860][SQL] Support setting of partitionOverWrite...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21818 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21818: [SPARK-24860][SQL] Support setting of partitionOverWrite...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21818 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93290/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21732: [SPARK-24762][SQL] Aggregator should be able to u...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21732#discussion_r203891595
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala ---
@@ -183,6 +184,43 @@ case class OptionBooleanAggregator(colName: String)
def OptionalBoolEncoder: Encoder[Option[Boolean]] = ExpressionEncoder()
}
+case class OptionBooleanIntAggregator(colName: String)
+extends Aggregator[Row, Option[(Boolean, Int)], Option[(Boolean,
Int)]] {
--- End diff --
For a non top-level encoder, the output schema of `Option[Product]` should
be struct column.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21746: [SPARK-24699] [SS]Make watermarks work with Trigger.Once...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21746 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21746: [SPARK-24699] [SS]Make watermarks work with Trigger.Once...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21746 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1142/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21746: [SPARK-24699] [SS]Make watermarks work with Trigger.Once...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21746 **[Test build #93300 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93300/testReport)** for PR 21746 at commit [`584c96e`](https://github.com/apache/spark/commit/584c96e5e8ecbeeb4ae4eefe1184606173b83ade). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21746: [SPARK-24699] [SS]Make watermarks work with Trigger.Once...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21746 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1143/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21746: [SPARK-24699] [SS]Make watermarks work with Trigger.Once...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21746 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21746: [SPARK-24699] [SS]Make watermarks work with Trigger.Once...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21746 **[Test build #93301 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93301/testReport)** for PR 21746 at commit [`10b5c2f`](https://github.com/apache/spark/commit/10b5c2fcf578a8fdd565a860f7d976454b57e1fb). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21774: [SPARK-24811][SQL]Avro: add new function from_avro and t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21774 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21820: [SPARK-24868][PYTHON]add sequence function in Python
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21820 **[Test build #93303 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93303/testReport)** for PR 21820 at commit [`a7c434c`](https://github.com/apache/spark/commit/a7c434c49248703e182ae9904d344246b13f7b87). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21813: [SPARK-24424][SQL] Support ANSI-SQL compliant syntax for...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21813 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1145/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21821: [SPARK-24867] [SQL] Add AnalysisBarrier to DataFr...
GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/21821
[SPARK-24867] [SQL] Add AnalysisBarrier to DataFrameWriter
## What changes were proposed in this pull request?
```Scala
val udf1 = udf({(x: Int, y: Int) => x + y})
val df = spark.range(0, 3).toDF("a")
.withColumn("b", udf1($"a", udf1($"a", lit(10
df.cache()
df.write.saveAsTable("t")
```
Cache is not being used because the plans do not match with the cached
plan. This is a regression caused by the changes we made in AnalysisBarrier,
since not all the Analyzer rules are idempotent.
## How was this patch tested?
Added a test.
Also found a bug in the DSV1 write path. This is not a regression. Thus,
opened a separate JIRA https://issues.apache.org/jira/browse/SPARK-24869
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark testMaster22
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21821.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21821
commit 23ec09fc3bbedd2f34c594daf461cebd9c0295a6
Author: Xiao Li
Date: 2018-07-19T23:38:44Z
fix
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21821: [SPARK-24867] [SQL] Add AnalysisBarrier to DataFrameWrit...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21821 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21821: [SPARK-24867] [SQL] Add AnalysisBarrier to DataFr...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21821#discussion_r203905148
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
@@ -254,7 +254,7 @@ final class DataFrameWriter[T] private[sql](ds:
Dataset[T]) {
val writer = ws.createWriter(jobId, df.logicalPlan.schema, mode,
options)
if (writer.isPresent) {
runCommand(df.sparkSession, "save") {
- WriteToDataSourceV2(writer.get(), df.logicalPlan)
+ WriteToDataSourceV2(writer.get(), df.planWithBarrier)
--- End diff --
This change is not needed but it is safe to have.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21821: [SPARK-24867] [SQL] Add AnalysisBarrier to DataFrameWrit...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21821 **[Test build #93305 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93305/testReport)** for PR 21821 at commit [`23ec09f`](https://github.com/apache/spark/commit/23ec09fc3bbedd2f34c594daf461cebd9c0295a6). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21821: [SPARK-24867] [SQL] Add AnalysisBarrier to DataFrameWrit...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21821 **[Test build #93305 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93305/testReport)** for PR 21821 at commit [`23ec09f`](https://github.com/apache/spark/commit/23ec09fc3bbedd2f34c594daf461cebd9c0295a6). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21818: [SPARK-24860][SQL] Support setting of partitionOverWrite...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21818 **[Test build #93295 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93295/testReport)** for PR 21818 at commit [`7d0752b`](https://github.com/apache/spark/commit/7d0752bc0f7e53542dec3bbc01a2f4e00e051f42). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21818: [SPARK-24860][SQL] Support setting of partitionOverWrite...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21818 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21818: [SPARK-24860][SQL] Support setting of partitionOverWrite...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21818 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93295/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21820: [SPARK-24868][PYTHON]add sequence function in Python
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21820 **[Test build #93303 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93303/testReport)** for PR 21820 at commit [`a7c434c`](https://github.com/apache/spark/commit/a7c434c49248703e182ae9904d344246b13f7b87). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21822 **[Test build #93306 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93306/testReport)** for PR 21822 at commit [`f6f2bcc`](https://github.com/apache/spark/commit/f6f2bccd6d887b50b035c492c88e9e76d1ef4754). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21820: [SPARK-24868][PYTHON]add sequence function in Python
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21820 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93303/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21820: [SPARK-24868][PYTHON]add sequence function in Python
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21820 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21822 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1147/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21822 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21822 **[Test build #93307 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93307/testReport)** for PR 21822 at commit [`8ccafca`](https://github.com/apache/spark/commit/8ccafcab20a70df7a625912fdf4e43be7fb87954). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21822 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1148/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21822 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21822 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93306/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21653: [SPARK-13343] speculative tasks that didn't commit shoul...
Github user hthuynh2 commented on the issue: https://github.com/apache/spark/pull/21653 @tgravescs I updated it. Can you please have a look at it when you have time. Thank you. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21822 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93307/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21546: [SPARK-23030][SQL][PYTHON] Use Arrow stream forma...
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/21546#discussion_r203913178
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -3349,20 +3385,20 @@ class Dataset[T] private[sql](
}
}
- /** Convert to an RDD of ArrowPayload byte arrays */
- private[sql] def toArrowPayload(plan: SparkPlan): RDD[ArrowPayload] = {
+ /** Convert to an RDD of serialized ArrowRecordBatches. */
+ private[sql] def getArrowBatchRdd(plan: SparkPlan): RDD[Array[Byte]] = {
--- End diff --
Yeah, I can't remember why I changed it.. but I think you're right it so
I'll change it back.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21822: [SPARK-24865] Remove AnalysisBarrier - WIP
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21822 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93308/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21546: [SPARK-23030][SQL][PYTHON] Use Arrow stream format for c...
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/21546 >@BryanCutler, this takes longer then I thought. Will complete my review till this weekend. For clarification, still no objection about merging it in orthogonally with my review. No problem, thanks @HyukjinKwon ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21440#discussion_r203914040
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -659,6 +659,11 @@ private[spark] class BlockManager(
* Get block from remote block managers as serialized bytes.
*/
def getRemoteBytes(blockId: BlockId): Option[ChunkedByteBuffer] = {
+// TODO if we change this method to return the ManagedBuffer, then
getRemoteValues
+// could just use the inputStream on the temp file, rather than
memory-mapping the file.
+// Until then, replication can cause the process to use too much
memory and get killed
+// by the OS / cluster manager (not a java OOM, since its a
memory-mapped file) even though
+// we've read the data to disk.
--- End diff --
I see. I agree with you that YARN could have some issues in calculating the
exact memory usage.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21652: [SPARK-24551][K8S] Add integration tests for secrets
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21652 **[Test build #93298 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93298/testReport)** for PR 21652 at commit [`67df340`](https://github.com/apache/spark/commit/67df340d943d38afd1ea4c12c02b417b5434970f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
