[GitHub] [spark] AmplabJenkins removed a comment on issue #27252: [SPARK-29231][SQL] Constraints should be inferred from cast equality constraint
AmplabJenkins removed a comment on issue #27252: [SPARK-29231][SQL] Constraints should be inferred from cast equality constraint URL: https://github.com/apache/spark/pull/27252#issuecomment-576164934 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21868/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27252: [SPARK-29231][SQL] Constraints should be inferred from cast equality constraint
AmplabJenkins removed a comment on issue #27252: [SPARK-29231][SQL] Constraints should be inferred from cast equality constraint URL: https://github.com/apache/spark/pull/27252#issuecomment-576164923 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27263: [SPARK-30433][SQL][FOLLOW-UP] Optimize collect conflict plans
AmplabJenkins removed a comment on issue #27263: [SPARK-30433][SQL][FOLLOW-UP] Optimize collect conflict plans URL: https://github.com/apache/spark/pull/27263#issuecomment-576164870 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21867/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27243: [SPARK-30535][SQL] Migrate ALTER TABLE commands to the new framework
AmplabJenkins removed a comment on issue #27243: [SPARK-30535][SQL] Migrate ALTER TABLE commands to the new framework URL: https://github.com/apache/spark/pull/27243#issuecomment-576164945 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27263: [SPARK-30433][SQL][FOLLOW-UP] Optimize collect conflict plans
AmplabJenkins removed a comment on issue #27263: [SPARK-30433][SQL][FOLLOW-UP] Optimize collect conflict plans URL: https://github.com/apache/spark/pull/27263#issuecomment-576164858 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26805: [SPARK-15616][SQL] Add optimizer rule PruneHiveTablePartitions
cloud-fan commented on a change in pull request #26805: [SPARK-15616][SQL] Add
optimizer rule PruneHiveTablePartitions
URL: https://github.com/apache/spark/pull/26805#discussion_r368419917
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/PruneHiveTablePartitions.scala
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.execution
+
+import org.apache.hadoop.hive.common.StatsSetupConst
+
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.CastSupport
+import org.apache.spark.sql.catalyst.catalog.{CatalogStatistics, CatalogTable,
CatalogTablePartition, ExternalCatalogUtils, HiveTableRelation}
+import org.apache.spark.sql.catalyst.expressions.{And, AttributeSet,
Expression, ExpressionSet, SubqueryExpression}
+import org.apache.spark.sql.catalyst.planning.PhysicalOperation
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan,
Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.datasources.DataSourceStrategy
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * TODO: merge this with PruneFileSourcePartitions after we completely make
hive as a data source.
+ */
+private[sql] class PruneHiveTablePartitions(session: SparkSession)
+ extends Rule[LogicalPlan] with CastSupport {
+
+ override val conf: SQLConf = session.sessionState.conf
+
+ /**
+ * Extract the partition filters from the filters on the table.
+ */
+ private def getPartitionKeyFilters(
+ filters: Seq[Expression],
+ relation: HiveTableRelation): ExpressionSet = {
+val normalizedFilters = DataSourceStrategy.normalizeExprs(
+ filters.filter(f => f.deterministic &&
!SubqueryExpression.hasSubquery(f)), relation.output)
+val partitionColumnSet = AttributeSet(relation.partitionCols)
+ExpressionSet(normalizedFilters.filter { f =>
+ !f.references.isEmpty && f.references.subsetOf(partitionColumnSet)
+})
+ }
+
+ /**
+ * Prune the hive table using filters on the partitions of the table.
+ */
+ private def prunePartitions(
+ relation: HiveTableRelation,
+ partitionFilters: ExpressionSet): Seq[CatalogTablePartition] = {
+if (conf.metastorePartitionPruning) {
+ session.sessionState.catalog.listPartitionsByFilter(
+relation.tableMeta.identifier, partitionFilters.toSeq)
+} else {
+ ExternalCatalogUtils.prunePartitionsByFilter(relation.tableMeta,
+
session.sessionState.catalog.listPartitions(relation.tableMeta.identifier),
+partitionFilters.toSeq, conf.sessionLocalTimeZone)
+}
+ }
+
+ /**
+ * Update the statistics of the table.
+ */
+ private def updateTableMeta(
+ tableMeta: CatalogTable,
+ prunedPartitions: Seq[CatalogTablePartition]): CatalogTable = {
+val sizeOfPartitions = prunedPartitions.map { partition =>
+ val rawDataSize =
partition.parameters.get(StatsSetupConst.RAW_DATA_SIZE).map(_.toLong)
+ val totalSize =
partition.parameters.get(StatsSetupConst.TOTAL_SIZE).map(_.toLong)
+ if (rawDataSize.isDefined && rawDataSize.get > 0) {
+rawDataSize.get
+ } else if (totalSize.isDefined && totalSize.get > 0L) {
+totalSize.get
+ } else {
+0L
+ }
+}
+if (sizeOfPartitions.forall(s => s>0)) {
+ val sizeInBytes = sizeOfPartitions.sum
+ tableMeta.copy(stats = Some(CatalogStatistics(sizeInBytes =
BigInt(sizeInBytes
+} else {
+ tableMeta
+}
+ }
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+case op @ PhysicalOperation(projections, filters, relation:
HiveTableRelation)
+ if filters.nonEmpty && relation.isPartitioned &&
relation.prunedPartitions.isEmpty =>
+ val partitionKeyFilters = getPartitionKeyFilters(filters, relation)
+ if (partitionKeyFilters.nonEmpty) {
+val newPartitions = prunePartitions(relation, partitionKeyFilters)
+val newTableMeta = updateTableMeta(relation.tableMeta, newPartitions)
+val newRelation = relation.copy(
+ tableMeta = newTableMeta, prunedPartitions = Some(newPartitions))
+
[GitHub] [spark] zuston commented on a change in pull request #27257: [SPARK-30540][Web UI] HistoryServer application link is incorrect when one application having multiple attempts
zuston commented on a change in pull request #27257: [SPARK-30540][Web UI]
HistoryServer application link is incorrect when one application having
multiple attempts
URL: https://github.com/apache/spark/pull/27257#discussion_r368419626
##
File path: core/src/main/resources/org/apache/spark/ui/static/historypage.js
##
@@ -140,7 +140,8 @@ $(document).ready(function() {
(attempt.hasOwnProperty("attemptId") ? attempt["attemptId"] + "/"
: "") + "logs";
attempt["durationMillisec"] = attempt["duration"];
attempt["duration"] = formatDuration(attempt["duration"]);
- var app_clone = {"id" : id, "name" : name, "version": version, "num"
: num, "attempts" : [attempt]};
+ var app_clone = {"id" : id, "name" : name, "version": version, "num"
: num, "attempts" : [attempt],
Review comment:
Ok, i will remove it. Thanks~
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zuston commented on a change in pull request #27257: [SPARK-30540][Web UI] HistoryServer application link is incorrect when one application having multiple attempts
zuston commented on a change in pull request #27257: [SPARK-30540][Web UI]
HistoryServer application link is incorrect when one application having
multiple attempts
URL: https://github.com/apache/spark/pull/27257#discussion_r368419626
##
File path: core/src/main/resources/org/apache/spark/ui/static/historypage.js
##
@@ -140,7 +140,8 @@ $(document).ready(function() {
(attempt.hasOwnProperty("attemptId") ? attempt["attemptId"] + "/"
: "") + "logs";
attempt["durationMillisec"] = attempt["duration"];
attempt["duration"] = formatDuration(attempt["duration"]);
- var app_clone = {"id" : id, "name" : name, "version": version, "num"
: num, "attempts" : [attempt]};
+ var app_clone = {"id" : id, "name" : name, "version": version, "num"
: num, "attempts" : [attempt],
Review comment:
Ok, i will remove it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27243: [SPARK-30535][SQL] Migrate ALTER TABLE commands to the new framework
AmplabJenkins removed a comment on issue #27243: [SPARK-30535][SQL] Migrate ALTER TABLE commands to the new framework URL: https://github.com/apache/spark/pull/27243#issuecomment-576164956 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21869/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zuston commented on a change in pull request #27257: [SPARK-30540][Web UI] HistoryServer application link is incorrect when one application having multiple attempts
zuston commented on a change in pull request #27257: [SPARK-30540][Web UI]
HistoryServer application link is incorrect when one application having
multiple attempts
URL: https://github.com/apache/spark/pull/27257#discussion_r368419303
##
File path: core/src/main/resources/org/apache/spark/ui/static/historypage.js
##
@@ -140,7 +140,8 @@ $(document).ready(function() {
(attempt.hasOwnProperty("attemptId") ? attempt["attemptId"] + "/"
: "") + "logs";
Review comment:
The `attemptId` is not checked with line of `{{attemptId}}` in
`historypage-template.html`. And need to ignore this link when `attempId` is
empty or null. Do you have some ideas?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27260: [SPARK-30549][SQL] Fix the subquery shown issue in UI When enable AQE
cloud-fan commented on a change in pull request #27260: [SPARK-30549][SQL] Fix
the subquery shown issue in UI When enable AQE
URL: https://github.com/apache/spark/pull/27260#discussion_r368424055
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala
##
@@ -344,6 +344,18 @@ class SQLAppStatusListener(
update(exec)
}
+ private def onAdaptiveAccumUpdates(event:
SparkListenerSQLAdaptiveAccumUpdates): Unit = {
+val SparkListenerSQLAdaptiveAccumUpdates(executionId,
accumIdsToMetricType) = event
+
+val stages = liveExecutions.get(executionId).stages
+accumIdsToMetricType.map { case (accumulatorId, metricType) =>
Review comment:
why do we need to loop it? we can just update
`liveStageMetric.accumIdsToMetricType` as
```
liveStageMetric.copy(accumIdsToMetricType =
liveStageMetric.accumIdsToMetricType ++ accumIdsToMetricType)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27260: [SPARK-30549][SQL] Fix the subquery shown issue in UI When enable AQE
cloud-fan commented on issue #27260: [SPARK-30549][SQL] Fix the subquery shown issue in UI When enable AQE URL: https://github.com/apache/spark/pull/27260#issuecomment-576169530 @JkSelf can you try it locally and post some screenshots in the PR description? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27260: [SPARK-30549][SQL] Fix the subquery shown issue in UI When enable AQE
cloud-fan commented on a change in pull request #27260: [SPARK-30549][SQL] Fix
the subquery shown issue in UI When enable AQE
URL: https://github.com/apache/spark/pull/27260#discussion_r368424177
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala
##
@@ -344,6 +344,18 @@ class SQLAppStatusListener(
update(exec)
}
+ private def onAdaptiveAccumUpdates(event:
SparkListenerSQLAdaptiveAccumUpdates): Unit = {
+val SparkListenerSQLAdaptiveAccumUpdates(executionId,
accumIdsToMetricType) = event
+
+val stages = liveExecutions.get(executionId).stages
+accumIdsToMetricType.map { case (accumulatorId, metricType) =>
+ stages.foreach { stageId =>
+val liveStageMetric = stageMetrics.get(stageId)
+liveStageMetric.accumIdsToMetricType += (accumulatorId -> metricType)
Review comment:
It's too hacky to make a UI data class mutable.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] manuzhang commented on a change in pull request #27253: [SPARK-30524] [SQL] follow up SPARK-30524 to resolve comments
manuzhang commented on a change in pull request #27253: [SPARK-30524] [SQL]
follow up SPARK-30524 to resolve comments
URL: https://github.com/apache/spark/pull/27253#discussion_r368424224
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala
##
@@ -247,9 +225,9 @@ case class OptimizeSkewedJoin(conf: SQLConf) extends
Rule[SparkPlan] {
if (shuffleStages.length == 2) {
// When multi table join, there will be too many complex combination to
consider.
// Currently we only handle 2 table join like following two use cases.
Review comment:
minor: only one case now
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27288: [SPARK-30579][DOC] Document ORDER BY Clause of SELECT statement in SQL Reference
AmplabJenkins removed a comment on issue #27288: [SPARK-30579][DOC] Document ORDER BY Clause of SELECT statement in SQL Reference URL: https://github.com/apache/spark/pull/27288#issuecomment-576168580 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117101/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference.
AmplabJenkins removed a comment on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference. URL: https://github.com/apache/spark/pull/27289#issuecomment-576186157 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21872/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference.
AmplabJenkins commented on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference. URL: https://github.com/apache/spark/pull/27289#issuecomment-576186157 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21872/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27249: [SPARK-30019][SQL] Add the owner property to v2 table
AmplabJenkins commented on issue #27249: [SPARK-30019][SQL] Add the owner property to v2 table URL: https://github.com/apache/spark/pull/27249#issuecomment-576186229 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21873/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference.
AmplabJenkins removed a comment on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference. URL: https://github.com/apache/spark/pull/27289#issuecomment-576186146 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference.
AmplabJenkins commented on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference. URL: https://github.com/apache/spark/pull/27289#issuecomment-576186146 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27249: [SPARK-30019][SQL] Add the owner property to v2 table
AmplabJenkins removed a comment on issue #27249: [SPARK-30019][SQL] Add the owner property to v2 table URL: https://github.com/apache/spark/pull/27249#issuecomment-576186229 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21873/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27249: [SPARK-30019][SQL] Add the owner property to v2 table
AmplabJenkins removed a comment on issue #27249: [SPARK-30019][SQL] Add the owner property to v2 table URL: https://github.com/apache/spark/pull/27249#issuecomment-576186221 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27249: [SPARK-30019][SQL] Add the owner property to v2 table
AmplabJenkins commented on issue #27249: [SPARK-30019][SQL] Add the owner property to v2 table URL: https://github.com/apache/spark/pull/27249#issuecomment-576186221 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference.
AmplabJenkins removed a comment on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference. URL: https://github.com/apache/spark/pull/27289#issuecomment-576190016 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference.
SparkQA commented on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference. URL: https://github.com/apache/spark/pull/27289#issuecomment-576189891 **[Test build #117108 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117108/testReport)** for PR 27289 at commit [`0f37760`](https://github.com/apache/spark/commit/0f3776006a5296f72cf9f5b20e8b92c850732095). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference.
AmplabJenkins removed a comment on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference. URL: https://github.com/apache/spark/pull/27289#issuecomment-576190027 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117108/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27292: [SPARK-30582][Web UI] Spark UI is not showing Aggregated Metrics by Executor in stage page
AmplabJenkins commented on issue #27292: [SPARK-30582][Web UI] Spark UI is not showing Aggregated Metrics by Executor in stage page URL: https://github.com/apache/spark/pull/27292#issuecomment-576198553 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27258: [SPARK-30547][SQL] Add unstable annotation to the CalendarInterval class
yaooqinn commented on a change in pull request #27258: [SPARK-30547][SQL] Add unstable annotation to the CalendarInterval class URL: https://github.com/apache/spark/pull/27258#discussion_r368502206 ## File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/CalendarInterval.java ## @@ -27,8 +29,21 @@ import static org.apache.spark.sql.catalyst.util.DateTimeConstants.*; /** - * The internal representation of interval type. + * The class representing calendar intervals. The calendar interval is stored internally in + * three components: + * + * an integer value representing the number of `months` in this interval, + * an integer value representing the number of `days` in this interval, + * a long value representing the number of `microseconds` in this interval. + * + * + * The `months` and `days` are not units of time with a constant length (unlike hours, seconds), so + * they are two separated fields from microseconds. One month may be equal to 28, 29, 30 or 31 days + * and one day may be equal to 23, 24 or 25 hours (daylight saving). + * + * @since 1.5.0 Review comment: checked the java doc about since https://www.oracle.com/technetwork/java/javase/documentation/index-137868.html#@since >When a class (or interface) is introduced, specify one since tag in its class description and no since tags in the members. Add a since tag only to members added in a later version than the class. ... > If a member changes from protected to public in a later release, the since tag would not change, even though it is now usable by any caller, not just subclassers. `1.5.0` should be fine I guess This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27277: [SPARK-30568][SQL] Invalidate interval type as a field table schema
AmplabJenkins commented on issue #27277: [SPARK-30568][SQL] Invalidate interval type as a field table schema URL: https://github.com/apache/spark/pull/27277#issuecomment-576239626 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27277: [SPARK-30568][SQL] Invalidate interval type as a field table schema
AmplabJenkins commented on issue #27277: [SPARK-30568][SQL] Invalidate interval type as a field table schema URL: https://github.com/apache/spark/pull/27277#issuecomment-576239635 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21878/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] bettermouse commented on issue #27268: [SPARK-30553][DOCS] fix structured-streaming java example error
bettermouse commented on issue #27268: [SPARK-30553][DOCS] fix structured-streaming java example error URL: https://github.com/apache/spark/pull/27268#issuecomment-576239481 @HeartSaVioR Sorry.This is my first PR.Can you help me understand what `plan information` is. I think we just need update the documentation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27002: [SPARK-30346][CORE]Improve logging when events dropped
AmplabJenkins removed a comment on issue #27002: [SPARK-30346][CORE]Improve logging when events dropped URL: https://github.com/apache/spark/pull/27002#issuecomment-576238007 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117111/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #27268: [SPARK-30553][DOCS] fix structured-streaming java example error
HeartSaVioR commented on issue #27268: [SPARK-30553][DOCS] fix structured-streaming java example error URL: https://github.com/apache/spark/pull/27268#issuecomment-576250811 I assume you are now having two example queries "before the fix" vs "after the fix". Once you run the query and ingest some data, you'll see the query plan in SQL tab, as you've added like `HashAggregate(keys=[window#11-T1ms, value#39]`. It would be enough to fill up the content and add below: ``` > Before the fix \``` \``` > After the fix \``` \``` ``` Please remove `\` above when pasting to the content of PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27237: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression
AmplabJenkins commented on issue #27237: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression URL: https://github.com/apache/spark/pull/27237#issuecomment-576158504 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21860/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27249: [SPARK-30019][SQL] Add ALTER TABLE SET OWNER syntax
cloud-fan commented on a change in pull request #27249: [SPARK-30019][SQL] Add
ALTER TABLE SET OWNER syntax
URL: https://github.com/apache/spark/pull/27249#discussion_r368417560
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##
@@ -54,6 +54,7 @@ import
org.apache.spark.sql.catalyst.catalog.CatalogTypes.TablePartitionSpec
import org.apache.spark.sql.catalyst.expressions.Expression
import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
import org.apache.spark.sql.connector.catalog.SupportsNamespaces._
+import org.apache.spark.sql.connector.catalog.TableCatalog
Review comment:
ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27249: [SPARK-30019][SQL] Add ALTER TABLE SET OWNER syntax
cloud-fan commented on a change in pull request #27249: [SPARK-30019][SQL] Add ALTER TABLE SET OWNER syntax URL: https://github.com/apache/spark/pull/27249#discussion_r368417020 ## File path: sql/core/src/test/scala/org/apache/spark/sql/ShowCreateTableSuite.scala ## @@ -19,6 +19,7 @@ package org.apache.spark.sql import org.apache.spark.sql.catalyst.TableIdentifier import org.apache.spark.sql.catalyst.catalog.CatalogTable +import org.apache.spark.sql.connector.catalog.TableCatalog Review comment: unnecessary This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] fuwhu commented on a change in pull request #26805: [SPARK-15616][SQL] Add optimizer rule PruneHiveTablePartitions
fuwhu commented on a change in pull request #26805: [SPARK-15616][SQL] Add
optimizer rule PruneHiveTablePartitions
URL: https://github.com/apache/spark/pull/26805#discussion_r368422320
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/PruneHiveTablePartitions.scala
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.execution
+
+import org.apache.hadoop.hive.common.StatsSetupConst
+
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.CastSupport
+import org.apache.spark.sql.catalyst.catalog.{CatalogStatistics, CatalogTable,
CatalogTablePartition, ExternalCatalogUtils, HiveTableRelation}
+import org.apache.spark.sql.catalyst.expressions.{And, AttributeSet,
Expression, ExpressionSet, SubqueryExpression}
+import org.apache.spark.sql.catalyst.planning.PhysicalOperation
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan,
Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.datasources.DataSourceStrategy
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * TODO: merge this with PruneFileSourcePartitions after we completely make
hive as a data source.
+ */
+private[sql] class PruneHiveTablePartitions(session: SparkSession)
+ extends Rule[LogicalPlan] with CastSupport {
+
+ override val conf: SQLConf = session.sessionState.conf
+
+ /**
+ * Extract the partition filters from the filters on the table.
+ */
+ private def getPartitionKeyFilters(
+ filters: Seq[Expression],
+ relation: HiveTableRelation): ExpressionSet = {
+val normalizedFilters = DataSourceStrategy.normalizeExprs(
+ filters.filter(f => f.deterministic &&
!SubqueryExpression.hasSubquery(f)), relation.output)
+val partitionColumnSet = AttributeSet(relation.partitionCols)
+ExpressionSet(normalizedFilters.filter { f =>
+ !f.references.isEmpty && f.references.subsetOf(partitionColumnSet)
+})
+ }
+
+ /**
+ * Prune the hive table using filters on the partitions of the table.
+ */
+ private def prunePartitions(
+ relation: HiveTableRelation,
+ partitionFilters: ExpressionSet): Seq[CatalogTablePartition] = {
+if (conf.metastorePartitionPruning) {
+ session.sessionState.catalog.listPartitionsByFilter(
+relation.tableMeta.identifier, partitionFilters.toSeq)
+} else {
+ ExternalCatalogUtils.prunePartitionsByFilter(relation.tableMeta,
+
session.sessionState.catalog.listPartitions(relation.tableMeta.identifier),
+partitionFilters.toSeq, conf.sessionLocalTimeZone)
+}
+ }
+
+ /**
+ * Update the statistics of the table.
+ */
+ private def updateTableMeta(
+ tableMeta: CatalogTable,
+ prunedPartitions: Seq[CatalogTablePartition]): CatalogTable = {
+val sizeOfPartitions = prunedPartitions.map { partition =>
+ val rawDataSize =
partition.parameters.get(StatsSetupConst.RAW_DATA_SIZE).map(_.toLong)
+ val totalSize =
partition.parameters.get(StatsSetupConst.TOTAL_SIZE).map(_.toLong)
+ if (rawDataSize.isDefined && rawDataSize.get > 0) {
+rawDataSize.get
+ } else if (totalSize.isDefined && totalSize.get > 0L) {
+totalSize.get
+ } else {
+0L
+ }
+}
+if (sizeOfPartitions.forall(s => s>0)) {
+ val sizeInBytes = sizeOfPartitions.sum
+ tableMeta.copy(stats = Some(CatalogStatistics(sizeInBytes =
BigInt(sizeInBytes
+} else {
+ tableMeta
+}
+ }
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+case op @ PhysicalOperation(projections, filters, relation:
HiveTableRelation)
+ if filters.nonEmpty && relation.isPartitioned &&
relation.prunedPartitions.isEmpty =>
+ val partitionKeyFilters = getPartitionKeyFilters(filters, relation)
+ if (partitionKeyFilters.nonEmpty) {
+val newPartitions = prunePartitions(relation, partitionKeyFilters)
+val newTableMeta = updateTableMeta(relation.tableMeta, newPartitions)
+val newRelation = relation.copy(
+ tableMeta = newTableMeta, prunedPartitions = Some(newPartitions))
+
[GitHub] [spark] cloud-fan commented on a change in pull request #27260: [SPARK-30549][SQL] Fix the subquery shown issue in UI When enable AQE
cloud-fan commented on a change in pull request #27260: [SPARK-30549][SQL] Fix
the subquery shown issue in UI When enable AQE
URL: https://github.com/apache/spark/pull/27260#discussion_r368422515
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##
@@ -132,6 +134,17 @@ case class AdaptiveSparkPlanExec(
executedPlan.resetMetrics()
}
+ private def collectSQLMetrics(plan: SparkPlan): Seq[SQLMetric] = {
+val metrics = new mutable.ArrayBuffer[SQLMetric]()
+collect(plan) {
Review comment:
we should use the normal `collect`. We don't need to get the SQLMetrics of
already materialized query stages.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27260: [SPARK-30549][SQL] Fix the subquery shown issue in UI When enable AQE
cloud-fan commented on a change in pull request #27260: [SPARK-30549][SQL] Fix
the subquery shown issue in UI When enable AQE
URL: https://github.com/apache/spark/pull/27260#discussion_r368422890
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##
@@ -151,6 +164,9 @@ case class AdaptiveSparkPlanExec(
currentPhysicalPlan = result.newPlan
if (result.newStages.nonEmpty) {
stagesToReplace = result.newStages ++ stagesToReplace
+ if (isSubquery) {
Review comment:
can we put the code in `onUpdatePlan`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference.
SparkQA commented on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference. URL: https://github.com/apache/spark/pull/27289#issuecomment-576185512 **[Test build #117108 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117108/testReport)** for PR 27289 at commit [`0f37760`](https://github.com/apache/spark/commit/0f3776006a5296f72cf9f5b20e8b92c850732095). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27249: [SPARK-30019][SQL] Add the owner property to v2 table
SparkQA commented on issue #27249: [SPARK-30019][SQL] Add the owner property to v2 table URL: https://github.com/apache/spark/pull/27249#issuecomment-576185544 **[Test build #117109 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117109/testReport)** for PR 27249 at commit [`8e70ec1`](https://github.com/apache/spark/commit/8e70ec1e59b81a777c8c8efdc4d663c3fbb77f9d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24601: [SPARK-27702][K8S] Allow using some alternatives for service accounts
SparkQA commented on issue #24601: [SPARK-27702][K8S] Allow using some alternatives for service accounts URL: https://github.com/apache/spark/pull/24601#issuecomment-576192009 **[Test build #117112 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117112/testReport)** for PR 24601 at commit [`3c9e2d8`](https://github.com/apache/spark/commit/3c9e2d82664aa0bd43b31977e65cacc385de6992). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27292: [SPARK-30582][Web UI] Spark UI is not showing Aggregated Metrics by Executor in stage page
AmplabJenkins commented on issue #27292: [SPARK-30582][Web UI] Spark UI is not showing Aggregated Metrics by Executor in stage page URL: https://github.com/apache/spark/pull/27292#issuecomment-576195674 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SaurabhChawla100 opened a new pull request #27292: [SPARK-30582][Web UI] Spark UI is not showing Aggregated Metrics by Executor in stage page
SaurabhChawla100 opened a new pull request #27292: [SPARK-30582][Web UI] Spark UI is not showing Aggregated Metrics by Executor in stage page URL: https://github.com/apache/spark/pull/27292 ### What changes were proposed in this pull request? There are scenarios where Spark History Server is located behind the VPC. So whenever api calls hit to get the executor Summary(allexecutors). There can be delay in getting the response of executor summary and in mean time "stage-page-template.html" is loaded and the response of executor Summary is not added to the stage-page-template.html. As the result of which Aggregated Metrics by Executor in stage page is showing blank. This scenario can be easily found in the cases when there is some proxy-server which is responsible for sending the request and response to spark History server. This can be reproduced in Knox/In-house proxy servers which are used to send and receive response to Spark History Server. Alternative scenario to test this case, Open the spark UI in developer mode in browser add some breakpoint in stagepage.js, this will add some delay in getting the response and now if we check the spark UI for stage Aggregated Metrics by Executor in stage page is showing blank. So In-order to fix this there is a need to add the change in stagepage.js . There is a need to add the api call to get the html page(stage-page-template.html) first and after that other api calls to get the data that needs to attached in the stagepage (like executor Summary, stageExecutorSummaryInfoKeys exc) ### Why are the changes needed? Since stage page is useful for debugging purpose, This helps in understanding how many task ran on the particular executor and information related to shuffle read and write on that executor. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Manually tested. Testing this in a reproducible way requires a running browser or HTML rendering engine that executes the JavaScript.Open the spark UI in developer mode in browser add some breakpoint in stagepage.js, this will add some delay in getting the response and now if we check the spark UI for stage Aggregated Metrics by Executor in stage page is showing blank. Before fix https://user-images.githubusercontent.com/34540906/72716739-bcfd3500-3b98-11ea-8dbe-90a135822f92.png;> After fix https://user-images.githubusercontent.com/34540906/72716782-d30af580-3b98-11ea-8764-2bde77764604.png;> This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 opened a new pull request #27293: [SPARK-29938][SQL][FOLLOW-UP] Improve AlterTableAddPartitionCommand
Ngone51 opened a new pull request #27293: [SPARK-29938][SQL][FOLLOW-UP] Improve AlterTableAddPartitionCommand URL: https://github.com/apache/spark/pull/27293 ### What changes were proposed in this pull request? This PR improves `AlterTableAddPartitionCommand` by: 1. adds an internal config for partitions batch size to avoid hard code 2. reuse `InMemoryFileIndex.bulkListLeafFiles` to perform parallel file listing to improve code reuse ### Why are the changes needed? Improve code quality. ### Does this PR introduce any user-facing change? Yes. We renamed `spark.sql.statistics.parallelFileListingInStatsComputation.enabled` to `spark.sql.parallelFileListingInCommands.enabled` as a side effect of this change. ### How was this patch tested? Pass Jenkins. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27293: [SPARK-29938][SQL][FOLLOW-UP] Improve AlterTableAddPartitionCommand
Ngone51 commented on issue #27293: [SPARK-29938][SQL][FOLLOW-UP] Improve AlterTableAddPartitionCommand URL: https://github.com/apache/spark/pull/27293#issuecomment-576208719 cc @prakharjain09 @hvanhovell @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27293: [SPARK-29938][SQL][FOLLOW-UP] Improve AlterTableAddPartitionCommand
SparkQA commented on issue #27293: [SPARK-29938][SQL][FOLLOW-UP] Improve AlterTableAddPartitionCommand URL: https://github.com/apache/spark/pull/27293#issuecomment-576208446 **[Test build #117113 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117113/testReport)** for PR 27293 at commit [`03ca0a3`](https://github.com/apache/spark/commit/03ca0a3907173411cc6d083adba51a9b9feaaca2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn edited a comment on issue #27249: [SPARK-30019][SQL] Add the owner property to v2 table
yaooqinn edited a comment on issue #27249: [SPARK-30019][SQL] Add the owner property to v2 table URL: https://github.com/apache/spark/pull/27249#issuecomment-576180489 > can we add some tests to make sure CREATE/ALTER TABLE fails if setting `owner` property? > https://github.com/apache/spark/blob/9bb78eb82fd3eecd401c4a0175e02a0f6aff925d/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala#L906-L951 We have a general test for reserved ones > Also let's update the PR title. Now we are just adding an owner property to v2 table. OK This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27277: [SPARK-30568][SQL] Invalidate interval type as a field table schema
SparkQA commented on issue #27277: [SPARK-30568][SQL] Invalidate interval type as a field table schema URL: https://github.com/apache/spark/pull/27277#issuecomment-576239251 **[Test build #117115 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117115/testReport)** for PR 27277 at commit [`9e5b209`](https://github.com/apache/spark/commit/9e5b20919b6ba95054b4541c2148bb2049a40e28). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27002: [SPARK-30346][CORE]Improve logging when events dropped
SparkQA removed a comment on issue #27002: [SPARK-30346][CORE]Improve logging when events dropped URL: https://github.com/apache/spark/pull/27002#issuecomment-576191992 **[Test build #117111 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117111/testReport)** for PR 27002 at commit [`a34d020`](https://github.com/apache/spark/commit/a34d0207b6b763f9ff9732cf5d6646c1d5359152). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27002: [SPARK-30346][CORE]Improve logging when events dropped
AmplabJenkins removed a comment on issue #27002: [SPARK-30346][CORE]Improve logging when events dropped URL: https://github.com/apache/spark/pull/27002#issuecomment-576238002 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
zero323 commented on a change in pull request #27278:
[SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
URL: https://github.com/apache/spark/pull/27278#discussion_r368510992
##
File path: R/pkg/R/functions.R
##
@@ -1332,6 +1332,48 @@ setMethod("quarter",
column(jc)
})
+#' @details
+#' \code{percentile_approx} Returns the approximate percentile value of
+#' numeric column at the given percentage.
+#'
+#' @param percentage numeric percentage at which percentile should be computed
+#' All values should be between 0 and 1.
+#' If length equals to 1 resulting column is of type double,
+#' otherwise array/
+#' @param accuracy parameter (default: 1) is a positive numeric literal
which
+#' controls approximation accuracy at the cost of memory. Higher value of
accuracy
+#' yields better accuracy, 1.0/accuracy is the relative error of the
approximation.
+#'
+#' @rdname column_aggregate_functions
+#' @aliases percentile_approx percentile_approx,Column-method
+#' @note percentile_approx since 3.0.0
+setMethod("percentile_approx",
+ signature(x = "characterOrColumn", percentage = "numeric", accuracy
= "numeric"),
+ function(x, percentage, accuracy) {
Review comment:
I think this will requires switching to ellipsis in generic:
```r
setGeneric("percentile_approx", function(x, percentage, ...) {
standardGeneric("percentile_approx") })
setMethod("percentile_approx",
signature(x = "characterOrColumn", percentage = "numeric"),
function(x, percentage, accuracy = 1) {
...
```
or `ANY` in a `signature`
```r
setGeneric("percentile_approx",
function(x, percentage, accuracy = 1) {
standardGeneric("percentile_approx") })
setMethod("percentile_approx",
signature(x = "characterOrColumn", percentage = "numeric",
accuracy = "ANY"),
function(x, percentage, accuracy = 1) {
...
```
and none seemed very precise, but I don't have strong opinion about it.
Maybe the latter.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27237: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression
SparkQA commented on issue #27237: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression URL: https://github.com/apache/spark/pull/27237#issuecomment-576250893 **[Test build #117096 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117096/testReport)** for PR 27237 at commit [`e39ef9a`](https://github.com/apache/spark/commit/e39ef9aa9b1023a6fff9013e3180adfeb0dd8bac). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27237: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression
SparkQA removed a comment on issue #27237: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression URL: https://github.com/apache/spark/pull/27237#issuecomment-576158036 **[Test build #117096 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117096/testReport)** for PR 27237 at commit [`e39ef9a`](https://github.com/apache/spark/commit/e39ef9aa9b1023a6fff9013e3180adfeb0dd8bac). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27237: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression
AmplabJenkins removed a comment on issue #27237: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression URL: https://github.com/apache/spark/pull/27237#issuecomment-576158504 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21860/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27288: [SPARK-30579][DOC] Document ORDER BY Clause of SELECT statement in SQL Reference
AmplabJenkins removed a comment on issue #27288: [SPARK-30579][DOC] Document ORDER BY Clause of SELECT statement in SQL Reference URL: https://github.com/apache/spark/pull/27288#issuecomment-576164781 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21865/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27243: [SPARK-30535][SQL] Migrate ALTER TABLE commands to the new framework
AmplabJenkins commented on issue #27243: [SPARK-30535][SQL] Migrate ALTER TABLE commands to the new framework URL: https://github.com/apache/spark/pull/27243#issuecomment-576164945 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27288: [SPARK-30579][DOC] Document ORDER BY Clause of SELECT statement in SQL Reference
AmplabJenkins commented on issue #27288: [SPARK-30579][DOC] Document ORDER BY Clause of SELECT statement in SQL Reference URL: https://github.com/apache/spark/pull/27288#issuecomment-576164781 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21865/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27277: [SPARK-30568][SQL] Invalidate interval type as a field table schema
AmplabJenkins removed a comment on issue #27277: [SPARK-30568][SQL] Invalidate interval type as a field table schema URL: https://github.com/apache/spark/pull/27277#issuecomment-576164808 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21866/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27263: [SPARK-30433][SQL][FOLLOW-UP] Optimize collect conflict plans
AmplabJenkins commented on issue #27263: [SPARK-30433][SQL][FOLLOW-UP] Optimize collect conflict plans URL: https://github.com/apache/spark/pull/27263#issuecomment-576164858 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27288: [SPARK-30579][DOC] Document ORDER BY Clause of SELECT statement in SQL Reference
AmplabJenkins removed a comment on issue #27288: [SPARK-30579][DOC] Document ORDER BY Clause of SELECT statement in SQL Reference URL: https://github.com/apache/spark/pull/27288#issuecomment-576164769 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27243: [SPARK-30535][SQL] Migrate ALTER TABLE commands to the new framework
AmplabJenkins commented on issue #27243: [SPARK-30535][SQL] Migrate ALTER TABLE commands to the new framework URL: https://github.com/apache/spark/pull/27243#issuecomment-576164956 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21869/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27277: [SPARK-30568][SQL] Invalidate interval type as a field table schema
AmplabJenkins commented on issue #27277: [SPARK-30568][SQL] Invalidate interval type as a field table schema URL: https://github.com/apache/spark/pull/27277#issuecomment-576164808 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21866/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27252: [SPARK-29231][SQL] Constraints should be inferred from cast equality constraint
AmplabJenkins commented on issue #27252: [SPARK-29231][SQL] Constraints should be inferred from cast equality constraint URL: https://github.com/apache/spark/pull/27252#issuecomment-576164934 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21868/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27252: [SPARK-29231][SQL] Constraints should be inferred from cast equality constraint
AmplabJenkins commented on issue #27252: [SPARK-29231][SQL] Constraints should be inferred from cast equality constraint URL: https://github.com/apache/spark/pull/27252#issuecomment-576164923 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27263: [SPARK-30433][SQL][FOLLOW-UP] Optimize collect conflict plans
AmplabJenkins commented on issue #27263: [SPARK-30433][SQL][FOLLOW-UP] Optimize collect conflict plans URL: https://github.com/apache/spark/pull/27263#issuecomment-576164870 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21867/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27288: [SPARK-30579][DOC] Document ORDER BY Clause of SELECT statement in SQL Reference
AmplabJenkins commented on issue #27288: [SPARK-30579][DOC] Document ORDER BY Clause of SELECT statement in SQL Reference URL: https://github.com/apache/spark/pull/27288#issuecomment-576164769 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27277: [SPARK-30568][SQL] Invalidate interval type as a field table schema
AmplabJenkins commented on issue #27277: [SPARK-30568][SQL] Invalidate interval type as a field table schema URL: https://github.com/apache/spark/pull/27277#issuecomment-576164798 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27277: [SPARK-30568][SQL] Invalidate interval type as a field table schema
AmplabJenkins removed a comment on issue #27277: [SPARK-30568][SQL] Invalidate interval type as a field table schema URL: https://github.com/apache/spark/pull/27277#issuecomment-576164798 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zuston commented on a change in pull request #27257: [SPARK-30540][Web UI] HistoryServer application link is incorrect when one application having multiple attempts
zuston commented on a change in pull request #27257: [SPARK-30540][Web UI]
HistoryServer application link is incorrect when one application having
multiple attempts
URL: https://github.com/apache/spark/pull/27257#discussion_r368419303
##
File path: core/src/main/resources/org/apache/spark/ui/static/historypage.js
##
@@ -140,7 +140,8 @@ $(document).ready(function() {
(attempt.hasOwnProperty("attemptId") ? attempt["attemptId"] + "/"
: "") + "logs";
Review comment:
The `attemptId` is not checked with line of `{{attemptId}}` in
`historypage-template.html`.
Need to ignore this link when `attempId` is empty or null? Do you have some
ideas?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27288: [SPARK-30579][DOC] Document ORDER BY Clause of SELECT statement in SQL Reference
SparkQA commented on issue #27288: [SPARK-30579][DOC] Document ORDER BY Clause of SELECT statement in SQL Reference URL: https://github.com/apache/spark/pull/27288#issuecomment-576164208 **[Test build #117101 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117101/testReport)** for PR 27288 at commit [`581a1e9`](https://github.com/apache/spark/commit/581a1e926d4347ad2f5586c3ab5212e67ff98ebc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27243: [SPARK-30535][SQL] Migrate ALTER TABLE commands to the new framework
SparkQA commented on issue #27243: [SPARK-30535][SQL] Migrate ALTER TABLE commands to the new framework URL: https://github.com/apache/spark/pull/27243#issuecomment-576164275 **[Test build #117105 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117105/testReport)** for PR 27243 at commit [`bb21f16`](https://github.com/apache/spark/commit/bb21f16203ff1917fae60bcb2e7244aac0eaa8db). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27252: [SPARK-29231][SQL] Constraints should be inferred from cast equality constraint
SparkQA commented on issue #27252: [SPARK-29231][SQL] Constraints should be inferred from cast equality constraint URL: https://github.com/apache/spark/pull/27252#issuecomment-576164274 **[Test build #117104 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117104/testReport)** for PR 27252 at commit [`7dcfe91`](https://github.com/apache/spark/commit/7dcfe915087dbe274b470928600197745a645f5e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27277: [SPARK-30568][SQL] Invalidate interval type as a field table schema
SparkQA commented on issue #27277: [SPARK-30568][SQL] Invalidate interval type as a field table schema URL: https://github.com/apache/spark/pull/27277#issuecomment-576164283 **[Test build #117102 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117102/testReport)** for PR 27277 at commit [`4fc3f4b`](https://github.com/apache/spark/commit/4fc3f4b93c212639489dc4b5b6b3432c58c5cd5a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27263: [SPARK-30433][SQL][FOLLOW-UP] Optimize collect conflict plans
SparkQA commented on issue #27263: [SPARK-30433][SQL][FOLLOW-UP] Optimize collect conflict plans URL: https://github.com/apache/spark/pull/27263#issuecomment-576164280 **[Test build #117103 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117103/testReport)** for PR 27263 at commit [`fbbf785`](https://github.com/apache/spark/commit/fbbf78542b2ceeb4c98e5416d60641fbcac6d0dd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal opened a new pull request #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference.
dilipbiswal opened a new pull request #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference. URL: https://github.com/apache/spark/pull/27289 ### What changes were proposed in this pull request? Document SORT BY clause of SELECT statement in SQL Reference Guide. ### Why are the changes needed? Currently Spark lacks documentation on the supported SQL constructs causing confusion among users who sometimes have to look at the code to understand the usage. This is aimed at addressing this issue. ### Does this PR introduce any user-facing change? Yes. **Before:** There was no documentation for this. **After.** https://user-images.githubusercontent.com/14225158/72714701-00698c00-3b24-11ea-810e-28400e196ae9.png;> https://user-images.githubusercontent.com/14225158/72714706-02cbe600-3b24-11ea-9072-6d5e6f256400.png;> https://user-images.githubusercontent.com/14225158/72714712-07909a00-3b24-11ea-9aed-51b6bb0849f2.png;> https://user-images.githubusercontent.com/14225158/72714722-0a8b8a80-3b24-11ea-9fea-4d2a166e9d92.png;> https://user-images.githubusercontent.com/14225158/72714731-0f503e80-3b24-11ea-9f6d-8223e5d88c65.png;> ### How was this patch tested? Tested using jykyll build --serve This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference.
AmplabJenkins commented on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference. URL: https://github.com/apache/spark/pull/27289#issuecomment-576190016 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference.
AmplabJenkins commented on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference. URL: https://github.com/apache/spark/pull/27289#issuecomment-576190027 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117108/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference.
SparkQA removed a comment on issue #27289: [SPARK-30581][DOC] Document SORT BY Clause of SELECT statement in SQLReference. URL: https://github.com/apache/spark/pull/27289#issuecomment-576185512 **[Test build #117108 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117108/testReport)** for PR 27289 at commit [`0f37760`](https://github.com/apache/spark/commit/0f3776006a5296f72cf9f5b20e8b92c850732095). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] liupc commented on a change in pull request #27002: [SPARK-30346][CORE]Improve logging when events dropped
liupc commented on a change in pull request #27002: [SPARK-30346][CORE]Improve
logging when events dropped
URL: https://github.com/apache/spark/pull/27002#discussion_r368448689
##
File path: core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala
##
@@ -167,20 +170,27 @@ private class AsyncEventQueue(
}
logTrace(s"Dropping event $event")
-val droppedCount = droppedEventsCounter.get
+val droppedCount = droppedEventsCounter.get - lastDroppedEventsCounter
+val lastReportTime = lastReportTimestamp.get
+val curTime = System.currentTimeMillis()
if (droppedCount > 0) {
// Don't log too frequently
- if (System.currentTimeMillis() - lastReportTimestamp >= 60 * 1000) {
-// There may be multiple threads trying to decrease
droppedEventsCounter.
-// Use "compareAndSet" to make sure only one thread can win.
-// And if another thread is increasing droppedEventsCounter,
"compareAndSet" will fail and
-// then that thread will update it.
-if (droppedEventsCounter.compareAndSet(droppedCount, 0)) {
- val prevLastReportTimestamp = lastReportTimestamp
- lastReportTimestamp = System.currentTimeMillis()
- val previous = new java.util.Date(prevLastReportTimestamp)
+ if (curTime - lastReportTime >= LOGGING_INTERVAL) {
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27293: [SPARK-29938][SQL][FOLLOW-UP] Improve AlterTableAddPartitionCommand
AmplabJenkins commented on issue #27293: [SPARK-29938][SQL][FOLLOW-UP] Improve AlterTableAddPartitionCommand URL: https://github.com/apache/spark/pull/27293#issuecomment-576209129 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21877/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27293: [SPARK-29938][SQL][FOLLOW-UP] Improve AlterTableAddPartitionCommand
AmplabJenkins commented on issue #27293: [SPARK-29938][SQL][FOLLOW-UP] Improve AlterTableAddPartitionCommand URL: https://github.com/apache/spark/pull/27293#issuecomment-576209122 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27293: [SPARK-29938][SQL][FOLLOW-UP] Improve AlterTableAddPartitionCommand
AmplabJenkins removed a comment on issue #27293: [SPARK-29938][SQL][FOLLOW-UP] Improve AlterTableAddPartitionCommand URL: https://github.com/apache/spark/pull/27293#issuecomment-576209129 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21877/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27293: [SPARK-29938][SQL][FOLLOW-UP] Improve AlterTableAddPartitionCommand
AmplabJenkins removed a comment on issue #27293: [SPARK-29938][SQL][FOLLOW-UP] Improve AlterTableAddPartitionCommand URL: https://github.com/apache/spark/pull/27293#issuecomment-576209122 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] tdas commented on a change in pull request #27265: [SPARK-30555][SQL] MERGE INTO insert action should only access columns from source table
tdas commented on a change in pull request #27265: [SPARK-30555][SQL] MERGE
INTO insert action should only access columns from source table
URL: https://github.com/apache/spark/pull/27265#discussion_r368494219
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -1326,13 +1326,20 @@ class Analyzer(
DeleteAction(resolvedDeleteCondition)
case UpdateAction(updateCondition, assignments) =>
val resolvedUpdateCondition =
updateCondition.map(resolveExpressionTopDown(_, m))
-UpdateAction(resolvedUpdateCondition,
resolveAssignments(assignments, m))
+// The update value can access columns from both target and source
tables.
+UpdateAction(
+ resolvedUpdateCondition,
+ resolveAssignments(assignments, m, resolveValuesWithSourceOnly =
false))
case o => o
}
val newNotMatchedActions = m.notMatchedActions.map {
case InsertAction(insertCondition, assignments) =>
val resolvedInsertCondition =
insertCondition.map(resolveExpressionTopDown(_, m))
Review comment:
The insert condition also has to be resolved only on the source, and not on
the target. I had the same mistake in my Delta merge as well! This bug made me
double-check my own implementation in Delta.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26162: [SPARK-29438][SS] Use partition ID of StateStoreAwareZipPartitionsRDD for determining partition ID of state store in stream-stream join
SparkQA commented on issue #26162: [SPARK-29438][SS] Use partition ID of StateStoreAwareZipPartitionsRDD for determining partition ID of state store in stream-stream join URL: https://github.com/apache/spark/pull/26162#issuecomment-576227730 **[Test build #117114 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117114/testReport)** for PR 26162 at commit [`5b4ec06`](https://github.com/apache/spark/commit/5b4ec06e4631d30384ea6b85601354909819436b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27002: [SPARK-30346][CORE]Improve logging when events dropped
SparkQA commented on issue #27002: [SPARK-30346][CORE]Improve logging when events dropped URL: https://github.com/apache/spark/pull/27002#issuecomment-576237689 **[Test build #117111 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117111/testReport)** for PR 27002 at commit [`a34d020`](https://github.com/apache/spark/commit/a34d0207b6b763f9ff9732cf5d6646c1d5359152). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #22878: [SPARK-25789][SQL] Support for Dataset of Avro
AmplabJenkins removed a comment on issue #22878: [SPARK-25789][SQL] Support for Dataset of Avro URL: https://github.com/apache/spark/pull/22878#issuecomment-576241707 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117107/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
AmplabJenkins commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-576242409 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21879/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-576242399 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
AmplabJenkins commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-576242399 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-576242409 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21879/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-576245287 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-576245295 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21880/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27237: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression
AmplabJenkins removed a comment on issue #27237: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression URL: https://github.com/apache/spark/pull/27237#issuecomment-576251543 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117096/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27237: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression
AmplabJenkins commented on issue #27237: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression URL: https://github.com/apache/spark/pull/27237#issuecomment-576251521 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27237: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression
AmplabJenkins commented on issue #27237: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression URL: https://github.com/apache/spark/pull/27237#issuecomment-576251543 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117096/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27237: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression
AmplabJenkins removed a comment on issue #27237: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression URL: https://github.com/apache/spark/pull/27237#issuecomment-576251521 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #22878: [SPARK-25789][SQL] Support for Dataset of Avro
SparkQA commented on issue #22878: [SPARK-25789][SQL] Support for Dataset of Avro URL: https://github.com/apache/spark/pull/22878#issuecomment-576255130 **[Test build #117117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117117/testReport)** for PR 22878 at commit [`e59e58c`](https://github.com/apache/spark/commit/e59e58c847389a3c7d674745667f8be0f8f7fb0e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27265: [SPARK-30555][SQL] MERGE INTO insert action should only access columns from source table
AmplabJenkins removed a comment on issue #27265: [SPARK-30555][SQL] MERGE INTO insert action should only access columns from source table URL: https://github.com/apache/spark/pull/27265#issuecomment-576161675 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
