[GitHub] [spark] AmplabJenkins commented on issue #25304: [SPARK-28570][CORE][SHUFFLE] Make UnsafeShuffleWriter use the new API.
AmplabJenkins commented on issue #25304: [SPARK-28570][CORE][SHUFFLE] Make UnsafeShuffleWriter use the new API. URL: https://github.com/apache/spark/pull/25304#issuecomment-526774311 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14993/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile closed pull request #25529: [SPARK-28804][DOCS][SQL] Document DESCRIBE QUERY in SQL Reference
gatorsmile closed pull request #25529: [SPARK-28804][DOCS][SQL] Document DESCRIBE QUERY in SQL Reference URL: https://github.com/apache/spark/pull/25529 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #25529: [SPARK-28804][DOCS][SQL] Document DESCRIBE QUERY in SQL Reference
gatorsmile commented on issue #25529: [SPARK-28804][DOCS][SQL] Document DESCRIBE QUERY in SQL Reference URL: https://github.com/apache/spark/pull/25529#issuecomment-526774147 LGTM Thanks! Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25514: [SPARK-28784][SS] Use CheckpointFileManager in StreamExecution/StreamingQueryManager for checkpoint dirs
AmplabJenkins removed a comment on issue #25514: [SPARK-28784][SS] Use CheckpointFileManager in StreamExecution/StreamingQueryManager for checkpoint dirs URL: https://github.com/apache/spark/pull/25514#issuecomment-526773964 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25514: [SPARK-28784][SS] Use CheckpointFileManager in StreamExecution/StreamingQueryManager for checkpoint dirs
AmplabJenkins removed a comment on issue #25514: [SPARK-28784][SS] Use CheckpointFileManager in StreamExecution/StreamingQueryManager for checkpoint dirs URL: https://github.com/apache/spark/pull/25514#issuecomment-526773965 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109960/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25514: [SPARK-28784][SS] Use CheckpointFileManager in StreamExecution/StreamingQueryManager for checkpoint dirs
AmplabJenkins commented on issue #25514: [SPARK-28784][SS] Use CheckpointFileManager in StreamExecution/StreamingQueryManager for checkpoint dirs URL: https://github.com/apache/spark/pull/25514#issuecomment-526773965 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109960/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25514: [SPARK-28784][SS] Use CheckpointFileManager in StreamExecution/StreamingQueryManager for checkpoint dirs
AmplabJenkins commented on issue #25514: [SPARK-28784][SS] Use CheckpointFileManager in StreamExecution/StreamingQueryManager for checkpoint dirs URL: https://github.com/apache/spark/pull/25514#issuecomment-526773964 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25514: [SPARK-28784][SS] Use CheckpointFileManager in StreamExecution/StreamingQueryManager for checkpoint dirs
SparkQA removed a comment on issue #25514: [SPARK-28784][SS] Use CheckpointFileManager in StreamExecution/StreamingQueryManager for checkpoint dirs URL: https://github.com/apache/spark/pull/25514#issuecomment-526722142 **[Test build #109960 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109960/testReport)** for PR 25514 at commit [`73ff29b`](https://github.com/apache/spark/commit/73ff29b057fad3980a35b82197a646a875512e7a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25514: [SPARK-28784][SS] Use CheckpointFileManager in StreamExecution/StreamingQueryManager for checkpoint dirs
SparkQA commented on issue #25514: [SPARK-28784][SS] Use CheckpointFileManager in StreamExecution/StreamingQueryManager for checkpoint dirs URL: https://github.com/apache/spark/pull/25514#issuecomment-526773690 **[Test build #109960 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109960/testReport)** for PR 25514 at commit [`73ff29b`](https://github.com/apache/spark/commit/73ff29b057fad3980a35b82197a646a875512e7a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
SparkQA commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526773409 **[Test build #109970 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109970/testReport)** for PR 25614 at commit [`a73e42f`](https://github.com/apache/spark/commit/a73e42f27b144cd22e947f081dbc352b6515e73f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25638: [SPARK-28932][CORE] Add scala dependency to network-common module for JDK11
SparkQA commented on issue #25638: [SPARK-28932][CORE] Add scala dependency to network-common module for JDK11 URL: https://github.com/apache/spark/pull/25638#issuecomment-526773393 **[Test build #109971 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109971/testReport)** for PR 25638 at commit [`d55904f`](https://github.com/apache/spark/commit/d55904f32bc039790ef7b1381462c4c7ef27163d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25638: [SPARK-28932][CORE] Add scala dependency to network-common module for JDK11
AmplabJenkins removed a comment on issue #25638: [SPARK-28932][CORE] Add scala dependency to network-common module for JDK11 URL: https://github.com/apache/spark/pull/25638#issuecomment-526773033 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25638: [SPARK-28932][CORE] Add scala dependency to network-common module for JDK11
AmplabJenkins removed a comment on issue #25638: [SPARK-28932][CORE] Add scala dependency to network-common module for JDK11 URL: https://github.com/apache/spark/pull/25638#issuecomment-526773037 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14990/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25638: [SPARK-28932][CORE] Add scala dependency to network-common module for JDK11
AmplabJenkins commented on issue #25638: [SPARK-28932][CORE] Add scala dependency to network-common module for JDK11 URL: https://github.com/apache/spark/pull/25638#issuecomment-526773033 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25638: [SPARK-28932][CORE] Add scala dependency to network-common module for JDK11
AmplabJenkins commented on issue #25638: [SPARK-28932][CORE] Add scala dependency to network-common module for JDK11 URL: https://github.com/apache/spark/pull/25638#issuecomment-526773037 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14990/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25638: [SPARK-28932][CORE] Add scala dependency to network-common module
SparkQA commented on issue #25638: [SPARK-28932][CORE] Add scala dependency to network-common module URL: https://github.com/apache/spark/pull/25638#issuecomment-526772082 **[Test build #109969 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109969/testReport)** for PR 25638 at commit [`7623d2b`](https://github.com/apache/spark/commit/7623d2ba3053862174b70edfed0954d494823450). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] rdblue commented on a change in pull request #25363: [SPARK-28628][SQL] Implement SupportsNamespaces in V2SessionCatalog
rdblue commented on a change in pull request #25363: [SPARK-28628][SQL]
Implement SupportsNamespaces in V2SessionCatalog
URL: https://github.com/apache/spark/pull/25363#discussion_r319693755
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/V2SessionCatalogSuite.scala
##
@@ -22,41 +22,55 @@ import java.util.Collections
import scala.collection.JavaConverters._
-import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
+import org.scalatest.BeforeAndAfter
import org.apache.spark.SparkFunSuite
import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalog.v2.{Catalogs, Identifier, TableCatalog,
TableChange}
-import org.apache.spark.sql.catalyst.analysis.{NoSuchTableException,
TableAlreadyExistsException}
+import org.apache.spark.sql.catalog.v2.{Catalogs, Identifier, NamespaceChange,
TableChange}
+import
org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException,
NoSuchNamespaceException, NoSuchTableException, TableAlreadyExistsException}
import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSparkSession
import org.apache.spark.sql.types.{DoubleType, IntegerType, LongType,
StringType, StructField, StructType, TimestampType}
import org.apache.spark.sql.util.CaseInsensitiveStringMap
-class V2SessionCatalogSuite
-extends SparkFunSuite with SharedSparkSession with BeforeAndAfter {
- import org.apache.spark.sql.catalog.v2.CatalogV2Implicits._
+class V2SessionCatalogSuite extends SparkFunSuite with SharedSparkSession with
BeforeAndAfter {
- private val emptyProps: util.Map[String, String] =
Collections.emptyMap[String, String]
- private val schema: StructType = new StructType()
+ val emptyProps: util.Map[String, String] = Collections.emptyMap[String,
String]
+ val schema: StructType = new StructType()
.add("id", IntegerType)
.add("data", StringType)
+ val testNs: Array[String] = Array("db")
+ val defaultNs: Array[String] = Array("default")
+ val testIdent: Identifier = Identifier.of(testNs, "test_table")
+
+ def newCatalog(): V2SessionCatalog = {
+val newCatalog = new V2SessionCatalog(spark.sessionState)
+newCatalog.initialize("test", CaseInsensitiveStringMap.empty())
+newCatalog
+ }
+}
+
+class V2SessionCatalogTableSuite extends V2SessionCatalogSuite {
+
+ import org.apache.spark.sql.catalog.v2.CatalogV2Implicits._
+
override protected def beforeAll(): Unit = {
super.beforeAll()
-spark.sql("""CREATE DATABASE IF NOT EXISTS db""")
-spark.sql("""CREATE DATABASE IF NOT EXISTS db2""")
-spark.sql("""CREATE DATABASE IF NOT EXISTS ns""")
-spark.sql("""CREATE DATABASE IF NOT EXISTS ns2""")
+val catalog = newCatalog()
+catalog.createNamespace(Array("db"), emptyProps)
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #25638: [SPARK-28932][CORE] Add scala dependency to network-common module
dongjoon-hyun opened a new pull request #25638: [SPARK-28932][CORE] Add scala dependency to network-common module URL: https://github.com/apache/spark/pull/25638 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce any user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] rdblue commented on a change in pull request #25363: [SPARK-28628][SQL] Implement SupportsNamespaces in V2SessionCatalog
rdblue commented on a change in pull request #25363: [SPARK-28628][SQL]
Implement SupportsNamespaces in V2SessionCatalog
URL: https://github.com/apache/spark/pull/25363#discussion_r319693641
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/V2SessionCatalogSuite.scala
##
@@ -22,41 +22,55 @@ import java.util.Collections
import scala.collection.JavaConverters._
-import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
+import org.scalatest.BeforeAndAfter
import org.apache.spark.SparkFunSuite
import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalog.v2.{Catalogs, Identifier, TableCatalog,
TableChange}
-import org.apache.spark.sql.catalyst.analysis.{NoSuchTableException,
TableAlreadyExistsException}
+import org.apache.spark.sql.catalog.v2.{Catalogs, Identifier, NamespaceChange,
TableChange}
+import
org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException,
NoSuchNamespaceException, NoSuchTableException, TableAlreadyExistsException}
import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSparkSession
import org.apache.spark.sql.types.{DoubleType, IntegerType, LongType,
StringType, StructField, StructType, TimestampType}
import org.apache.spark.sql.util.CaseInsensitiveStringMap
-class V2SessionCatalogSuite
-extends SparkFunSuite with SharedSparkSession with BeforeAndAfter {
- import org.apache.spark.sql.catalog.v2.CatalogV2Implicits._
+class V2SessionCatalogSuite extends SparkFunSuite with SharedSparkSession with
BeforeAndAfter {
Review comment:
Updated to `V2SessionCatalogBaseSuite` because it is the superclass of a
suite that provides setup. So more of a suite than just util, but it doesn't
contain tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25636: [SPARK-28778][MESOS][2.4] Fixed executors advertised address in virtual network
AmplabJenkins commented on issue #25636: [SPARK-28778][MESOS][2.4] Fixed executors advertised address in virtual network URL: https://github.com/apache/spark/pull/25636#issuecomment-526771039 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25636: [SPARK-28778][MESOS][2.4] Fixed executors advertised address in virtual network
AmplabJenkins removed a comment on issue #25636: [SPARK-28778][MESOS][2.4] Fixed executors advertised address in virtual network URL: https://github.com/apache/spark/pull/25636#issuecomment-526771039 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25636: [SPARK-28778][MESOS][2.4] Fixed executors advertised address in virtual network
AmplabJenkins removed a comment on issue #25636: [SPARK-28778][MESOS][2.4] Fixed executors advertised address in virtual network URL: https://github.com/apache/spark/pull/25636#issuecomment-526771044 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109956/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25636: [SPARK-28778][MESOS][2.4] Fixed executors advertised address in virtual network
AmplabJenkins commented on issue #25636: [SPARK-28778][MESOS][2.4] Fixed executors advertised address in virtual network URL: https://github.com/apache/spark/pull/25636#issuecomment-526771044 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109956/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25636: [SPARK-28778][MESOS][2.4] Fixed executors advertised address in virtual network
SparkQA removed a comment on issue #25636: [SPARK-28778][MESOS][2.4] Fixed executors advertised address in virtual network URL: https://github.com/apache/spark/pull/25636#issuecomment-526694985 **[Test build #109956 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109956/testReport)** for PR 25636 at commit [`f0a9274`](https://github.com/apache/spark/commit/f0a92747d4c2d18625a1b4b3334e5bb421ec03c3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25636: [SPARK-28778][MESOS][2.4] Fixed executors advertised address in virtual network
SparkQA commented on issue #25636: [SPARK-28778][MESOS][2.4] Fixed executors advertised address in virtual network URL: https://github.com/apache/spark/pull/25636#issuecomment-526770826 **[Test build #109956 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109956/testReport)** for PR 25636 at commit [`f0a9274`](https://github.com/apache/spark/commit/f0a92747d4c2d18625a1b4b3334e5bb421ec03c3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
SparkQA commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526770780 **[Test build #109968 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109968/testReport)** for PR 20965 at commit [`f617dd8`](https://github.com/apache/spark/commit/f617dd8b2539adebcfad7cc58292694fab4e0255). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
SparkQA commented on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API URL: https://github.com/apache/spark/pull/25354#issuecomment-526770773 **[Test build #109967 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109967/testReport)** for PR 25354 at commit [`57e6c5b`](https://github.com/apache/spark/commit/57e6c5be49add3cc76f402a16ebe88c28fc07bed). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
AmplabJenkins removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API URL: https://github.com/apache/spark/pull/25354#issuecomment-526770438 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
AmplabJenkins removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API URL: https://github.com/apache/spark/pull/25354#issuecomment-526770441 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14988/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526770418 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14989/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
AmplabJenkins commented on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API URL: https://github.com/apache/spark/pull/25354#issuecomment-526770441 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14988/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526770418 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14989/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526770415 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-526770415 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
AmplabJenkins commented on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API URL: https://github.com/apache/spark/pull/25354#issuecomment-526770438 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #20965: [SPARK-21870][SQL] Split aggregation code into small functions
maropu commented on a change in pull request #20965: [SPARK-21870][SQL] Split
aggregation code into small functions
URL: https://github.com/apache/spark/pull/20965#discussion_r319691667
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
##
@@ -824,59 +936,158 @@ case class HashAggregateExec(
// generating input columns, we use `currentVars`.
ctx.currentVars = new Array[ExprCode](aggregateBufferAttributes.length) ++
input
+val aggNames = aggregateExpressions.map(_.aggregateFunction.prettyName)
+// Computes start offsets for each aggregation function code
+// in the underlying buffer row.
+val bufferStartOffsets = {
+ val offsets = mutable.ArrayBuffer[Int]()
+ var curOffset = 0
+ updateExprs.foreach { exprsForOneFunc =>
+offsets += curOffset
+curOffset += exprsForOneFunc.length
+ }
+ offsets.toArray
+}
+
val updateRowInRegularHashMap: String = {
ctx.INPUT_ROW = unsafeRowBuffer
- val boundUpdateExpr = bindReferences(updateExpr, inputAttr)
- val subExprs =
ctx.subexpressionEliminationForWholeStageCodegen(boundUpdateExpr)
+ val boundUpdateExprs = updateExprs.map { updateExprsForOneFunc =>
+bindReferences(updateExprsForOneFunc, inputAttr)
+ }
+ val subExprs =
ctx.subexpressionEliminationForWholeStageCodegen(boundUpdateExprs.flatten)
val effectiveCodes = subExprs.codes.mkString("\n")
- val unsafeRowBufferEvals =
ctx.withSubExprEliminationExprs(subExprs.states) {
-boundUpdateExpr.map(_.genCode(ctx))
+ val unsafeRowBufferEvals = boundUpdateExprs.map {
boundUpdateExprsForOneFunc =>
+ctx.withSubExprEliminationExprs(subExprs.states) {
Review comment:
which function?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25637: SPARK-28921: Use latest kubernetes client for 2.3 branch [WIP]
AmplabJenkins removed a comment on issue #25637: SPARK-28921: Use latest kubernetes client for 2.3 branch [WIP] URL: https://github.com/apache/spark/pull/25637#issuecomment-526768868 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25637: SPARK-28921: Use latest kubernetes client for 2.3 branch [WIP]
AmplabJenkins commented on issue #25637: SPARK-28921: Use latest kubernetes client for 2.3 branch [WIP] URL: https://github.com/apache/spark/pull/25637#issuecomment-526768950 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25637: SPARK-28921: Use latest kubernetes client for 2.3 branch [WIP]
AmplabJenkins removed a comment on issue #25637: SPARK-28921: Use latest kubernetes client for 2.3 branch [WIP] URL: https://github.com/apache/spark/pull/25637#issuecomment-526768000 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25637: SPARK-28921: Use latest kubernetes client for 2.3 branch [WIP]
AmplabJenkins commented on issue #25637: SPARK-28921: Use latest kubernetes client for 2.3 branch [WIP] URL: https://github.com/apache/spark/pull/25637#issuecomment-526768868 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #20965: [SPARK-21870][SQL] Split aggregation code into small functions
maropu commented on a change in pull request #20965: [SPARK-21870][SQL] Split
aggregation code into small functions
URL: https://github.com/apache/spark/pull/20965#discussion_r319691046
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
##
@@ -255,41 +261,145 @@ case class HashAggregateExec(
""".stripMargin
}
+ private def isValidParamLength(paramLength: Int): Boolean = {
+sqlContext.getConf("spark.sql.HashAggregateExec.isValidParamLength", null)
match {
Review comment:
ur, i forgot to add a comment here "only for testing"...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #20965: [SPARK-21870][SQL] Split aggregation code into small functions
maropu commented on a change in pull request #20965: [SPARK-21870][SQL] Split
aggregation code into small functions
URL: https://github.com/apache/spark/pull/20965#discussion_r319690798
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -1047,6 +1047,15 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val CODEGEN_SPLIT_AGGREGATE_FUNC =
+buildConf("spark.sql.codegen.aggregate.splitAggregateFunc.enabled")
+ .internal()
+ .doc("When true, the code generator would aggregate code into individual
methods " +
Review comment:
oh... right. I'll fix. thansk
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] andygrove commented on issue #25637: SPARK-28921: Use latest kubernetes client for 2.3 branch [WIP]
andygrove commented on issue #25637: SPARK-28921: Use latest kubernetes client for 2.3 branch [WIP] URL: https://github.com/apache/spark/pull/25637#issuecomment-526768172 we may need to add some exclusions because the latest kubernetes client uses jackson 2.9.9 instead of 2.7.7 ... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25637: SPARK-28921: Use latest kubernetes client for 2.3 branch [WIP]
AmplabJenkins commented on issue #25637: SPARK-28921: Use latest kubernetes client for 2.3 branch [WIP] URL: https://github.com/apache/spark/pull/25637#issuecomment-526768000 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] andygrove opened a new pull request #25637: SPARK-28921: Use latest kubernetes client for 2.3 branch [WIP]
andygrove opened a new pull request #25637: SPARK-28921: Use latest kubernetes client for 2.3 branch [WIP] URL: https://github.com/apache/spark/pull/25637 ### What changes were proposed in this pull request? Upgrade the version of kubernets-client JAR ### Why are the changes needed? It is no longer possible to run Spark 2.3 jobs on certain EKS versions since AWS patched some CVEs in the past couple days ### Does this PR introduce any user-facing change? No ### How was this patch tested? I am in the process of manually verifying this (hence the WIP in the title) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type.
AmplabJenkins commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526766500 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109957/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type.
AmplabJenkins commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526766495 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data typ
AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526766500 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109957/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data typ
AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526766495 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type.
SparkQA commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526766134 **[Test build #109957 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109957/testReport)** for PR 25458 at commit [`b787483`](https://github.com/apache/spark/commit/b78748379992e37f6d7f550265451e549e0c5cc0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type.
SparkQA removed a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526705322 **[Test build #109957 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109957/testReport)** for PR 25458 at commit [`b787483`](https://github.com/apache/spark/commit/b78748379992e37f6d7f550265451e549e0c5cc0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25630: [SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results
HyukjinKwon commented on issue #25630: [SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results URL: https://github.com/apache/spark/pull/25630#issuecomment-526765167 Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #25630: [SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results
dongjoon-hyun closed pull request #25630: [SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results URL: https://github.com/apache/spark/pull/25630 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25630: [SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results
dongjoon-hyun commented on issue #25630: [SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results URL: https://github.com/apache/spark/pull/25630#issuecomment-526764817 I compared both results. +1, LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25630: [SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results
dongjoon-hyun commented on issue #25630: [SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results URL: https://github.com/apache/spark/pull/25630#issuecomment-526764561 Got it. It's much better! So, in the successful run, there is no information in the last Jenkins. When we have a failure, `Error Message` will have that information. Did I understand correctly? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #25630: [SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results
HyukjinKwon edited a comment on issue #25630: [SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results URL: https://github.com/apache/spark/pull/25630#issuecomment-526762342 Yes but the log on failure test has its file name now -  I tried to find a better way but couldn't find to fix in Jenkins or SBT. This PR proposes a workaround by appending the test file name into the assert log as I described in PR description. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #25630: [SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results
HyukjinKwon edited a comment on issue #25630: [SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results URL: https://github.com/apache/spark/pull/25630#issuecomment-526762342 Yes but the log on failure test has its file name now - https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109942/testReport/org.apache.spark.sql/SQLQueryTestSuite/sql___Scalar_Pandas_UDF/ I tried to find a better way but couldn't find to fix in Jenkins or SBT. This PR proposes a workaround by appending the test file name into the assert log as I described in PR description. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #25630: [SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results
HyukjinKwon edited a comment on issue #25630: [SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results URL: https://github.com/apache/spark/pull/25630#issuecomment-526762342 Yes but the log on failure test has its file name now - https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109942/testReport/org.apache.spark.sql/SQLQueryTestSuite/sql/ I tried to find a better way but couldn't find to fix in Jenkins or SBT. This PR proposes a workaround by appending the test file name into the assert log as I described in PR description. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on issue #25630: [SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results
dilipbiswal commented on issue #25630: [SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results URL: https://github.com/apache/spark/pull/25630#issuecomment-526762939 @HyukjinKwon Its much better than what we had before. thanks !! Looks good to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526762201 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14987/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526762195 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25630: [SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results
HyukjinKwon commented on issue #25630: [SPARK-28894][SQL][TESTS] Add a clue to make it easier to debug via Jenkins's test results URL: https://github.com/apache/spark/pull/25630#issuecomment-526762342 Yes but the log on failure test has it's file name - https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109942/testReport/org.apache.spark.sql/SQLQueryTestSuite/sql/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526762201 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14987/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526762195 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526761705 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109966/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526761698 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526760408 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14985/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526761698 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526761705 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109966/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
SparkQA removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526759736 **[Test build #109966 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109966/testReport)** for PR 25614 at commit [`87772a3`](https://github.com/apache/spark/commit/87772a3e3acdd3302120be306e82faf1172b5871). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
SparkQA commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526761688 **[Test build #109966 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109966/testReport)** for PR 25614 at commit [`87772a3`](https://github.com/apache/spark/commit/87772a3e3acdd3302120be306e82faf1172b5871). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526759954 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14984/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #20965: [SPARK-21870][SQL] Split aggregation code into small functions
viirya commented on a change in pull request #20965: [SPARK-21870][SQL] Split
aggregation code into small functions
URL: https://github.com/apache/spark/pull/20965#discussion_r319684092
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
##
@@ -824,59 +936,158 @@ case class HashAggregateExec(
// generating input columns, we use `currentVars`.
ctx.currentVars = new Array[ExprCode](aggregateBufferAttributes.length) ++
input
+val aggNames = aggregateExpressions.map(_.aggregateFunction.prettyName)
+// Computes start offsets for each aggregation function code
+// in the underlying buffer row.
+val bufferStartOffsets = {
+ val offsets = mutable.ArrayBuffer[Int]()
+ var curOffset = 0
+ updateExprs.foreach { exprsForOneFunc =>
+offsets += curOffset
+curOffset += exprsForOneFunc.length
+ }
+ offsets.toArray
+}
+
val updateRowInRegularHashMap: String = {
ctx.INPUT_ROW = unsafeRowBuffer
- val boundUpdateExpr = bindReferences(updateExpr, inputAttr)
- val subExprs =
ctx.subexpressionEliminationForWholeStageCodegen(boundUpdateExpr)
+ val boundUpdateExprs = updateExprs.map { updateExprsForOneFunc =>
+bindReferences(updateExprsForOneFunc, inputAttr)
+ }
+ val subExprs =
ctx.subexpressionEliminationForWholeStageCodegen(boundUpdateExprs.flatten)
val effectiveCodes = subExprs.codes.mkString("\n")
- val unsafeRowBufferEvals =
ctx.withSubExprEliminationExprs(subExprs.states) {
-boundUpdateExpr.map(_.genCode(ctx))
+ val unsafeRowBufferEvals = boundUpdateExprs.map {
boundUpdateExprsForOneFunc =>
+ctx.withSubExprEliminationExprs(subExprs.states) {
Review comment:
Will sub expr be put into function argument?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526760401 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526760408 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14985/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526760401 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526759950 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526759950 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526759954 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14984/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
SparkQA commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526759736 **[Test build #109966 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109966/testReport)** for PR 25614 at commit [`87772a3`](https://github.com/apache/spark/commit/87772a3e3acdd3302120be306e82faf1172b5871). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25629: [SPARK-28931][CORE][TESTS] Fix couple of bugs in FsHistoryProviderSuite
SparkQA commented on issue #25629: [SPARK-28931][CORE][TESTS] Fix couple of bugs in FsHistoryProviderSuite URL: https://github.com/apache/spark/pull/25629#issuecomment-526759731 **[Test build #109965 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109965/testReport)** for PR 25629 at commit [`c11b50d`](https://github.com/apache/spark/commit/c11b50dc753d8c5eaae58274aaf3c5654b90a4d2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on issue #25618: [SPARK-28908][SS]Implement Kafka EOS sink for Structured Streaming
HeartSaVioR edited a comment on issue #25618: [SPARK-28908][SS]Implement Kafka EOS sink for Structured Streaming URL: https://github.com/apache/spark/pull/25618#issuecomment-526593592 Well, someone could say it as 2PC since the behavior is similar, but generally 2PC assumes coordinator and participants. In second phase, coordinator "ask" for commit/abort to participants, not committing/aborting things directly participants just did in first phase. Based on that, driver should request tasks to commit their outputs, but Spark doesn't provide such flow. So that's pretty simplified version of 2PC and also pretty limited. I think the point is whether we are feeling OK to have exactly-once with some restrictions end users need to be aware of. Could you please initiate discussion on this in Spark dev mailing list? That would be good to hear others' voices. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on issue #25618: [SPARK-28908][SS]Implement Kafka EOS sink for Structured Streaming
HeartSaVioR edited a comment on issue #25618: [SPARK-28908][SS]Implement Kafka EOS sink for Structured Streaming URL: https://github.com/apache/spark/pull/25618#issuecomment-526593592 Well, someone could say it as 2PC since the behavior is similar, but generally 2PC assumes coordinator and participants. In second phase, coordinator "ask" for commit/abort to participants, not committing/aborting things directly participants just did in first phase by itself. Based on that, driver should request tasks to commit their outputs, but Spark doesn't provide such flow. So that's pretty simplified version of 2PC and also pretty limited. I think the point is whether we are feeling OK to have exactly-once with some restrictions end users need to be aware of. Could you please initiate discussion on this in Spark dev mailing list? That would be good to hear others' voices. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] younggyuchun commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type.
younggyuchun commented on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526758839 Thank yoy all. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] younggyuchun edited a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type.
younggyuchun edited a comment on issue #25458: [SPARK-27931][SQL] Accept "true", "yes", "1", "false", "no", "0", and unique prefixes as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-526758839 Thank you all. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526758431 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
SparkQA removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526757991 **[Test build #109964 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109964/testReport)** for PR 25614 at commit [`da6025f`](https://github.com/apache/spark/commit/da6025f32375cb16bc03f1c5bcca19f92e763c53). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins removed a comment on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526758433 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109964/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jinxingwang commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
jinxingwang commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526758515 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
SparkQA commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526758423 **[Test build #109964 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109964/testReport)** for PR 25614 at commit [`da6025f`](https://github.com/apache/spark/commit/da6025f32375cb16bc03f1c5bcca19f92e763c53). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526758433 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109964/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
AmplabJenkins commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526758431 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #25607: [SPARK-28891][BUILD][2.3] backport do-release-docker.sh to branch-2.3
dongjoon-hyun edited a comment on issue #25607: [SPARK-28891][BUILD][2.3] backport do-release-docker.sh to branch-2.3 URL: https://github.com/apache/spark/pull/25607#issuecomment-526753946 Since the vote passed, I'll merge this to `branch-2.3` as the last commit on `branch-2.3`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #25607: [SPARK-28891][BUILD][2.3] backport do-release-docker.sh to branch-2.3
dongjoon-hyun closed pull request #25607: [SPARK-28891][BUILD][2.3] backport do-release-docker.sh to branch-2.3 URL: https://github.com/apache/spark/pull/25607 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
AmplabJenkins removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API URL: https://github.com/apache/spark/pull/25354#issuecomment-526757772 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109963/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25607: [SPARK-28891][BUILD][2.3] backport do-release-docker.sh to branch-2.3
dongjoon-hyun commented on issue #25607: [SPARK-28891][BUILD][2.3] backport do-release-docker.sh to branch-2.3 URL: https://github.com/apache/spark/pull/25607#issuecomment-526758255 Thank you, @kiszk and @srowen . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
SparkQA removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API URL: https://github.com/apache/spark/pull/25354#issuecomment-526756118 **[Test build #109963 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109963/testReport)** for PR 25354 at commit [`e424c2c`](https://github.com/apache/spark/commit/e424c2c36ef2c38a689f90a12fe13ca4ff9a6098). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix
SparkQA commented on issue #25614: [WIP][SPARK-28887][K8S] Executor pod status fix URL: https://github.com/apache/spark/pull/25614#issuecomment-526757991 **[Test build #109964 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109964/testReport)** for PR 25614 at commit [`da6025f`](https://github.com/apache/spark/commit/da6025f32375cb16bc03f1c5bcca19f92e763c53). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
AmplabJenkins removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API URL: https://github.com/apache/spark/pull/25354#issuecomment-526757763 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
AmplabJenkins removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API URL: https://github.com/apache/spark/pull/25354#issuecomment-526757496 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
