[GitHub] [spark] AmplabJenkins commented on pull request #32914: [SPARK-35763][SS] Add a new copy method to StateStoreCustomMetric
AmplabJenkins commented on pull request #32914: URL: https://github.com/apache/spark/pull/32914#issuecomment-861233427 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32912: [SPARK-35429][CORE] Remove commons-httpclient due to EOL and CVEs
SparkQA commented on pull request #32912: URL: https://github.com/apache/spark/pull/32912#issuecomment-861235546 **[Test build #139797 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139797/testReport)** for PR 32912 at commit [`3c0eb47`](https://github.com/apache/spark/commit/3c0eb4763bf74a67eabb21fc02565790d9505ee0). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32645: [SPARK-35129][SQL] Construct year-month interval column from integral fields
SparkQA commented on pull request #32645: URL: https://github.com/apache/spark/pull/32645#issuecomment-861235933 **[Test build #139798 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139798/testReport)** for PR 32645 at commit [`b9248aa`](https://github.com/apache/spark/commit/b9248aafc830c2f5b4daee9b6a6ed8b5d572d038). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32904: [SPARK-33122][FOLLOWUP][SQL] Distinct aggs are not duplicate sensitive
dongjoon-hyun commented on pull request #32904: URL: https://github.com/apache/spark/pull/32904#issuecomment-861236385 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #32904: [SPARK-33122][FOLLOWUP][SQL] Distinct aggs are not duplicate sensitive
dongjoon-hyun edited a comment on pull request #32904: URL: https://github.com/apache/spark/pull/32904#issuecomment-861236385 Hi, @tanelk . Could you take a look at GitHub Action UT failures? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32904: [SPARK-33122][FOLLOWUP][SQL] Distinct aggs are not duplicate sensitive
dongjoon-hyun commented on pull request #32904: URL: https://github.com/apache/spark/pull/32904#issuecomment-861236940 cc @maropu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32904: [SPARK-33122][FOLLOWUP][SQL] Distinct aggs are not duplicate sensitive
SparkQA commented on pull request #32904: URL: https://github.com/apache/spark/pull/32904#issuecomment-861238629 **[Test build #139799 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139799/testReport)** for PR 32904 at commit [`3f0cdae`](https://github.com/apache/spark/commit/3f0cdae64db7a3a5dafd628f890e936cf743d16d). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32904: [SPARK-33122][FOLLOWUP][SQL] Distinct aggs are not duplicate sensitive
AmplabJenkins removed a comment on pull request #32904: URL: https://github.com/apache/spark/pull/32904#issuecomment-860597774 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32645: [SPARK-35129][SQL] Construct year-month interval column from integral fields
AmplabJenkins removed a comment on pull request #32645: URL: https://github.com/apache/spark/pull/32645#issuecomment-861233047 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139795/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32881: [SPARK-33298][CORE] Decouple file naming from FileCommitProtocol
SparkQA commented on pull request #32881: URL: https://github.com/apache/spark/pull/32881#issuecomment-861248200 **[Test build #139796 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139796/testReport)** for PR 32881 at commit [`1bb6e16`](https://github.com/apache/spark/commit/1bb6e16721eb1b0d76762da3b8349988583ae692). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32881: [SPARK-33298][CORE] Decouple file naming from FileCommitProtocol
SparkQA removed a comment on pull request #32881: URL: https://github.com/apache/spark/pull/32881#issuecomment-861174335 **[Test build #139796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139796/testReport)** for PR 32881 at commit [`1bb6e16`](https://github.com/apache/spark/commit/1bb6e16721eb1b0d76762da3b8349988583ae692). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32885: [SPARK-35742][SQL] Expression.semanticEquals should be symmetrical
SparkQA commented on pull request #32885: URL: https://github.com/apache/spark/pull/32885#issuecomment-861256012 **[Test build #139789 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139789/testReport)** for PR 32885 at commit [`5095fff`](https://github.com/apache/spark/commit/5095fffcd0cd76548eadef802cdb81261029ff12). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32885: [SPARK-35742][SQL] Expression.semanticEquals should be symmetrical
SparkQA removed a comment on pull request #32885: URL: https://github.com/apache/spark/pull/32885#issuecomment-861132129 **[Test build #139789 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139789/testReport)** for PR 32885 at commit [`5095fff`](https://github.com/apache/spark/commit/5095fffcd0cd76548eadef802cdb81261029ff12). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32880: [SPARK-35056][SQL] Group exception messages in execution/streaming
SparkQA commented on pull request #32880: URL: https://github.com/apache/spark/pull/32880#issuecomment-861257115 **[Test build #139790 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139790/testReport)** for PR 32880 at commit [`7c8d275`](https://github.com/apache/spark/commit/7c8d275364b1ccf83492e55f0a63f357ec7fb8ad). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32880: [SPARK-35056][SQL] Group exception messages in execution/streaming
SparkQA removed a comment on pull request #32880: URL: https://github.com/apache/spark/pull/32880#issuecomment-861132135 **[Test build #139790 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139790/testReport)** for PR 32880 at commit [`7c8d275`](https://github.com/apache/spark/commit/7c8d275364b1ccf83492e55f0a63f357ec7fb8ad). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang opened a new pull request #32915: [SPARK-35764][SQL] Assign pretty names to TimestampWithoutTZType
gengliangwang opened a new pull request #32915: URL: https://github.com/apache/spark/pull/32915 ### What changes were proposed in this pull request? In the PR, I propose to override the typeName() method in TimestampWithoutTZType, and assign it a name according to the ANSI SQL standard  ### Why are the changes needed? To improve Spark SQL user experience, and have readable types in error messages. ### Does this PR introduce _any_ user-facing change? No, the new timestamp type is not released yet. ### How was this patch tested? Unit test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32645: [SPARK-35129][SQL] Construct year-month interval column from integral fields
SparkQA commented on pull request #32645: URL: https://github.com/apache/spark/pull/32645#issuecomment-861271831 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44324/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32904: [SPARK-33122][FOLLOWUP][SQL] Distinct aggs are not duplicate sensitive
SparkQA commented on pull request #32904: URL: https://github.com/apache/spark/pull/32904#issuecomment-861274681 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44325/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #32385: [WIP][SPARK-35275][CORE] Add checksum for shuffle blocks and diagnose corruption
Ngone51 commented on pull request #32385: URL: https://github.com/apache/spark/pull/32385#issuecomment-861274556 oh..@mridulm Sorry if I confused you here. I have planed to split this PR into two separate PRs to ease the review: * write checksum file (Ready to review https://github.com/apache/spark/pull/32401) * diagnose corruption (Not done yet) So please help review the smaller PR there. And I'll try to resolve your comments in the separate PRs. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32881: [SPARK-33298][CORE] Decouple file naming from FileCommitProtocol
AmplabJenkins commented on pull request #32881: URL: https://github.com/apache/spark/pull/32881#issuecomment-861276632 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139796/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32880: [SPARK-35056][SQL] Group exception messages in execution/streaming
AmplabJenkins commented on pull request #32880: URL: https://github.com/apache/spark/pull/32880#issuecomment-861276637 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139790/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32885: [SPARK-35742][SQL] Expression.semanticEquals should be symmetrical
AmplabJenkins commented on pull request #32885: URL: https://github.com/apache/spark/pull/32885#issuecomment-861276633 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139789/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on pull request #32885: [SPARK-35742][SQL] Expression.semanticEquals should be symmetrical
viirya commented on pull request #32885: URL: https://github.com/apache/spark/pull/32885#issuecomment-861276752 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32881: [SPARK-33298][CORE] Decouple file naming from FileCommitProtocol
AmplabJenkins removed a comment on pull request #32881: URL: https://github.com/apache/spark/pull/32881#issuecomment-861276632 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139796/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32880: [SPARK-35056][SQL] Group exception messages in execution/streaming
AmplabJenkins removed a comment on pull request #32880: URL: https://github.com/apache/spark/pull/32880#issuecomment-861276637 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139790/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32885: [SPARK-35742][SQL] Expression.semanticEquals should be symmetrical
AmplabJenkins removed a comment on pull request #32885: URL: https://github.com/apache/spark/pull/32885#issuecomment-861276633 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139789/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32645: [SPARK-35129][SQL] Construct year-month interval column from integral fields
MaxGekk commented on a change in pull request #32645:
URL: https://github.com/apache/spark/pull/32645#discussion_r651546525
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/intervalExpressions.scala
##
@@ -345,6 +346,66 @@ case class MakeInterval(
)
}
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(years, months) - Make year-month interval from years,
months.",
+ arguments = """
+Arguments:
+ * years - the number of years, positive or negative
+ * months - the number of months, positive or negative
+ """,
+ examples = """
+Examples:
+ > SELECT _FUNC_(1, 2);
+ 1-2
+ > SELECT _FUNC_(1, 0);
+ 1-0
+ > SELECT _FUNC_(0, 1);
+ 0-1
+ """,
+ since = "3.2.0",
+ group = "datetime_funcs")
+// scalastyle:on line.size.limit
+case class MakeYMInterval(years: Expression, months: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes with NullIntolerant {
+
+ def this(years: Expression) = this(years, Literal(0))
+ def this() = this(Literal(0))
+
+ override def left: Expression = years
+ override def right: Expression = months
+ override def inputTypes: Seq[AbstractDataType] = Seq(IntegerType,
IntegerType)
+ override def dataType: DataType = YearMonthIntervalType
+ override def nullable: Boolean = children.exists(_.nullable)
+
+ private def evalIntValue(dt: DataType, value: Any): Long = dt match {
+case _: ByteType | _: ShortType | _: IntegerType =>
value.asInstanceOf[Number].longValue()
+case _: LongType => value.asInstanceOf[Long]
+ }
+
+ override def nullSafeEval(year: Any, month: Any): Any = {
+LongExactNumeric.toInt(Math.addExact(evalIntValue(right.dataType, month),
+ Math.multiplyExact(evalIntValue(left.dataType, year), MONTHS_PER_YEAR)))
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+defineCodeGen(ctx, ev, (years, months) => {
+ val extractor = LongExactNumeric.getClass.getName.stripSuffix("$")
+ s"""
+ |$extractor.toInt(java.lang.Math.addExact($months,
+ | java.lang.Math.multiplyExact($years, $MONTHS_PER_YEAR)))
+ |""".stripMargin
+})
+ }
+
+ override def prettyName: String = "make_ym_interval"
+
+ override protected def withNewChildrenInternal(
+ newLeft: Expression,
+ newRight: Expression): Expression =
+copy(years = newLeft, months = newRight)
+}
Review comment:
ok. I see. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dgd-contributor opened a new pull request #32916: [WIP][SPARK-35064][SQL] group error in spark-catalyst
dgd-contributor opened a new pull request #32916: URL: https://github.com/apache/spark/pull/32916 What changes were proposed in this pull request? This PR group exception messages in sql/catalyst/src/main/scala/org/apache/spark/sql (except catalyst) Why are the changes needed? It will largely help with standardization of error messages and its maintenance. Does this PR introduce any user-facing change? No. Error messages remain unchanged. How was this patch tested? No new tests - pass all original tests to make sure it doesn't break any existing behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32916: [WIP][SPARK-35064][SQL] group error in spark-catalyst
AmplabJenkins commented on pull request #32916: URL: https://github.com/apache/spark/pull/32916#issuecomment-861279329 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32915: [SPARK-35764][SQL] Assign pretty names to TimestampWithoutTZType
SparkQA commented on pull request #32915: URL: https://github.com/apache/spark/pull/32915#issuecomment-861279596 **[Test build #139800 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139800/testReport)** for PR 32915 at commit [`1e53571`](https://github.com/apache/spark/commit/1e5357158acd7fe371de542c09e69052708e5f89). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32821: [SPARK-35342][PYTHON] Introduce DecimalOps and make `isnull` method data-type-based
SparkQA commented on pull request #32821: URL: https://github.com/apache/spark/pull/32821#issuecomment-861279903 **[Test build #139801 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139801/testReport)** for PR 32821 at commit [`bca1d84`](https://github.com/apache/spark/commit/bca1d84b288a166e367ff0785b298808c670b17b). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32645: [SPARK-35129][SQL] Construct year-month interval column from integral fields
SparkQA commented on pull request #32645: URL: https://github.com/apache/spark/pull/32645#issuecomment-861280008 **[Test build #139802 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139802/testReport)** for PR 32645 at commit [`06c908b`](https://github.com/apache/spark/commit/06c908b487ab218bd14e1e679d179f702f8c3e97). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32645: [SPARK-35129][SQL] Construct year-month interval column from integral fields
MaxGekk commented on a change in pull request #32645:
URL: https://github.com/apache/spark/pull/32645#discussion_r651548989
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/intervalExpressions.scala
##
@@ -345,6 +346,66 @@ case class MakeInterval(
)
}
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(years, months) - Make year-month interval from years,
months.",
+ arguments = """
+Arguments:
+ * years - the number of years, positive or negative
+ * months - the number of months, positive or negative
+ """,
+ examples = """
+Examples:
+ > SELECT _FUNC_(1, 2);
+ 1-2
+ > SELECT _FUNC_(1, 0);
+ 1-0
+ > SELECT _FUNC_(0, 1);
+ 0-1
+ """,
+ since = "3.2.0",
+ group = "datetime_funcs")
+// scalastyle:on line.size.limit
+case class MakeYMInterval(years: Expression, months: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes with NullIntolerant {
+
+ def this(years: Expression) = this(years, Literal(0))
+ def this() = this(Literal(0))
+
+ override def left: Expression = years
+ override def right: Expression = months
+ override def inputTypes: Seq[AbstractDataType] = Seq(IntegerType,
IntegerType)
+ override def dataType: DataType = YearMonthIntervalType
+ override def nullable: Boolean = children.exists(_.nullable)
+
+ private def evalIntValue(dt: DataType, value: Any): Long = dt match {
+case _: ByteType | _: ShortType | _: IntegerType =>
value.asInstanceOf[Number].longValue()
Review comment:
Let's follow `MakeInterval`, and handle/accept only `IntegerType` as it
is declared by:
```scala
override def inputTypes: Seq[AbstractDataType] = Seq(IntegerType,
IntegerType)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32904: [SPARK-33122][FOLLOWUP][SQL] Distinct aggs are not duplicate sensitive
SparkQA commented on pull request #32904: URL: https://github.com/apache/spark/pull/32904#issuecomment-861281750 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44326/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak opened a new pull request #32917: [SPARK-35758][DOCS] Update the document about building Spark with Hadoop
sarutak opened a new pull request #32917: URL: https://github.com/apache/spark/pull/32917 ### What changes were proposed in this pull request? This PR updates the document about building Spark with Hadoop for Hadoop 3.x and Hadoop 3.2. ### Why are the changes needed? The document says about how to build like as follows: ``` ./build/mvn -Pyarn -Dhadoop.version=2.8.5 -DskipTests clean package ``` But this command fails because the default build settings are for Hadoop 3.x. So, we need to modify the command example. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I confirmed both of these commands successfully finished. ``` ./build/mvn -Pyarn -Dhadoop.version=3.3.0 -DskipTests package ./build/mvn -Phadoop-2.7 -Pyarn -Dhadoop.version=2.8.5 -DskipTests package ``` I also built the document and confirmed the result. This is before:  And this is after:  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32917: [SPARK-35758][DOCS] Update the document about building Spark with Hadoop
SparkQA commented on pull request #32917: URL: https://github.com/apache/spark/pull/32917#issuecomment-861282868 **[Test build #139803 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139803/testReport)** for PR 32917 at commit [`06e7b30`](https://github.com/apache/spark/commit/06e7b301ecf3c8e96332c5c57725c101ca61bfb4). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32912: [SPARK-35429][CORE] Remove commons-httpclient due to EOL and CVEs
SparkQA commented on pull request #32912: URL: https://github.com/apache/spark/pull/32912#issuecomment-861283860 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44323/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32912: [SPARK-35429][CORE] Remove commons-httpclient due to EOL and CVEs
AmplabJenkins commented on pull request #32912: URL: https://github.com/apache/spark/pull/32912#issuecomment-861283903 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44323/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32912: [SPARK-35429][CORE] Remove commons-httpclient due to EOL and CVEs
AmplabJenkins removed a comment on pull request #32912: URL: https://github.com/apache/spark/pull/32912#issuecomment-861283903 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44323/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #32845: [SPARK-35691][CORE] addFile/addJar/addDirectory should put CanonicalFile
sarutak commented on pull request #32845: URL: https://github.com/apache/spark/pull/32845#issuecomment-861294700 retest this please. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #32845: [SPARK-35691][CORE] addFile/addJar/addDirectory should put CanonicalFile
sarutak commented on pull request #32845: URL: https://github.com/apache/spark/pull/32845#issuecomment-861294828 LGTM. Pending Jenkins. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32917: [SPARK-35758][DOCS] Update the document about building Spark with Hadoop for Hadoop 2.x and 3.x
SparkQA commented on pull request #32917: URL: https://github.com/apache/spark/pull/32917#issuecomment-861295559 **[Test build #139803 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139803/testReport)** for PR 32917 at commit [`06e7b30`](https://github.com/apache/spark/commit/06e7b301ecf3c8e96332c5c57725c101ca61bfb4). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #32909: [WIP][SPARK-35680][SQL] Add fields to `YearMonthIntervalType`
MaxGekk commented on pull request #32909: URL: https://github.com/apache/spark/pull/32909#issuecomment-861295970 @cloud-fan I think it is ready for review. I just need to replace `SPARK-X` in the code, and open sub-tasks in JIRA. Could you take a look at this, please. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32917: [SPARK-35758][DOCS] Update the document about building Spark with Hadoop for Hadoop 2.x and 3.x
SparkQA removed a comment on pull request #32917: URL: https://github.com/apache/spark/pull/32917#issuecomment-861282868 **[Test build #139803 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139803/testReport)** for PR 32917 at commit [`06e7b30`](https://github.com/apache/spark/commit/06e7b301ecf3c8e96332c5c57725c101ca61bfb4). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32645: [SPARK-35129][SQL] Construct year-month interval column from integral fields
SparkQA commented on pull request #32645: URL: https://github.com/apache/spark/pull/32645#issuecomment-861297404 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44324/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32904: [SPARK-33122][FOLLOWUP][SQL] Distinct aggs are not duplicate sensitive
SparkQA commented on pull request #32904: URL: https://github.com/apache/spark/pull/32904#issuecomment-861297951 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44325/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32904: [SPARK-33122][FOLLOWUP][SQL] Distinct aggs are not duplicate sensitive
SparkQA commented on pull request #32904: URL: https://github.com/apache/spark/pull/32904#issuecomment-861309244 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44326/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32821: [SPARK-35342][PYTHON] Introduce DecimalOps and make `isnull` method data-type-based
SparkQA commented on pull request #32821: URL: https://github.com/apache/spark/pull/32821#issuecomment-861312401 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44328/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32645: [SPARK-35129][SQL] Construct year-month interval column from integral fields
SparkQA commented on pull request #32645: URL: https://github.com/apache/spark/pull/32645#issuecomment-861314335 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44329/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #32885: [SPARK-35742][SQL] Expression.semanticEquals should be symmetrical
cloud-fan commented on pull request #32885: URL: https://github.com/apache/spark/pull/32885#issuecomment-861315658 thanks for the review, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #32885: [SPARK-35742][SQL] Expression.semanticEquals should be symmetrical
cloud-fan closed pull request #32885: URL: https://github.com/apache/spark/pull/32885 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #32845: [SPARK-35691][CORE] addFile/addJar/addDirectory should put CanonicalFile
sarutak commented on pull request #32845: URL: https://github.com/apache/spark/pull/32845#issuecomment-861316002 retest this please. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32915: [SPARK-35764][SQL] Assign pretty names to TimestampWithoutTZType
SparkQA commented on pull request #32915: URL: https://github.com/apache/spark/pull/32915#issuecomment-861318378 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44327/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32904: [SPARK-33122][FOLLOWUP][SQL] Distinct aggs are not duplicate sensitive
AmplabJenkins commented on pull request #32904: URL: https://github.com/apache/spark/pull/32904#issuecomment-861320182 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32917: [SPARK-35758][DOCS] Update the document about building Spark with Hadoop for Hadoop 2.x and 3.x
AmplabJenkins commented on pull request #32917: URL: https://github.com/apache/spark/pull/32917#issuecomment-861320181 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139803/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32645: [SPARK-35129][SQL] Construct year-month interval column from integral fields
AmplabJenkins commented on pull request #32645: URL: https://github.com/apache/spark/pull/32645#issuecomment-861320187 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44324/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32881: [SPARK-33298][CORE] Decouple file naming from FileCommitProtocol
SparkQA commented on pull request #32881: URL: https://github.com/apache/spark/pull/32881#issuecomment-861321549 **[Test build #139805 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139805/testReport)** for PR 32881 at commit [`65346ab`](https://github.com/apache/spark/commit/65346ab1e7dd110777d2243d3337b1d335bb9326). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32909: [WIP][SPARK-35680][SQL] Add fields to `YearMonthIntervalType`
SparkQA commented on pull request #32909: URL: https://github.com/apache/spark/pull/32909#issuecomment-861321432 **[Test build #139804 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139804/testReport)** for PR 32909 at commit [`efe001c`](https://github.com/apache/spark/commit/efe001cde3a9b0a0b4b8e4f8db0c4334f0e0ca2f). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32845: [SPARK-35691][CORE] addFile/addJar/addDirectory should put CanonicalFile
SparkQA commented on pull request #32845: URL: https://github.com/apache/spark/pull/32845#issuecomment-861321691 **[Test build #139806 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139806/testReport)** for PR 32845 at commit [`f3ec9a3`](https://github.com/apache/spark/commit/f3ec9a3229e937ddde0ff9f72342043a058147c4). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32742: [SPARK-35608][SQL] Support AQE optimizer side transformUpWithPruning
SparkQA commented on pull request #32742: URL: https://github.com/apache/spark/pull/32742#issuecomment-861321820 **[Test build #139807 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139807/testReport)** for PR 32742 at commit [`1a770bb`](https://github.com/apache/spark/commit/1a770bb1cee034ad67a256df94e8022dd32bdbb9). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] tanelk commented on pull request #32904: [SPARK-33122][FOLLOWUP][SQL] Distinct aggs are not duplicate sensitive
tanelk commented on pull request #32904: URL: https://github.com/apache/spark/pull/32904#issuecomment-861325178 > Hi, @tanelk . Could you take a look at GitHub Action UT failures? I ran `StreamingAggregationSuite` and `KafkaMicroBatchV1SourceWithAdminSuite` localy and all tests passed. Lets wait for the jenkins results. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32904: [SPARK-33122][FOLLOWUP][SQL] Distinct aggs are not duplicate sensitive
AmplabJenkins removed a comment on pull request #32904: URL: https://github.com/apache/spark/pull/32904#issuecomment-861320182 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32645: [SPARK-35129][SQL] Construct year-month interval column from integral fields
AmplabJenkins removed a comment on pull request #32645: URL: https://github.com/apache/spark/pull/32645#issuecomment-861320187 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44324/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32917: [SPARK-35758][DOCS] Update the document about building Spark with Hadoop for Hadoop 2.x and 3.x
AmplabJenkins removed a comment on pull request #32917: URL: https://github.com/apache/spark/pull/32917#issuecomment-861320181 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139803/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #32915: [SPARK-35764][SQL] Assign pretty names to TimestampWithoutTZType
MaxGekk commented on pull request #32915: URL: https://github.com/apache/spark/pull/32915#issuecomment-861332617 +1, LGTM. Merging to master. Thank you, @gengliangwang . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32917: [SPARK-35758][DOCS] Update the document about building Spark with Hadoop for Hadoop 2.x and 3.x
AmplabJenkins commented on pull request #32917: URL: https://github.com/apache/spark/pull/32917#issuecomment-861333624 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44330/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32917: [SPARK-35758][DOCS] Update the document about building Spark with Hadoop for Hadoop 2.x and 3.x
SparkQA commented on pull request #32917: URL: https://github.com/apache/spark/pull/32917#issuecomment-861333598 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44330/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk closed pull request #32915: [SPARK-35764][SQL] Assign pretty names to TimestampWithoutTZType
MaxGekk closed pull request #32915: URL: https://github.com/apache/spark/pull/32915 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32917: [SPARK-35758][DOCS] Update the document about building Spark with Hadoop for Hadoop 2.x and 3.x
AmplabJenkins removed a comment on pull request #32917: URL: https://github.com/apache/spark/pull/32917#issuecomment-861333624 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44330/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #32880: [SPARK-35056][SQL] Group exception messages in execution/streaming
MaxGekk commented on pull request #32880: URL: https://github.com/apache/spark/pull/32880#issuecomment-861336109 +1, LGTM. Merging to master. Thank you, @beliefer . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk closed pull request #32880: [SPARK-35056][SQL] Group exception messages in execution/streaming
MaxGekk closed pull request #32880: URL: https://github.com/apache/spark/pull/32880 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #32880: [SPARK-35056][SQL] Group exception messages in execution/streaming
beliefer commented on pull request #32880: URL: https://github.com/apache/spark/pull/32880#issuecomment-861338050 @MaxGekk Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32821: [SPARK-35342][PYTHON] Introduce DecimalOps and make `isnull` method data-type-based
SparkQA commented on pull request #32821: URL: https://github.com/apache/spark/pull/32821#issuecomment-861338489 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44328/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32645: [SPARK-35129][SQL] Construct year-month interval column from integral fields
SparkQA commented on pull request #32645: URL: https://github.com/apache/spark/pull/32645#issuecomment-861339534 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44329/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #32916: [WIP][SPARK-35064][SQL] Group error in spark-catalyst
beliefer commented on pull request #32916: URL: https://github.com/apache/spark/pull/32916#issuecomment-861340459 @dgd-contributor Do you check all the `Exception` in package `sql/catalyst/src/main/scala/org/apache/spark/sql` ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #32916: [WIP][SPARK-35064][SQL] Group error in spark-catalyst
beliefer commented on a change in pull request #32916:
URL: https://github.com/apache/spark/pull/32916#discussion_r651613992
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala
##
@@ -146,7 +146,7 @@ private[sql] object CatalogV2Util {
Some(parent.copy(dataType = addField(parentType, newField,
add.position(
case _ =>
- throw new IllegalArgumentException(s"Not a struct:
${names.init.last}")
Review comment:
`IllegalArgumentException` is assert-like Exception, please restore it.
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala
##
@@ -146,7 +146,7 @@ private[sql] object CatalogV2Util {
Some(parent.copy(dataType = addField(parentType, newField,
add.position(
case _ =>
- throw new IllegalArgumentException(s"Not a struct:
${names.init.last}")
Review comment:
`IllegalArgumentException` is assert-like exception, please restore it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #32916: [WIP][SPARK-35064][SQL] Group error in spark-catalyst
beliefer commented on a change in pull request #32916:
URL: https://github.com/apache/spark/pull/32916#discussion_r651614405
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala
##
@@ -171,7 +171,7 @@ private[sql] object CatalogV2Util {
case update: UpdateColumnPosition =>
def updateFieldPos(struct: StructType, name: String): StructType = {
val oldField = struct.fields.find(_.name == name).getOrElse {
- throw new IllegalArgumentException("Field not found: " + name)
Review comment:
ditto
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala
##
@@ -185,7 +185,7 @@ private[sql] object CatalogV2Util {
case parentType: StructType =>
Some(parent.copy(dataType = updateFieldPos(parentType,
names.last)))
case _ =>
- throw new IllegalArgumentException(s"Not a struct:
${names.init.last}")
Review comment:
ditto
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala
##
@@ -211,7 +211,7 @@ private[sql] object CatalogV2Util {
val afterCol = position.asInstanceOf[After].column()
val fieldIndex = schema.fields.indexWhere(_.name == afterCol)
if (fieldIndex == -1) {
-throw new IllegalArgumentException("AFTER column not found: " +
afterCol)
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #32916: [WIP][SPARK-35064][SQL] Group error in spark-catalyst
beliefer commented on a change in pull request #32916:
URL: https://github.com/apache/spark/pull/32916#discussion_r651614603
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala
##
@@ -224,7 +224,7 @@ private[sql] object CatalogV2Util {
update: StructField => Option[StructField]): StructType = {
val pos = struct.getFieldIndex(fieldNames.head)
-.getOrElse(throw new IllegalArgumentException(s"Cannot find field:
${fieldNames.head}"))
Review comment:
ditto
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala
##
@@ -236,15 +236,15 @@ private[sql] object CatalogV2Util {
case (Seq("key"), map @ MapType(keyType, _, _)) =>
val updated = update(StructField("key", keyType, nullable = false))
-.getOrElse(throw new IllegalArgumentException(s"Cannot delete map
key"))
Review comment:
ditto
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala
##
@@ -236,15 +236,15 @@ private[sql] object CatalogV2Util {
case (Seq("key"), map @ MapType(keyType, _, _)) =>
val updated = update(StructField("key", keyType, nullable = false))
-.getOrElse(throw new IllegalArgumentException(s"Cannot delete map
key"))
+.getOrElse(throw QueryExecutionErrors.cannotDelete("map key"))
Some(field.copy(dataType = map.copy(keyType = updated.dataType)))
case (Seq("key", names @ _*), map @ MapType(keyStruct: StructType, _,
_)) =>
Some(field.copy(dataType = map.copy(keyType = replace(keyStruct,
names, update
case (Seq("value"), map @ MapType(_, mapValueType, isNullable)) =>
val updated = update(StructField("value", mapValueType, nullable =
isNullable))
-.getOrElse(throw new IllegalArgumentException(s"Cannot delete map
value"))
Review comment:
ditto
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala
##
@@ -254,7 +254,7 @@ private[sql] object CatalogV2Util {
case (Seq("element"), array @ ArrayType(elementType, isNullable)) =>
val updated = update(StructField("element", elementType, nullable =
isNullable))
-.getOrElse(throw new IllegalArgumentException(s"Cannot delete
array element"))
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #32916: [WIP][SPARK-35064][SQL] Group error in spark-catalyst
beliefer commented on a change in pull request #32916:
URL: https://github.com/apache/spark/pull/32916#discussion_r651614899
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala
##
@@ -263,8 +263,7 @@ private[sql] object CatalogV2Util {
Some(field.copy(dataType = array.copy(elementType =
replace(elementStruct, names, update
case (names, dataType) =>
-throw new IllegalArgumentException(
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32915: [SPARK-35764][SQL] Assign pretty names to TimestampWithoutTZType
SparkQA commented on pull request #32915: URL: https://github.com/apache/spark/pull/32915#issuecomment-861345046 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44327/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32645: [SPARK-35129][SQL] Construct year-month interval column from integral fields
SparkQA commented on pull request #32645: URL: https://github.com/apache/spark/pull/32645#issuecomment-861354363 **[Test build #139798 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139798/testReport)** for PR 32645 at commit [`b9248aa`](https://github.com/apache/spark/commit/b9248aafc830c2f5b4daee9b6a6ed8b5d572d038). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32645: [SPARK-35129][SQL] Construct year-month interval column from integral fields
SparkQA removed a comment on pull request #32645: URL: https://github.com/apache/spark/pull/32645#issuecomment-861235933 **[Test build #139798 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139798/testReport)** for PR 32645 at commit [`b9248aa`](https://github.com/apache/spark/commit/b9248aafc830c2f5b4daee9b6a6ed8b5d572d038). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
cloud-fan commented on a change in pull request #32470:
URL: https://github.com/apache/spark/pull/32470#discussion_r651629291
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
##
@@ -652,3 +652,15 @@ case object UnresolvedSeed extends LeafExpression with
Unevaluable {
override def dataType: DataType = throw new UnresolvedException("dataType")
override lazy val resolved = false
}
+
+/**
+ * An intermediate expression to hold a resolved (nested) column. Some rules
may need to undo the
+ * column resolution and use this expression to keep the original column name.
+ */
+case class TempResolvedColumn(child: Expression, nameParts: Seq[String])
extends UnaryExpression
Review comment:
Actually, we don't require it to be a named expression, so it's more
robust to keep the type as `Expression`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32909: [WIP][SPARK-35680][SQL] Add fields to `YearMonthIntervalType`
SparkQA commented on pull request #32909: URL: https://github.com/apache/spark/pull/32909#issuecomment-861361771 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44331/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] peter-toth commented on a change in pull request #32885: [SPARK-35742][SQL] Expression.semanticEquals should be symmetrical
peter-toth commented on a change in pull request #32885:
URL: https://github.com/apache/spark/pull/32885#discussion_r651638388
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
##
@@ -73,7 +73,9 @@ case class HashAggregateExec(
// This is for testing. We force TungstenAggregationIterator to fall back to
the unsafe row hash
// map and/or the sort-based aggregation once it has processed a given
number of input rows.
private val testFallbackStartsAt: Option[(Int, Int)] = {
-sqlContext.getConf("spark.sql.TungstenAggregate.testFallbackStartsAt",
null) match {
+Option(sqlContext).map { sc =>
Review comment:
Sorry, for the late comment @cloud-fan. I think I've run into this issue
before:
https://github.com/apache/spark/pull/28885/files#diff-9b62cef6bfdeb6c802bb120c7a724a974d5067a69585285bebb64c48603f8d6fR105-R108.
The point is that there might be other nodes where canonicalization on
executor side can cause issues. `SortExec.enableRadixSort` is the other one I
found.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32645: [SPARK-35129][SQL] Construct year-month interval column from integral fields
AmplabJenkins commented on pull request #32645: URL: https://github.com/apache/spark/pull/32645#issuecomment-861363317 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32821: [SPARK-35342][PYTHON] Introduce DecimalOps and make `isnull` method data-type-based
AmplabJenkins commented on pull request #32821: URL: https://github.com/apache/spark/pull/32821#issuecomment-861363319 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44328/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32915: [SPARK-35764][SQL] Assign pretty names to TimestampWithoutTZType
AmplabJenkins commented on pull request #32915: URL: https://github.com/apache/spark/pull/32915#issuecomment-861363313 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44327/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32645: [SPARK-35129][SQL] Construct year-month interval column from integral fields
AmplabJenkins removed a comment on pull request #32645: URL: https://github.com/apache/spark/pull/32645#issuecomment-861363316 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32915: [SPARK-35764][SQL] Assign pretty names to TimestampWithoutTZType
AmplabJenkins removed a comment on pull request #32915: URL: https://github.com/apache/spark/pull/32915#issuecomment-861363313 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44327/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32821: [SPARK-35342][PYTHON] Introduce DecimalOps and make `isnull` method data-type-based
AmplabJenkins removed a comment on pull request #32821: URL: https://github.com/apache/spark/pull/32821#issuecomment-861363319 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44328/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] peter-toth commented on a change in pull request #32885: [SPARK-35742][SQL] Expression.semanticEquals should be symmetrical
peter-toth commented on a change in pull request #32885:
URL: https://github.com/apache/spark/pull/32885#discussion_r651638388
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
##
@@ -73,7 +73,9 @@ case class HashAggregateExec(
// This is for testing. We force TungstenAggregationIterator to fall back to
the unsafe row hash
// map and/or the sort-based aggregation once it has processed a given
number of input rows.
private val testFallbackStartsAt: Option[(Int, Int)] = {
-sqlContext.getConf("spark.sql.TungstenAggregate.testFallbackStartsAt",
null) match {
+Option(sqlContext).map { sc =>
Review comment:
Sorry for the late comment @cloud-fan, but I think I've run into this
issue before:
https://github.com/apache/spark/pull/28885/files#diff-9b62cef6bfdeb6c802bb120c7a724a974d5067a69585285bebb64c48603f8d6fR105-R108.
The point is that there might be other nodes where canonicalization on
executor side can cause issues. `SortExec.enableRadixSort` is the other one I
found.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
SparkQA commented on pull request #32470: URL: https://github.com/apache/spark/pull/32470#issuecomment-861365825 **[Test build #139808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139808/testReport)** for PR 32470 at commit [`b362a09`](https://github.com/apache/spark/commit/b362a098978be65ab1fc033fe0213a78a467b6ae). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32912: [SPARK-35429][CORE] Remove commons-httpclient due to EOL and CVEs
SparkQA commented on pull request #32912: URL: https://github.com/apache/spark/pull/32912#issuecomment-861367891 **[Test build #139797 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139797/testReport)** for PR 32912 at commit [`3c0eb47`](https://github.com/apache/spark/commit/3c0eb4763bf74a67eabb21fc02565790d9505ee0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32845: [SPARK-35691][CORE] addFile/addJar/addDirectory should put CanonicalFile
SparkQA commented on pull request #32845: URL: https://github.com/apache/spark/pull/32845#issuecomment-861367881 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44333/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32912: [SPARK-35429][CORE] Remove commons-httpclient due to EOL and CVEs
AmplabJenkins commented on pull request #32912: URL: https://github.com/apache/spark/pull/32912#issuecomment-861369339 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139797/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32881: [SPARK-33298][CORE] Decouple file naming from FileCommitProtocol
SparkQA commented on pull request #32881: URL: https://github.com/apache/spark/pull/32881#issuecomment-861370670 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44332/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32912: [SPARK-35429][CORE] Remove commons-httpclient due to EOL and CVEs
AmplabJenkins removed a comment on pull request #32912: URL: https://github.com/apache/spark/pull/32912#issuecomment-861369339 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139797/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32912: [SPARK-35429][CORE] Remove commons-httpclient due to EOL and CVEs
SparkQA removed a comment on pull request #32912: URL: https://github.com/apache/spark/pull/32912#issuecomment-861235546 **[Test build #139797 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139797/testReport)** for PR 32912 at commit [`3c0eb47`](https://github.com/apache/spark/commit/3c0eb4763bf74a67eabb21fc02565790d9505ee0). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32742: [SPARK-35608][SQL] Support AQE optimizer side transformUpWithPruning
AmplabJenkins commented on pull request #32742: URL: https://github.com/apache/spark/pull/32742#issuecomment-861375308 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44334/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
