[GitHub] [spark] AmplabJenkins removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
AmplabJenkins removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632418445 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27593/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
AmplabJenkins removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632418438 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632418419 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/27593/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
AmplabJenkins commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632418438 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28598: [SPARK-31785][SQL][TESTS] Add a helper function to test all parquet readers
HyukjinKwon commented on a change in pull request #28598:
URL: https://github.com/apache/spark/pull/28598#discussion_r428987921
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTest.scala
##
@@ -162,4 +162,36 @@ private[sql] trait ParquetTest extends
FileBasedDataSourceTest {
protected def getResourceParquetFilePath(name: String): String = {
Thread.currentThread().getContextClassLoader.getResource(name).toString
}
+
+ def withParquetReaderFlags[T](vectorizedOSS: Boolean)
+(code: => T): T = {
+val sqlConfs = Seq(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key ->
vectorizedOSS.toString)
+var ret = null.asInstanceOf[T]
+withSQLConf(sqlConfs: _*) { ret = code }
+ret
+ }
+
+ def withParquetReader[T](reader: String)(code: => T): T = reader match {
+case "parquet-mr" => withParquetReaderFlags(false)(code)
+case "vectorized-oss" => withParquetReaderFlags(true)(code)
+case unknown =>
+ // scalastyle:off throwerror
+ throw new NotImplementedError(s"Unsupported Parquet reader '$unknown'.")
+ // scalastyle:on throwerror
+ }
+
+ def withParquetReaders(readers: String*)(code: => Unit): Unit = for (reader
<- readers) {
+// scalastyle:off
+println(s"with $reader reader")
+// scalastyle:on
+withParquetReader(reader)(code)
+ }
+
+ def withOssParquetReaders(code: => Unit): Unit = {
+withParquetReaders("parquet-mr", "vectorized-oss")(code)
+ }
+
+ def withAllParquetReaders(code: => Unit): Unit = {
Review comment:
I would say code- > func but no big deal.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column
AmplabJenkins commented on pull request #28560: URL: https://github.com/apache/spark/pull/28560#issuecomment-632417641 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28363: [SPARK-27188][SS] FileStreamSink: provide a new option to have retention on output files
AmplabJenkins commented on pull request #28363: URL: https://github.com/apache/spark/pull/28363#issuecomment-632417665 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column
AmplabJenkins removed a comment on pull request #28560: URL: https://github.com/apache/spark/pull/28560#issuecomment-632417641 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28363: [SPARK-27188][SS] FileStreamSink: provide a new option to have retention on output files
AmplabJenkins removed a comment on pull request #28363: URL: https://github.com/apache/spark/pull/28363#issuecomment-632417665 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28600: [SPARK-31761][SQL] cast integer to Long to avoid IntegerOverflow for IntegralDivide operator
maropu commented on a change in pull request #28600:
URL: https://github.com/apache/spark/pull/28600#discussion_r428987485
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
##
@@ -337,11 +340,19 @@ trait DivModLike extends BinaryArithmetic {
} else {
s"${eval2.value} == 0"
}
-val javaType = CodeGenerator.javaType(dataType)
+val isIntegralDiv = this.isInstanceOf[IntegralDivide]
+// From SPARK-16323 IntegralDivision returns Long data type
+val javaType = if (isIntegralDiv) JAVA_LONG else
CodeGenerator.javaType(dataType)
+val operandJavaType = if (isIntegralDiv) operandsDataType match {
+ case _: IntegerType => JAVA_LONG
Review comment:
Have you tried to fix this issue in `TypeCoercion`? It seems that is a
more suitable place to fix it though.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28363: [SPARK-27188][SS] FileStreamSink: provide a new option to have retention on output files
SparkQA commented on pull request #28363: URL: https://github.com/apache/spark/pull/28363#issuecomment-632417393 **[Test build #122955 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122955/testReport)** for PR 28363 at commit [`d2e7ab3`](https://github.com/apache/spark/commit/d2e7ab35ac44dd461373e51a015b7a12e2c3af6e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column
SparkQA commented on pull request #28560: URL: https://github.com/apache/spark/pull/28560#issuecomment-632417352 **[Test build #122954 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122954/testReport)** for PR 28560 at commit [`b2e56d5`](https://github.com/apache/spark/commit/b2e56d55fe1dda39a47ba654e2b60a81fea7e492). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
dongjoon-hyun commented on pull request #27803: URL: https://github.com/apache/spark/pull/27803#issuecomment-632416179 Sorry for being late, @maropu . +1 for revisit this later. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28485: [SPARK-31642] Add Pagination Support for Structured Streaming Page
AmplabJenkins removed a comment on pull request #28485: URL: https://github.com/apache/spark/pull/28485#issuecomment-632415396 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28485: [SPARK-31642] Add Pagination Support for Structured Streaming Page
AmplabJenkins commented on pull request #28485: URL: https://github.com/apache/spark/pull/28485#issuecomment-632415396 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28485: [SPARK-31642] Add Pagination Support for Structured Streaming Page
SparkQA removed a comment on pull request #28485: URL: https://github.com/apache/spark/pull/28485#issuecomment-632293771 **[Test build #122941 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122941/testReport)** for PR 28485 at commit [`088c551`](https://github.com/apache/spark/commit/088c55186919e9c671942694c2f16933cc806a9d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28485: [SPARK-31642] Add Pagination Support for Structured Streaming Page
SparkQA commented on pull request #28485: URL: https://github.com/apache/spark/pull/28485#issuecomment-632414682 **[Test build #122941 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122941/testReport)** for PR 28485 at commit [`088c551`](https://github.com/apache/spark/commit/088c55186919e9c671942694c2f16933cc806a9d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 commented on a change in pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions
TJX2014 commented on a change in pull request #28534:
URL: https://github.com/apache/spark/pull/28534#discussion_r428984616
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
##
@@ -401,6 +401,83 @@ case class DayOfYear(child: Expression) extends
UnaryExpression with ImplicitCas
}
}
+abstract class NumberToTimestampBase extends UnaryExpression
+ with ExpectsInputTypes {
+
+ protected def upScaleFactor: Long
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(IntegralType)
+
+ override def dataType: DataType = TimestampType
+
+ override def nullSafeEval(input: Any): Any = {
+Math.multiplyExact(input.asInstanceOf[Number].longValue(), upScaleFactor)
+ }
+
+ override protected def doGenCode(ctx: CodegenContext, ev: ExprCode):
ExprCode = {
+if (upScaleFactor == 1) {
+ defineCodeGen(ctx, ev, c => c)
+} else {
+ defineCodeGen(ctx, ev, c => s"java.lang.Math.multiplyExact($c,
$upScaleFactor)")
Review comment:
Yes, otherwise `java.lang.Math.multiplyExact(1230219000, 1000)` will
cause the integer overflow.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632412704 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/27593/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu closed pull request #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
maropu closed pull request #27803: URL: https://github.com/apache/spark/pull/27803 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #26503: [SPARK-29880][CORE][YARN] Handle submit exception when submit to federation cluster
github-actions[bot] commented on pull request #26503: URL: https://github.com/apache/spark/pull/26503#issuecomment-632409128 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #27231: [SPARK-28478][SQL] Remove redundant null checks
github-actions[bot] closed pull request #27231: URL: https://github.com/apache/spark/pull/27231 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #27213: [SPARK-30516][SQL] involve partition filters in the statistic estimation of FileScan
github-actions[bot] commented on pull request #27213: URL: https://github.com/apache/spark/pull/27213#issuecomment-632409120 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #27436: [SPARK-30705][SQL] Improve CaseWhen sub-expression equality
github-actions[bot] closed pull request #27436: URL: https://github.com/apache/spark/pull/27436 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28604: [SPARK-29854][SQL][TESTS] Add tests to check lpad/rpad throw an exception for invalid length input
AmplabJenkins removed a comment on pull request #28604: URL: https://github.com/apache/spark/pull/28604#issuecomment-632409285 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28604: [SPARK-29854][SQL][TESTS] Add tests to check lpad/rpad throw an exception for invalid length input
AmplabJenkins commented on pull request #28604: URL: https://github.com/apache/spark/pull/28604#issuecomment-632409285 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28604: [SPARK-29854][SQL][TESTS] Add tests to check lpad/rpad throw an exception for invalid length input
SparkQA commented on pull request #28604: URL: https://github.com/apache/spark/pull/28604#issuecomment-632408912 **[Test build #122953 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122953/testReport)** for PR 28604 at commit [`4615923`](https://github.com/apache/spark/commit/4615923e35f5f603cb5be9fc18c4a4228867bace). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on pull request #28604: [SPARK-29854][SQL][TESTS] Add tests to check lpad/rpad throw an exception for invalid length input
maropu commented on pull request #28604: URL: https://github.com/apache/spark/pull/28604#issuecomment-632408836 @cloud-fan @gengliangwang I've checked the proposal in https://github.com/apache/spark/pull/27024 again and I think it's been already fixed in the master. So, we could close SPARK-29854 now. Just in case, I opened this PR to add trivial tests to check that. But, closing this PR looks okay if not needed. Could you check this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu opened a new pull request #28604: [SPARK-29854][SQL][TESTS] Add tests to check lpad/rpad throw an exception for invalid length input
maropu opened a new pull request #28604: URL: https://github.com/apache/spark/pull/28604 ### What changes were proposed in this pull request? This PR intends to add trivial tests to check https://github.com/apache/spark/pull/27024 has already been fixed in the master. Closes #27024 ### Why are the changes needed? For test coverage. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column
viirya commented on a change in pull request #28560:
URL: https://github.com/apache/spark/pull/28560#discussion_r428975902
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala
##
@@ -48,7 +53,10 @@ object NestedColumnAliasing {
case Project(projectList, child) =>
Project(
getNewProjectList(projectList, nestedFieldToAlias),
-replaceChildrenWithAliases(child, attrToAliases))
+replaceChildrenWithAliases(child, nestedFieldToAlias, attrToAliases))
+
+case other =>
Review comment:
Let me leave some comments, running `canPruneOn` twice seems redundant.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28603: [SPARK-31788][CORE] Fix UnionRDD of PairRDDs
AmplabJenkins commented on pull request #28603: URL: https://github.com/apache/spark/pull/28603#issuecomment-632402291 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28603: [SPARK-31788][CORE] Fix UnionRDD of PairRDDs
AmplabJenkins removed a comment on pull request #28603: URL: https://github.com/apache/spark/pull/28603#issuecomment-632401913 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28603: [SPARK-31788][CORE] Fix UnionRDD of PairRDDs
AmplabJenkins commented on pull request #28603: URL: https://github.com/apache/spark/pull/28603#issuecomment-632401913 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] redsanket opened a new pull request #28603: [SPARK-31788][CORE] Fix UnionRDD of PairRDDs
redsanket opened a new pull request #28603: URL: https://github.com/apache/spark/pull/28603 ### What changes were proposed in this pull request? UnionRDD of PairRDDs causing a bug. The fix is to check for instance type before proceeding ### Why are the changes needed? We can reproduce via SparkSession available as 'spark'. >>> rdd1 = sc.parallelize([1,2,3,4,5]) >>> rdd2 = sc.parallelize([6,7,8,9,10]) >>> pairRDD1 = rdd1.zip(rdd2) >>> unionRDD1 = sc.union([pairRDD1, pairRDD1]) Traceback (most recent call last): File "", line 1, in File "/home/gs/spark/latest/python/pyspark/context.py", line 870, in union jrdds[i] = rdds[i]._jrdd File "/home/gs/spark/latest/python/lib/py4j-0.10.9-src.zip/py4j/java_collections.py", line 238, in setitem File "/home/gs/spark/latest/python/lib/py4j-0.10.9-src.zip/py4j/java_collections.py", line 221, in __set_item File "/home/gs/spark/latest/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 332, in get_return_value py4j.protocol.Py4JError: An error occurred while calling None.None. Trace: py4j.Py4JException: Cannot convert org.apache.spark.api.java.JavaPairRDD to org.apache.spark.api.java.JavaRDD at py4j.commands.ArrayCommand.convertArgument(ArrayCommand.java:166) at py4j.commands.ArrayCommand.setArray(ArrayCommand.java:144) at py4j.commands.ArrayCommand.execute(ArrayCommand.java:97) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Tested with the reproduced piece of code above manually After the patch >>> rdd2 = sc.parallelize([6,7,8,9,10]) >>> pairRDD1 = rdd1.zip(rdd2) >>> unionRDD1 = sc.union([pairRDD1, pairRDD1]) >>> unionRDD1.collect() [(1, 6), (2, 7), (3, 8), (4, 9), (5, 10), (1, 6), (2, 7), (3, 8), (4, 9), (5, 10)] This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
AmplabJenkins removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632399668 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27589/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632399864 **[Test build #122952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122952/testReport)** for PR 28601 at commit [`4327940`](https://github.com/apache/spark/commit/43279402fbd10ee1cfecbc01d3fdec46f7cbc7d2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
AmplabJenkins removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632399661 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28553: [SPARK-31734][ML][PySpark] Add weight support in ClusteringEvaluator
AmplabJenkins removed a comment on pull request #28553: URL: https://github.com/apache/spark/pull/28553#issuecomment-632399475 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28553: [SPARK-31734][ML][PySpark] Add weight support in ClusteringEvaluator
AmplabJenkins commented on pull request #28553: URL: https://github.com/apache/spark/pull/28553#issuecomment-632399475 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632399647 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/27589/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
AmplabJenkins commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632399661 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28553: [SPARK-31734][ML][PySpark] Add weight support in ClusteringEvaluator
SparkQA removed a comment on pull request #28553: URL: https://github.com/apache/spark/pull/28553#issuecomment-632375761 **[Test build #122945 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122945/testReport)** for PR 28553 at commit [`06a04a1`](https://github.com/apache/spark/commit/06a04a1c7e2f73c0d1a9288e7200ee7ed8a51ea7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28553: [SPARK-31734][ML][PySpark] Add weight support in ClusteringEvaluator
SparkQA commented on pull request #28553: URL: https://github.com/apache/spark/pull/28553#issuecomment-632398947 **[Test build #122945 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122945/testReport)** for PR 28553 at commit [`06a04a1`](https://github.com/apache/spark/commit/06a04a1c7e2f73c0d1a9288e7200ee7ed8a51ea7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column
maropu commented on a change in pull request #28560:
URL: https://github.com/apache/spark/pull/28560#discussion_r428969502
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala
##
@@ -48,7 +53,10 @@ object NestedColumnAliasing {
case Project(projectList, child) =>
Project(
getNewProjectList(projectList, nestedFieldToAlias),
-replaceChildrenWithAliases(child, attrToAliases))
+replaceChildrenWithAliases(child, nestedFieldToAlias, attrToAliases))
+
+case other =>
Review comment:
Ah, I see. Thanks for the check. If so, it might be better to explicitly
filiter them here for readablity like `case other if canPruneOn(plan)`, or
leave some comments about that. Both is okay to me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632398364 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/27589/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
dongjoon-hyun commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632397839 Since there is a revert on master branch (https://github.com/apache/spark/commit/db5e5fce68f7e3a724eb4516908c4ba9786b0db3), I'll rebase once more to make it sure. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
AmplabJenkins removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632395037 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27588/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632395021 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/27588/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
AmplabJenkins removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632395035 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
AmplabJenkins commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632395035 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column
viirya commented on a change in pull request #28560:
URL: https://github.com/apache/spark/pull/28560#discussion_r428965331
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasingSuite.scala
##
@@ -341,6 +341,78 @@ class NestedColumnAliasingSuite extends SchemaPruningTest {
.analyze
comparePlans(optimized, expected)
}
+
+ test("Nested field pruning for Aggregate") {
+val query1 =
contact.groupBy($"id")(first($"name.first").as("first")).analyze
Review comment:
ok, let me add some tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column
viirya commented on a change in pull request #28560:
URL: https://github.com/apache/spark/pull/28560#discussion_r428965232
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala
##
@@ -48,7 +53,10 @@ object NestedColumnAliasing {
case Project(projectList, child) =>
Project(
getNewProjectList(projectList, nestedFieldToAlias),
-replaceChildrenWithAliases(child, attrToAliases))
+replaceChildrenWithAliases(child, nestedFieldToAlias, attrToAliases))
+
+case other =>
Review comment:
Yea, `canPruneOn` decides what operator will be pruned on.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632393616 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/27588/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
AmplabJenkins removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632392163 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27586/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
AmplabJenkins removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632392157 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632392143 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/27586/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
AmplabJenkins commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632392157 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28579: [SPARK-31757][CORE] Improve HistoryServerDiskManager.updateAccessTime()
AmplabJenkins removed a comment on pull request #28579: URL: https://github.com/apache/spark/pull/28579#issuecomment-632391005 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28596: [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call"
AmplabJenkins removed a comment on pull request #28596: URL: https://github.com/apache/spark/pull/28596#issuecomment-632390928 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28579: [SPARK-31757][CORE] Improve HistoryServerDiskManager.updateAccessTime()
AmplabJenkins commented on pull request #28579: URL: https://github.com/apache/spark/pull/28579#issuecomment-632391005 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28596: [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call"
AmplabJenkins commented on pull request #28596: URL: https://github.com/apache/spark/pull/28596#issuecomment-632390928 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632390734 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/27586/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28579: [SPARK-31757][CORE] Improve HistoryServerDiskManager.updateAccessTime()
SparkQA commented on pull request #28579: URL: https://github.com/apache/spark/pull/28579#issuecomment-632390622 **[Test build #122951 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122951/testReport)** for PR 28579 at commit [`f212f33`](https://github.com/apache/spark/commit/f212f33e9fe7f38e490f35f61e1e67fd466e5949). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28585: [SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0
SparkQA commented on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-632390619 **[Test build #122950 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122950/testReport)** for PR 28585 at commit [`f53735e`](https://github.com/apache/spark/commit/f53735e2e52a8cf3d70f162c3317c10383ae1aff). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28596: [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call"
SparkQA commented on pull request #28596: URL: https://github.com/apache/spark/pull/28596#issuecomment-632390624 **[Test build #122949 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122949/testReport)** for PR 28596 at commit [`2be2ca7`](https://github.com/apache/spark/commit/2be2ca728640ff68f4c9ff271d22ae67cf6db44f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on pull request #28596: [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call"
jiangxb1987 commented on pull request #28596: URL: https://github.com/apache/spark/pull/28596#issuecomment-632390331 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on pull request #28579: [SPARK-31757][CORE] Improve HistoryServerDiskManager.updateAccessTime()
jiangxb1987 commented on pull request #28579: URL: https://github.com/apache/spark/pull/28579#issuecomment-632389963 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28585: [SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0
AmplabJenkins removed a comment on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-632388650 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28585: [SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0
AmplabJenkins commented on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-632388650 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28602: Revert "[SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0"
SparkQA commented on pull request #28602: URL: https://github.com/apache/spark/pull/28602#issuecomment-632388260 **[Test build #122948 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122948/testReport)** for PR 28602 at commit [`4a72dff`](https://github.com/apache/spark/commit/4a72dffbc71723196874ee3cc68a26d1a0e8064b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang closed pull request #28602: Revert "[SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0"
gengliangwang closed pull request #28602: URL: https://github.com/apache/spark/pull/28602 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #28602: Revert "[SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0"
gengliangwang commented on pull request #28602: URL: https://github.com/apache/spark/pull/28602#issuecomment-632387870 Since this is the latest commit, I am going to merge it now. cc @sarutak This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #28602: Revert "[SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0"
gengliangwang commented on pull request #28602: URL: https://github.com/apache/spark/pull/28602#issuecomment-632387698 See the discussion in https://github.com/apache/spark/pull/28585#issuecomment-632386519 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang opened a new pull request #28602: Revert "[SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0"
gengliangwang opened a new pull request #28602: URL: https://github.com/apache/spark/pull/28602 This reverts commit 92877c4ef2ad113c156b7d9c359f396187c78fa3. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #28585: [SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0
gengliangwang commented on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-632387056 @sarutak sorry about that, let me do the revert This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #28585: [SPARK-31765][WEBUI] Upgrade HtmlUnit >= 2.37.0
sarutak commented on pull request #28585: URL: https://github.com/apache/spark/pull/28585#issuecomment-632386519 @gengliangwang This PR doesn't pass test. As I mentioned [here](https://github.com/apache/spark/pull/28585#issuecomment-631255845), #28578 should be merged first. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632386056 **[Test build #122947 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122947/testReport)** for PR 28601 at commit [`ed9ae40`](https://github.com/apache/spark/commit/ed9ae408c1ac86969239b8be7181a7602ee5cc5a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27366: [SPARK-30648][SQL] Support filters pushdown in JSON datasource
AmplabJenkins removed a comment on pull request #27366: URL: https://github.com/apache/spark/pull/27366#issuecomment-632383424 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27366: [SPARK-30648][SQL] Support filters pushdown in JSON datasource
AmplabJenkins commented on pull request #27366: URL: https://github.com/apache/spark/pull/27366#issuecomment-632383424 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #27366: [SPARK-30648][SQL] Support filters pushdown in JSON datasource
SparkQA removed a comment on pull request #27366: URL: https://github.com/apache/spark/pull/27366#issuecomment-632243504 **[Test build #122940 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122940/testReport)** for PR 27366 at commit [`262e3c7`](https://github.com/apache/spark/commit/262e3c7119c2ad890be184ed182aa156817764fd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27366: [SPARK-30648][SQL] Support filters pushdown in JSON datasource
SparkQA commented on pull request #27366: URL: https://github.com/apache/spark/pull/27366#issuecomment-632382607 **[Test build #122940 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122940/testReport)** for PR 27366 at commit [`262e3c7`](https://github.com/apache/spark/commit/262e3c7119c2ad890be184ed182aa156817764fd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.10.1
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632381162 **[Test build #122946 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122946/testReport)** for PR 28601 at commit [`3b27bdc`](https://github.com/apache/spark/commit/3b27bdc2766d29f515058257a80bcd378db5e29f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #28595: [SPARK-31781][ML][PySpark] Move param k (number of clusters) to shared params
huaxingao commented on pull request #28595: URL: https://github.com/apache/spark/pull/28595#issuecomment-632379756 I think this is an okay abstraction because ```k``` corresponds to number of clusters. Selecting the number of clusters is similar for the five different clustering algorithms. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28553: [SPARK-31734][ML][PySpark] Add weight support in ClusteringEvaluator
AmplabJenkins removed a comment on pull request #28553: URL: https://github.com/apache/spark/pull/28553#issuecomment-632376169 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28553: [SPARK-31734][ML][PySpark] Add weight support in ClusteringEvaluator
AmplabJenkins commented on pull request #28553: URL: https://github.com/apache/spark/pull/28553#issuecomment-632376169 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28553: [SPARK-31734][ML][PySpark] Add weight support in ClusteringEvaluator
SparkQA commented on pull request #28553: URL: https://github.com/apache/spark/pull/28553#issuecomment-632375761 **[Test build #122945 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122945/testReport)** for PR 28553 at commit [`06a04a1`](https://github.com/apache/spark/commit/06a04a1c7e2f73c0d1a9288e7200ee7ed8a51ea7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.10.0
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632375746 **[Test build #122944 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122944/testReport)** for PR 28601 at commit [`b8fca65`](https://github.com/apache/spark/commit/b8fca65e4201fe44ab0632dc3385ef55b58d68e7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
AmplabJenkins removed a comment on pull request #27617: URL: https://github.com/apache/spark/pull/27617#issuecomment-632375724 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
AmplabJenkins commented on pull request #27617: URL: https://github.com/apache/spark/pull/27617#issuecomment-632375724 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.10.0
dongjoon-hyun opened a new pull request #28601: URL: https://github.com/apache/spark/pull/28601 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #28553: [SPARK-31734][ML][PySpark] Add weight support in ClusteringEvaluator
huaxingao commented on pull request #28553: URL: https://github.com/apache/spark/pull/28553#issuecomment-632375218 Added the check to make sure weights are all positive. We probably need to do this for everything that has instance weight support. I will open a ticket to fix all of them? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
SparkQA commented on pull request #27617: URL: https://github.com/apache/spark/pull/27617#issuecomment-632375108 **[Test build #122939 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122939/testReport)** for PR 27617 at commit [`1eb480a`](https://github.com/apache/spark/commit/1eb480ae0fc81924f26235b1e81f7a9b2b4ae9a0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
SparkQA removed a comment on pull request #27617: URL: https://github.com/apache/spark/pull/27617#issuecomment-632243446 **[Test build #122939 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122939/testReport)** for PR 27617 at commit [`1eb480a`](https://github.com/apache/spark/commit/1eb480ae0fc81924f26235b1e81f7a9b2b4ae9a0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28598: [SPARK-31785][SQL][TESTS] Add a helper function to test all parquet readers
AmplabJenkins removed a comment on pull request #28598: URL: https://github.com/apache/spark/pull/28598#issuecomment-632359781 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28598: [SPARK-31785][SQL][TESTS] Add a helper function to test all parquet readers
AmplabJenkins commented on pull request #28598: URL: https://github.com/apache/spark/pull/28598#issuecomment-632359781 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28598: [SPARK-31785][SQL][TESTS] Add a helper function to test all parquet readers
SparkQA removed a comment on pull request #28598: URL: https://github.com/apache/spark/pull/28598#issuecomment-63711 **[Test build #122936 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122936/testReport)** for PR 28598 at commit [`5ffc1e4`](https://github.com/apache/spark/commit/5ffc1e43c6617e33b9750eb3c9032421c4d1cdca). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28598: [SPARK-31785][SQL][TESTS] Add a helper function to test all parquet readers
SparkQA commented on pull request #28598: URL: https://github.com/apache/spark/pull/28598#issuecomment-632358944 **[Test build #122936 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122936/testReport)** for PR 28598 at commit [`5ffc1e4`](https://github.com/apache/spark/commit/5ffc1e43c6617e33b9750eb3c9032421c4d1cdca). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28544: [SPARK-31387][SQL] Handle unknown operation/session ID in HiveThriftServer2Listener
dongjoon-hyun commented on pull request #28544: URL: https://github.com/apache/spark/pull/28544#issuecomment-632352326 It seems that this causes consistent failures in SBT in branch-3.0 with Hive 2.7. - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-3.0-test-sbt-hadoop-2.7-hive-2.3/526/ This is the opposite situation. Sorry again. My bad. I had a wrong assumption that this PR has no problem in SBT because it passed SBT before. For now, the new failure is only at 3.0. To stabilize branch-3.0, I'll revert this from `branch-3.0` only. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] kuhushukla commented on a change in pull request #28600: [SPARK-31761][SQL] cast integer to Long to avoid IntegerOverflow for IntegralDivide operator
kuhushukla commented on a change in pull request #28600:
URL: https://github.com/apache/spark/pull/28600#discussion_r428917111
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
##
@@ -423,13 +434,21 @@ case class IntegralDivide(
private lazy val div: (Any, Any) => Any = {
val integral = left.dataType match {
+ // if it is of Integer type than cast it to Long
Review comment:
Nit: than -> then
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28599: [SPARK-31787][K8S][TESTS][2.4] Fix Minikube.getIfNewMinikubeStatus to understand 1.5+
dongjoon-hyun commented on pull request #28599: URL: https://github.com/apache/spark/pull/28599#issuecomment-632347920 Merged to branch-2.4. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
