[GitHub] [spark] iRakson commented on pull request #28512: [SPARK-31688][WEBUI] Refactor Pagination framework
iRakson commented on pull request #28512: URL: https://github.com/apache/spark/pull/28512#issuecomment-632258679 Thank You. @srowen @sarutak This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen closed pull request #28512: [SPARK-31688][WEBUI] Refactor Pagination framework
srowen closed pull request #28512: URL: https://github.com/apache/spark/pull/28512 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on pull request #28512: [SPARK-31688][WEBUI] Refactor Pagination framework
srowen commented on pull request #28512: URL: https://github.com/apache/spark/pull/28512#issuecomment-632255678 Merged to master This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #28553: [SPARK-31734][ML][PySpark] Add weight support in ClusteringEvaluator
huaxingao commented on pull request #28553: URL: https://github.com/apache/spark/pull/28553#issuecomment-632252469 cc @srowen This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #28595: [SPARK-31781][ML][PySpark] Move param k (number of clusters) to shared params
huaxingao commented on pull request #28595: URL: https://github.com/apache/spark/pull/28595#issuecomment-632252188 cc @srowen This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #28595: [SPARK-31781][ML][PySpark] Move param k (number of clusters) to shared params
huaxingao commented on pull request #28595: URL: https://github.com/apache/spark/pull/28595#issuecomment-632251801 Actually the param in ALS is called ```rank``` instead of ```k```. I am a little hesitant to reuse ```k``` in RankingEvaluator because it has a totally different meaning. I can override it, though. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #28590: [SPARK-31768][ML] add getMetrics in Evaluators
huaxingao commented on pull request #28590: URL: https://github.com/apache/spark/pull/28590#issuecomment-632246256 > Is there any way to recompute another metric in evaluator without involving mllib.metrics? It might be better to create a ml wrapper for mllib.metrics instead of using it directly, something like the approach in ```LogisticRegressionSummary```. This wrapper can be used both by evaluator and summary. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28576: [SPARK-31755][SQL] allow missing year/hour when parsing date/timestamp string
AmplabJenkins removed a comment on pull request #28576: URL: https://github.com/apache/spark/pull/28576#issuecomment-632244984 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28576: [SPARK-31755][SQL] allow missing year/hour when parsing date/timestamp string
AmplabJenkins commented on pull request #28576: URL: https://github.com/apache/spark/pull/28576#issuecomment-632244984 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'
AmplabJenkins removed a comment on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632244007 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27366: [SPARK-30648][SQL] Support filters pushdown in JSON datasource
AmplabJenkins removed a comment on pull request #27366: URL: https://github.com/apache/spark/pull/27366#issuecomment-632244056 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
AmplabJenkins removed a comment on pull request #27617: URL: https://github.com/apache/spark/pull/27617#issuecomment-632244083 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28576: [SPARK-31755][SQL] allow missing year/hour when parsing date/timestamp string
SparkQA removed a comment on pull request #28576: URL: https://github.com/apache/spark/pull/28576#issuecomment-632068610 **[Test build #122932 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122932/testReport)** for PR 28576 at commit [`161223b`](https://github.com/apache/spark/commit/161223b5de92064a4cc3672702e09f868bd68870). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27366: [SPARK-30648][SQL] Support filters pushdown in JSON datasource
AmplabJenkins commented on pull request #27366: URL: https://github.com/apache/spark/pull/27366#issuecomment-632244056 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'
AmplabJenkins commented on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632244007 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28576: [SPARK-31755][SQL] allow missing year/hour when parsing date/timestamp string
SparkQA commented on pull request #28576: URL: https://github.com/apache/spark/pull/28576#issuecomment-632243655 **[Test build #122932 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122932/testReport)** for PR 28576 at commit [`161223b`](https://github.com/apache/spark/commit/161223b5de92064a4cc3672702e09f868bd68870). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
AmplabJenkins commented on pull request #27617: URL: https://github.com/apache/spark/pull/27617#issuecomment-632244075 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'
SparkQA commented on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632243456 **[Test build #122938 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122938/testReport)** for PR 28592 at commit [`b2abeeb`](https://github.com/apache/spark/commit/b2abeebfff20cfbc0a7f08fb472509bdc1dc9b55). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27617: [SPARK-30865][SQL] Refactor DateTimeUtils
SparkQA commented on pull request #27617: URL: https://github.com/apache/spark/pull/27617#issuecomment-632243446 **[Test build #122939 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122939/testReport)** for PR 27617 at commit [`1eb480a`](https://github.com/apache/spark/commit/1eb480ae0fc81924f26235b1e81f7a9b2b4ae9a0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27366: [SPARK-30648][SQL] Support filters pushdown in JSON datasource
SparkQA commented on pull request #27366: URL: https://github.com/apache/spark/pull/27366#issuecomment-632243504 **[Test build #122940 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122940/testReport)** for PR 27366 at commit [`262e3c7`](https://github.com/apache/spark/commit/262e3c7119c2ad890be184ed182aa156817764fd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'
AmplabJenkins removed a comment on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632239959 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'
AmplabJenkins commented on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632239959 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'
SparkQA commented on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632239244 **[Test build #122937 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122937/testReport)** for PR 28592 at commit [`c877ac5`](https://github.com/apache/spark/commit/c877ac541beb80b3abf57658c4c5564c7412d0bf). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28523: [SPARK-31706][SQL] add back the support of streaming update mode
cloud-fan commented on pull request #28523: URL: https://github.com/apache/spark/pull/28523#issuecomment-632230688 This PR (including code diff and PR description) is verbatim as it was when created. I thought you misread this PR, because your comment right after the veto comment actually agreed with this PR. If you prefer to review a PR after all tests pass, please let me know. Then I'll ping you after Jenkins passes, to involve you later and save time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28598: [SPARK-31785][SQL][TESTS] Add a helper function to test all parquet readers
SparkQA commented on pull request #28598: URL: https://github.com/apache/spark/pull/28598#issuecomment-63711 **[Test build #122936 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122936/testReport)** for PR 28598 at commit [`5ffc1e4`](https://github.com/apache/spark/commit/5ffc1e43c6617e33b9750eb3c9032421c4d1cdca). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #28598: [SPARK-31785][SQL][TESTS] Add a helper function to test all parquet readers
MaxGekk commented on a change in pull request #28598:
URL: https://github.com/apache/spark/pull/28598#discussion_r428779913
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTest.scala
##
@@ -162,4 +162,36 @@ private[sql] trait ParquetTest extends
FileBasedDataSourceTest {
protected def getResourceParquetFilePath(name: String): String = {
Thread.currentThread().getContextClassLoader.getResource(name).toString
}
+
+ def withParquetReaderFlags[T](vectorizedOSS: Boolean)
+(code: => T): T = {
+val sqlConfs = Seq(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key ->
vectorizedOSS.toString)
+var ret = null.asInstanceOf[T]
+withSQLConf(sqlConfs: _*) { ret = code }
+ret
+ }
+
+ def withParquetReader[T](reader: String)(code: => T): T = reader match {
+case "parquet-mr" => withParquetReaderFlags(false)(code)
+case "vectorized-oss" => withParquetReaderFlags(true)(code)
+case unknown =>
+ // scalastyle:off throwerror
+ throw new NotImplementedError(s"Unsupported Parquet reader '$unknown'.")
+ // scalastyle:on throwerror
+ }
+
+ def withParquetReaders(readers: String*)(code: => Unit): Unit = for (reader
<- readers) {
+// scalastyle:off
+println(s"with $reader reader")
+// scalastyle:on
+withParquetReader(reader)(code)
+ }
+
+ def withOssParquetReaders(code: => Unit): Unit = {
+withParquetReaders("parquet-mr", "vectorized-oss")(code)
+ }
+
+ def withAllParquetReaders(code: => Unit): Unit = {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28598: [SPARK-31785][SQL][TESTS] Add a helper function to test all parquet readers
AmplabJenkins removed a comment on pull request #28598: URL: https://github.com/apache/spark/pull/28598#issuecomment-632219275 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28598: [SPARK-31785][SQL][TESTS] Add a helper function to test all parquet readers
AmplabJenkins commented on pull request #28598: URL: https://github.com/apache/spark/pull/28598#issuecomment-632219275 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28598: [SPARK-31785][SQL][TESTS] Add a helper function to test all parquet readers
AmplabJenkins removed a comment on pull request #28598: URL: https://github.com/apache/spark/pull/28598#issuecomment-632188410 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28598: [SPARK-31785][SQL][TESTS] Add a helper function to test all parquet readers
AmplabJenkins commented on pull request #28598: URL: https://github.com/apache/spark/pull/28598#issuecomment-632188410 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28598: [SPARK-31785][SQL][TESTS] Add a helper function to test all parquet readers
SparkQA removed a comment on pull request #28598: URL: https://github.com/apache/spark/pull/28598#issuecomment-632045706 **[Test build #122930 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122930/testReport)** for PR 28598 at commit [`1c8b0e9`](https://github.com/apache/spark/commit/1c8b0e9186a7b804b2106587267f4875b26d32c3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28598: [SPARK-31785][SQL][TESTS] Add a helper function to test all parquet readers
SparkQA commented on pull request #28598: URL: https://github.com/apache/spark/pull/28598#issuecomment-632187258 **[Test build #122930 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122930/testReport)** for PR 28598 at commit [`1c8b0e9`](https://github.com/apache/spark/commit/1c8b0e9186a7b804b2106587267f4875b26d32c3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] rdblue commented on pull request #28523: [SPARK-31706][SQL] add back the support of streaming update mode
rdblue commented on pull request #28523: URL: https://github.com/apache/spark/pull/28523#issuecomment-632185172 This PR wasn't ready to review until late Monday where I am, and was merged less than a day later. I had also assumed that the fix would be in a separate PR, which is why I didn't follow this one too closely. In any case, if you want to merge a PR that has been vetoed, make sure concerns are satisfied. Being "active" is not a requirement and is poorly defined. That's not a valid excuse. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'
cloud-fan commented on a change in pull request #28592:
URL: https://github.com/apache/spark/pull/28592#discussion_r428759386
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala
##
@@ -86,6 +86,25 @@ trait DateTimeFormatterHelper {
throw e
}
}
+
+ // When the new DateTimeFormatter failed to initialize because of invalid
datetime pattern, it
+ // will throw IllegalArgumentException. If the pattern can be recognized by
the legacy formatter
+ // it will raise SparkUpgradeException to tell users to restore the previous
behavior via LEGACY
+ // policy or follow our guide to correct their pattern.
+ protected def checkLegacyFormatter[T1, T2](
Review comment:
what does the type parameter give us? The return type doesn't depend on
T1 or T2
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'
cloud-fan commented on a change in pull request #28592:
URL: https://github.com/apache/spark/pull/28592#discussion_r428758919
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
##
@@ -40,7 +41,11 @@ class Iso8601DateFormatter(
extends DateFormatter with DateTimeFormatterHelper {
@transient
- private lazy val formatter = getOrCreateFormatter(pattern, locale)
+ private lazy val formatter: DateTimeFormatter = {
+try {
+ getOrCreateFormatter(pattern, locale)
+} catch checkLegacyFormatter(pattern, legacyFormatter.format(0))
Review comment:
`legacyFormatter.format(0)` is hacky... let's add the `initialize` API
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28128: [SPARK-31354] SparkContext only register one SparkSession ApplicationEnd listener
AmplabJenkins removed a comment on pull request #28128: URL: https://github.com/apache/spark/pull/28128#issuecomment-632180091 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28128: [SPARK-31354] SparkContext only register one SparkSession ApplicationEnd listener
AmplabJenkins commented on pull request #28128: URL: https://github.com/apache/spark/pull/28128#issuecomment-632180091 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28128: [SPARK-31354] SparkContext only register one SparkSession ApplicationEnd listener
cloud-fan commented on pull request #28128: URL: https://github.com/apache/spark/pull/28128#issuecomment-632179830 thanks, merging to master/3.0! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #28128: [SPARK-31354] SparkContext only register one SparkSession ApplicationEnd listener
cloud-fan closed pull request #28128: URL: https://github.com/apache/spark/pull/28128 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28128: [SPARK-31354] SparkContext only register one SparkSession ApplicationEnd listener
SparkQA removed a comment on pull request #28128: URL: https://github.com/apache/spark/pull/28128#issuecomment-632038999 **[Test build #122929 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122929/testReport)** for PR 28128 at commit [`cfa1462`](https://github.com/apache/spark/commit/cfa14626fccd4ff2ab0e7175b8457b582c32017f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28128: [SPARK-31354] SparkContext only register one SparkSession ApplicationEnd listener
SparkQA commented on pull request #28128: URL: https://github.com/apache/spark/pull/28128#issuecomment-632178495 **[Test build #122929 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122929/testReport)** for PR 28128 at commit [`cfa1462`](https://github.com/apache/spark/commit/cfa14626fccd4ff2ab0e7175b8457b582c32017f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #27627: [WIP][SPARK-28067][SQL] Fix incorrect results for decimal aggregate sum by returning null on decimal overflow
cloud-fan commented on pull request #27627: URL: https://github.com/apache/spark/pull/27627#issuecomment-632166970 Hi @skambha , I've sent you a PR to only check overflow in `evaluateExpression`, and pass the tests. Can you take a look? https://github.com/skambha/spark/pull/1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28258: [SPARK-31486] [CORE] spark.submit.waitAppCompletion flag to control spark-submit exit in Standalone Cluster Mode
Ngone51 commented on a change in pull request #28258:
URL: https://github.com/apache/spark/pull/28258#discussion_r428729898
##
File path: core/src/main/scala/org/apache/spark/deploy/Client.scala
##
@@ -176,6 +202,22 @@ private class ClientEndpoint(
} else if (!Utils.responseFromBackup(message)) {
System.exit(-1)
}
+
+case DriverStatusResponse(found, state, _, _, _) =>
Review comment:
oh, sorry miss that. yea, thank you!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] akshatb1 commented on a change in pull request #28258: [SPARK-31486] [CORE] spark.submit.waitAppCompletion flag to control spark-submit exit in Standalone Cluster Mode
akshatb1 commented on a change in pull request #28258:
URL: https://github.com/apache/spark/pull/28258#discussion_r428727134
##
File path: core/src/main/scala/org/apache/spark/deploy/Client.scala
##
@@ -176,6 +202,22 @@ private class ClientEndpoint(
} else if (!Utils.responseFromBackup(message)) {
System.exit(-1)
}
+
+case DriverStatusResponse(found, state, _, _, _) =>
Review comment:
It's in the case "launch" as of now. I will move it to a global place
and refactor the code. Thanks for your suggestions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'
AmplabJenkins removed a comment on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632147078 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'
AmplabJenkins commented on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632147078 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'
SparkQA commented on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632146298 **[Test build #122935 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122935/testReport)** for PR 28592 at commit [`549a122`](https://github.com/apache/spark/commit/549a1225412756a2492c4c36191d8f18b845a23f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data
AmplabJenkins removed a comment on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-632142075 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data
AmplabJenkins commented on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-632142075 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data
SparkQA removed a comment on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-632058436 **[Test build #122931 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122931/testReport)** for PR 28525 at commit [`c59fcd6`](https://github.com/apache/spark/commit/c59fcd627f5132bf8ebe493a8f1282139b31f1cf). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data
SparkQA commented on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-632140854 **[Test build #122931 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122931/testReport)** for PR 28525 at commit [`c59fcd6`](https://github.com/apache/spark/commit/c59fcd627f5132bf8ebe493a8f1282139b31f1cf). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28258: [SPARK-31486] [CORE] spark.submit.waitAppCompletion flag to control spark-submit exit in Standalone Cluster Mode
Ngone51 commented on a change in pull request #28258:
URL: https://github.com/apache/spark/pull/28258#discussion_r428704776
##
File path: core/src/main/scala/org/apache/spark/deploy/Client.scala
##
@@ -176,6 +202,22 @@ private class ClientEndpoint(
} else if (!Utils.responseFromBackup(message)) {
System.exit(-1)
}
+
+case DriverStatusResponse(found, state, _, _, _) =>
Review comment:
we don't need to add `forwardMessageThread.scheduleAtFixedRate(...)`
into any `case` branches but just put it as a global one(just do what you do
now). I think it still works for `case "kill"`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] akshatb1 commented on a change in pull request #28258: [SPARK-31486] [CORE] spark.submit.waitAppCompletion flag to control spark-submit exit in Standalone Cluster Mode
akshatb1 commented on a change in pull request #28258:
URL: https://github.com/apache/spark/pull/28258#discussion_r428699825
##
File path: core/src/main/scala/org/apache/spark/deploy/Client.scala
##
@@ -176,6 +202,22 @@ private class ClientEndpoint(
} else if (!Utils.responseFromBackup(message)) {
System.exit(-1)
}
+
+case DriverStatusResponse(found, state, _, _, _) =>
Review comment:
I agree that we can change the delay to 5 seconds to keep it consistent
with current logic. My question is that should we add the following block in
`case "kill" =>` as well?
`forwardMessageThread.scheduleAtFixedRate(() =>
Utils.tryLogNonFatalError {
MonitorDriverStatus()
}, 0, REPORT_DRIVER_STATUS_INTERVAL, TimeUnit.MILLISECONDS)
`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] akshatb1 commented on a change in pull request #28258: [SPARK-31486] [CORE] spark.submit.waitAppCompletion flag to control spark-submit exit in Standalone Cluster Mode
akshatb1 commented on a change in pull request #28258:
URL: https://github.com/apache/spark/pull/28258#discussion_r428699825
##
File path: core/src/main/scala/org/apache/spark/deploy/Client.scala
##
@@ -176,6 +202,22 @@ private class ClientEndpoint(
} else if (!Utils.responseFromBackup(message)) {
System.exit(-1)
}
+
+case DriverStatusResponse(found, state, _, _, _) =>
Review comment:
I agree that we can change the delay to 5 seconds to keep it consistent
with current logic. My question is that should we add the following block in
`case "kill" =>` as well or should we just monitor with a single message
instead of scheduled messages?
`forwardMessageThread.scheduleAtFixedRate(() =>
Utils.tryLogNonFatalError {
MonitorDriverStatus()
}, 0, REPORT_DRIVER_STATUS_INTERVAL, TimeUnit.MILLISECONDS)
`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28258: [SPARK-31486] [CORE] spark.submit.waitAppCompletion flag to control spark-submit exit in Standalone Cluster Mode
Ngone51 commented on a change in pull request #28258:
URL: https://github.com/apache/spark/pull/28258#discussion_r428693738
##
File path: core/src/main/scala/org/apache/spark/deploy/Client.scala
##
@@ -176,6 +202,22 @@ private class ClientEndpoint(
} else if (!Utils.responseFromBackup(message)) {
System.exit(-1)
}
+
+case DriverStatusResponse(found, state, _, _, _) =>
Review comment:
that's why I said we need to change the delay (e.g. 5s) instead of 0 for
both submiting and killing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28258: [SPARK-31486] [CORE] spark.submit.waitAppCompletion flag to control spark-submit exit in Standalone Cluster Mode
Ngone51 commented on a change in pull request #28258:
URL: https://github.com/apache/spark/pull/28258#discussion_r428693738
##
File path: core/src/main/scala/org/apache/spark/deploy/Client.scala
##
@@ -176,6 +202,22 @@ private class ClientEndpoint(
} else if (!Utils.responseFromBackup(message)) {
System.exit(-1)
}
+
+case DriverStatusResponse(found, state, _, _, _) =>
Review comment:
that's why I said we need to change a the delay (e.g. 5s) instead of 0
for both submiting and killing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] akshatb1 commented on a change in pull request #28258: [SPARK-31486] [CORE] spark.submit.waitAppCompletion flag to control spark-submit exit in Standalone Cluster Mode
akshatb1 commented on a change in pull request #28258:
URL: https://github.com/apache/spark/pull/28258#discussion_r428691523
##
File path: core/src/main/scala/org/apache/spark/deploy/Client.scala
##
@@ -176,6 +202,22 @@ private class ClientEndpoint(
} else if (!Utils.responseFromBackup(message)) {
System.exit(-1)
}
+
+case DriverStatusResponse(found, state, _, _, _) =>
Review comment:
Scheduling to monitor driver status is done only in case of submit and
not in kill as of now. So we may need to explicitly send a message to monitor
driver status after 5 seconds delay in case of kill.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #25965: [SPARK-26425][SS] Add more constraint checks in file streaming source to avoid checkpoint corruption
SparkQA commented on pull request #25965: URL: https://github.com/apache/spark/pull/25965#issuecomment-632108254 **[Test build #122934 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122934/testReport)** for PR 25965 at commit [`1744179`](https://github.com/apache/spark/commit/17441794f67c102f2314005ac1b5c85afe8900be). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27664: [SPARK-30915][SS] CompactibleFileStreamLog: Avoid reading the metadata log file when finding the latest batch ID
AmplabJenkins removed a comment on pull request #27664: URL: https://github.com/apache/spark/pull/27664#issuecomment-632104886 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27664: [SPARK-30915][SS] CompactibleFileStreamLog: Avoid reading the metadata log file when finding the latest batch ID
AmplabJenkins commented on pull request #27664: URL: https://github.com/apache/spark/pull/27664#issuecomment-632104886 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27664: [SPARK-30915][SS] CompactibleFileStreamLog: Avoid reading the metadata log file when finding the latest batch ID
SparkQA commented on pull request #27664: URL: https://github.com/apache/spark/pull/27664#issuecomment-632104207 **[Test build #122933 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122933/testReport)** for PR 27664 at commit [`83451c1`](https://github.com/apache/spark/commit/83451c1fe28e40b62f40618429889f072971f866). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #27664: [SPARK-30915][SS] CompactibleFileStreamLog: Avoid reading the metadata log file when finding the latest batch ID
HeartSaVioR commented on a change in pull request #27664:
URL: https://github.com/apache/spark/pull/27664#discussion_r428668027
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala
##
@@ -182,19 +182,34 @@ class HDFSMetadataLog[T <: AnyRef :
ClassTag](sparkSession: SparkSession, path:
}
}
- override def getLatest(): Option[(Long, T)] = {
+ /**
+ * Return the latest batch Id without reading the file. This method only
checks for existence of
+ * file to avoid cost on reading and deserializing log file.
+ */
+ def getLatestBatchId(): Option[Long] = {
val batchIds = fileManager.list(metadataPath, batchFilesFilter)
.map(f => pathToBatchId(f.getPath))
.sorted(Ordering.Long.reverse)
for (batchId <- batchIds) {
- val batch = get(batchId)
- if (batch.isDefined) {
-return Some((batchId, batch.get))
+ val batchMetadataFile = batchIdToPath(batchId)
+ if (fileManager.exists(batchMetadataFile)) {
+return Some(batchId)
}
}
None
}
+ override def getLatest(): Option[(Long, T)] = {
+getLatestBatchId().map { batchId =>
+ val content = get(batchId).getOrElse {
+// This only happens in odd case where the file exists when
getLatestBatchId() is called,
+// but get() doesn't find it.
+throw new IllegalStateException(s"failed to read log file for batch
$batchId")
Review comment:
Just pulled the comment here. Either this or #25965 will have to resolve
merge conflict but wanted to be sure the code comment is clear in any way.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'
yaooqinn commented on a change in pull request #28592:
URL: https://github.com/apache/spark/pull/28592#discussion_r428667113
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
##
@@ -40,7 +43,23 @@ class Iso8601DateFormatter(
extends DateFormatter with DateTimeFormatterHelper {
@transient
- private lazy val formatter = getOrCreateFormatter(pattern, locale)
+ private lazy val formatter = {
Review comment:
Hmm, this one and the others are transient, so the lazy keyword is
required.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'
yaooqinn commented on a change in pull request #28592:
URL: https://github.com/apache/spark/pull/28592#discussion_r428667113
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
##
@@ -40,7 +43,23 @@ class Iso8601DateFormatter(
extends DateFormatter with DateTimeFormatterHelper {
@transient
- private lazy val formatter = getOrCreateFormatter(pattern, locale)
+ private lazy val formatter = {
Review comment:
Hmm, this one and the other are transient, so the lazy keyword is
required.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column
maropu commented on a change in pull request #28560:
URL: https://github.com/apache/spark/pull/28560#discussion_r428666366
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasingSuite.scala
##
@@ -341,6 +341,78 @@ class NestedColumnAliasingSuite extends SchemaPruningTest {
.analyze
comparePlans(optimized, expected)
}
+
+ test("Nested field pruning for Aggregate") {
+val query1 =
contact.groupBy($"id")(first($"name.first").as("first")).analyze
+val optimized1 = Optimize.execute(query1)
+val aliases1 = collectGeneratedAliases(optimized1)
+
+val expected1 = contact
+ .select($"id", 'name.getField("first").as(aliases1(0)))
+ .groupBy($"id")(first($"${aliases1(0)}").as("first")).analyze
+comparePlans(optimized1, expected1)
+
+val query2 =
contact.groupBy($"name.last")(first($"name.first").as("first")).analyze
+val optimized2 = Optimize.execute(query2)
+val aliases2 = collectGeneratedAliases(optimized2)
+
+val expected2 = contact
+ .select('name.getField("last").as(aliases2(0)),
'name.getField("first").as(aliases2(1)))
+ .groupBy($"${aliases2(0)}")(first($"${aliases2(1)}").as("first")).analyze
+comparePlans(optimized2, expected2)
+
+val query3 = contact.groupBy($"id")(first($"name"),
first($"name.first").as("first")).analyze
+val optimized3 = Optimize.execute(query3)
+val expected3 = contact.select($"id", $"name")
+ .groupBy($"id")(first($"name"), first($"name.first").as("first")).analyze
+comparePlans(optimized3, expected3)
+ }
+
+ test("Nested field pruning for Expand") {
+val query1 = Expand(
+ Seq(
+Seq($"name.first", $"name.middle"),
+Seq(ConcatWs(Seq($"name.first", $"name.middle")),
+ ConcatWs(Seq($"name.middle", $"name.first")))
+ ),
+ Seq('a.string, 'b.string),
+ contact
Review comment:
ditto; https://github.com/apache/spark/pull/28560#discussion_r428665197
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column
maropu commented on a change in pull request #28560:
URL: https://github.com/apache/spark/pull/28560#discussion_r428662926
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala
##
@@ -48,7 +53,10 @@ object NestedColumnAliasing {
case Project(projectList, child) =>
Project(
getNewProjectList(projectList, nestedFieldToAlias),
-replaceChildrenWithAliases(child, attrToAliases))
+replaceChildrenWithAliases(child, nestedFieldToAlias, attrToAliases))
+
+case other =>
Review comment:
This case only matches `Aggregate` and `Expand` now?
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasingSuite.scala
##
@@ -341,6 +341,78 @@ class NestedColumnAliasingSuite extends SchemaPruningTest {
.analyze
comparePlans(optimized, expected)
}
+
+ test("Nested field pruning for Aggregate") {
+val query1 =
contact.groupBy($"id")(first($"name.first").as("first")).analyze
Review comment:
Could you add tests for the cases,
`contact.(limit/repartition/sample).groupBy()...`? IIUC this PR could support
them, too?
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala
##
@@ -35,6 +35,11 @@ object NestedColumnAliasing {
case Project(projectList, child)
if SQLConf.get.nestedSchemaPruningEnabled &&
canProjectPushThrough(child) =>
getAliasSubMap(projectList)
+
+case plan if SQLConf.get.nestedSchemaPruningEnabled && canPruneOn(plan) =>
+ val exprsToPrune = plan.expressions
Review comment:
nit: the consistent var name?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27664: [SPARK-30915][SS] CompactibleFileStreamLog: Avoid reading the metadata log file when finding the latest batch ID
xuanyuanking commented on a change in pull request #27664:
URL: https://github.com/apache/spark/pull/27664#discussion_r428661867
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala
##
@@ -182,19 +182,34 @@ class HDFSMetadataLog[T <: AnyRef :
ClassTag](sparkSession: SparkSession, path:
}
}
- override def getLatest(): Option[(Long, T)] = {
+ /**
+ * Return the latest batch Id without reading the file. This method only
checks for existence of
+ * file to avoid cost on reading and deserializing log file.
+ */
+ def getLatestBatchId(): Option[Long] = {
val batchIds = fileManager.list(metadataPath, batchFilesFilter)
.map(f => pathToBatchId(f.getPath))
.sorted(Ordering.Long.reverse)
for (batchId <- batchIds) {
- val batch = get(batchId)
- if (batch.isDefined) {
-return Some((batchId, batch.get))
+ val batchMetadataFile = batchIdToPath(batchId)
+ if (fileManager.exists(batchMetadataFile)) {
+return Some(batchId)
}
}
None
}
+ override def getLatest(): Option[(Long, T)] = {
+getLatestBatchId().map { batchId =>
+ val content = get(batchId).getOrElse {
+// This only happens in odd case where the file exists when
getLatestBatchId() is called,
+// but get() doesn't find it.
+throw new IllegalStateException(s"failed to read log file for batch
$batchId")
Review comment:
Thanks for reference #25965, LGTM to this change.
I personally think the comment in
https://github.com/apache/spark/pull/25965/files#diff-aaeb546880508bb771df502318c40a99R183
is clearer. Either way is fine though.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28258: [SPARK-31486] [CORE] spark.submit.waitAppCompletion flag to control spark-submit exit in Standalone Cluster Mode
Ngone51 commented on a change in pull request #28258:
URL: https://github.com/apache/spark/pull/28258#discussion_r428661567
##
File path: core/src/main/scala/org/apache/spark/deploy/Client.scala
##
@@ -176,6 +202,22 @@ private class ClientEndpoint(
} else if (!Utils.responseFromBackup(message)) {
System.exit(-1)
}
+
+case DriverStatusResponse(found, state, _, _, _) =>
Review comment:
> If we removing polling from that, what logic should be handled there?
we use this:
```
forwardMessageThread.scheduleAtFixedRate(() => Utils.tryLogNonFatalError {
MonitorDriverStatus()
}, 0, REPORT_DRIVER_STATUS_INTERVAL, TimeUnit.MILLISECONDS)
```
(but the initial delay need to change)
in this way, submitting or killing drivers will still use it only for one
time when `waitAppCompletion=false`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] akshatb1 commented on a change in pull request #28258: [SPARK-31486] [CORE] spark.submit.waitAppCompletion flag to control spark-submit exit in Standalone Cluster Mode
akshatb1 commented on a change in pull request #28258:
URL: https://github.com/apache/spark/pull/28258#discussion_r428657865
##
File path: core/src/main/scala/org/apache/spark/deploy/Client.scala
##
@@ -176,6 +202,22 @@ private class ClientEndpoint(
} else if (!Utils.responseFromBackup(message)) {
System.exit(-1)
}
+
+case DriverStatusResponse(found, state, _, _, _) =>
Review comment:
@Ngone51 Thanks for your feedback. `pollAndReportStatus` is only being
used the first time after submitting or killing drivers. I am not sure which is
the duplicate logic you are referring to. Also, `pollAndReportStatus` is only
polling the driver status and handling the response. If we removing polling
from that, what logic should be handled there?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #27066: [SPARK-31317][SQL] Add withFields method to Column
cloud-fan commented on pull request #27066:
URL: https://github.com/apache/spark/pull/27066#issuecomment-632087480
I went through the discussion in the JIRA, there is one interesting use case
```
data.withColumn("newArray",
transform('array, structElem => structElem.withField("d",
lit("hello".show(false)
```
I agree that it's necessary to add more APIs in `Column`, so that it's
easier to use higher-order functions to operate nested fields. Given the fact
that we can always reuse the APIs of `Column` with `Dataset.withColumn`, it
seems fine to only add APIs to `Column`.
Is it possible to support `data.withColumn("a", $"a".withField("b.a",
lit(5)))`? This is much simpler to use IMO.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join
AmplabJenkins removed a comment on pull request #28556: URL: https://github.com/apache/spark/pull/28556#issuecomment-632082073 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] GuoPhilipse commented on a change in pull request #28593: [SPARK-31710][SQL] Add two compatibility flag to cast long to timestamp
GuoPhilipse commented on a change in pull request #28593:
URL: https://github.com/apache/spark/pull/28593#discussion_r428646486
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -2586,6 +2586,22 @@ object SQLConf {
.checkValue(_ > 0, "The timeout value must be positive")
.createWithDefault(10L)
+ val LEGACY_NUMERIC_CONVERT_TO_TIMESTAMP_ENABLE =
+buildConf("spark.sql.legacy.numericConvertToTimestampEnable")
+ .doc("when true,use legacy numberic can convert to timestamp")
+ .version("3.0.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ val LEGACY_NUMERIC_CONVERT_TO_TIMESTAMP_IN_SECONDS =
+buildConf("spark.sql.legacy.numericConvertToTimestampInSeconds")
+ .internal()
+ .doc("The legacy only works when
LEGACY_NUMERIC_CONVERT_TO_TIMESTAMP_ENABLE is true." +
+"when true,the value will be interpreted as seconds,which follow
spark style," +
+"when false,value is interpreted as milliseconds,which follow hive
style")
Review comment:
@cloud-fan how do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join
AmplabJenkins commented on pull request #28556: URL: https://github.com/apache/spark/pull/28556#issuecomment-632082073 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28258: [SPARK-31486] [CORE] spark.submit.waitAppCompletion flag to control spark-submit exit in Standalone Cluster Mode
Ngone51 commented on a change in pull request #28258:
URL: https://github.com/apache/spark/pull/28258#discussion_r428645814
##
File path: core/src/main/scala/org/apache/spark/deploy/Client.scala
##
@@ -176,6 +202,22 @@ private class ClientEndpoint(
} else if (!Utils.responseFromBackup(message)) {
System.exit(-1)
}
+
+case DriverStatusResponse(found, state, _, _, _) =>
Review comment:
It would be better if we could do some refactor on `pollAndReportStatus`
in order to reduce some duplicate logic.
For example, we can only call `pollAndReportStatus` here, and remove other
invocations in `SubmitDriverResponse`/ `KillDriverResponse`. And, of course,
the `pollAndReportStatus`(it also needs a new name) will not poll the status
anymore.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join
SparkQA removed a comment on pull request #28556: URL: https://github.com/apache/spark/pull/28556#issuecomment-631965191 **[Test build #122922 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122922/testReport)** for PR 28556 at commit [`db601df`](https://github.com/apache/spark/commit/db601dffb47c1e608b6d43708eceb4220a7cb0a7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28583: [SPARK-31764][CORE] JsonProtocol doesn't write RDDInfo#isBarrier
AmplabJenkins removed a comment on pull request #28583: URL: https://github.com/apache/spark/pull/28583#issuecomment-632081247 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join
SparkQA commented on pull request #28556: URL: https://github.com/apache/spark/pull/28556#issuecomment-632081113 **[Test build #122922 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122922/testReport)** for PR 28556 at commit [`db601df`](https://github.com/apache/spark/commit/db601dffb47c1e608b6d43708eceb4220a7cb0a7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28583: [SPARK-31764][CORE] JsonProtocol doesn't write RDDInfo#isBarrier
AmplabJenkins commented on pull request #28583: URL: https://github.com/apache/spark/pull/28583#issuecomment-632081247 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28583: [SPARK-31764][CORE] JsonProtocol doesn't write RDDInfo#isBarrier
SparkQA removed a comment on pull request #28583: URL: https://github.com/apache/spark/pull/28583#issuecomment-632016717 **[Test build #122927 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122927/testReport)** for PR 28583 at commit [`53ebb55`](https://github.com/apache/spark/commit/53ebb55e4119c5edf22cd8642907082831bab75c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28583: [SPARK-31764][CORE] JsonProtocol doesn't write RDDInfo#isBarrier
SparkQA commented on pull request #28583: URL: https://github.com/apache/spark/pull/28583#issuecomment-632080202 **[Test build #122927 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122927/testReport)** for PR 28583 at commit [`53ebb55`](https://github.com/apache/spark/commit/53ebb55e4119c5edf22cd8642907082831bab75c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28258: [SPARK-31486] [CORE] spark.submit.waitAppCompletion flag to control spark-submit exit in Standalone Cluster Mode
Ngone51 commented on a change in pull request #28258:
URL: https://github.com/apache/spark/pull/28258#discussion_r428643339
##
File path: core/src/main/scala/org/apache/spark/deploy/Client.scala
##
@@ -123,16 +134,24 @@ private class ClientEndpoint(
}(forwardMessageExecutionContext)
}
}
+ private def MonitorDriverStatus(): Unit = {
Review comment:
nit: method should start with lowercase.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28258: [SPARK-31486] [CORE] spark.submit.waitAppCompletion flag to control spark-submit exit in Standalone Cluster Mode
Ngone51 commented on a change in pull request #28258:
URL: https://github.com/apache/spark/pull/28258#discussion_r428643084
##
File path: core/src/main/scala/org/apache/spark/deploy/Client.scala
##
@@ -123,16 +134,24 @@ private class ClientEndpoint(
}(forwardMessageExecutionContext)
}
}
+ private def MonitorDriverStatus(): Unit = {
Review comment:
nit: need empty line above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join
maropu commented on pull request #28556: URL: https://github.com/apache/spark/pull/28556#issuecomment-632075943 I left some minor comments though, it looks okay. cc: @dongjoon-hyun @dbtsai This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28556: [SPARK-31736][SQL] Nested column aliasing for RepartitionByExpression/Join
maropu commented on a change in pull request #28556:
URL: https://github.com/apache/spark/pull/28556#discussion_r428635965
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasingSuite.scala
##
@@ -144,7 +144,6 @@ class NestedColumnAliasingSuite extends SchemaPruningTest {
test("Pushing a single nested field projection - negative") {
val ops = Seq(
(input: LogicalPlan) => input.distribute('name)(1),
- (input: LogicalPlan) => input.distribute($"name.middle")(1),
Review comment:
Ah, looks nice. This PR could support this case.
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaPruningSuite.scala
##
@@ -338,6 +349,93 @@ abstract class SchemaPruningSuite
}
}
+ testSchemaPruning("select one deep nested complex field after repartition") {
+val query = sql("select * from contacts")
+ .repartition(100)
+ .where("employer.company.address is not null")
+ .selectExpr("employer.id as employer_id")
+checkScan(query,
+ "struct>>")
+checkAnswer(query, Row(0) :: Nil)
+ }
+
+ testSchemaPruning("select one deep nested complex field after repartition by
expression") {
+val query1 = sql("select * from contacts")
+ .repartition(100, col("id"))
+ .where("employer.company.address is not null")
+ .selectExpr("employer.id as employer_id")
+checkScan(query1,
+ "struct>>")
+checkAnswer(query1, Row(0) :: Nil)
+
+val query2 = sql("select * from contacts")
+ .repartition(100, col("employer"))
+ .where("employer.company.address is not null")
+ .selectExpr("employer.id as employer_id")
+checkScan(query2,
+
"struct>>")
+checkAnswer(query2, Row(0) :: Nil)
+
+val query3 = sql("select * from contacts")
+ .repartition(100, col("employer.company"))
+ .where("employer.company.address is not null")
+ .selectExpr("employer.company as employer_company")
+checkScan(query3,
+ "struct>>")
+checkAnswer(query3, Row(Row("abc", "123 Business Street")) :: Nil)
+
+val query4 = sql("select * from contacts")
+ .repartition(100, col("employer.company.address"))
+ .where("employer.company.address is not null")
+ .selectExpr("employer.company.address as employer_company_addr")
+checkScan(query4,
+ "struct>>")
+checkAnswer(query4, Row("123 Business Street") :: Nil)
+ }
+
+ testSchemaPruning("select one deep nested complex field after join") {
+val query1 = sql("select contacts.name.middle from contacts, departments
where " +
+"contacts.id = departments.contactId")
+checkScan(query1,
+ "struct>",
+"struct")
+checkAnswer(query1, Row("X.") :: Row("Y.") :: Nil)
+
+val query2 = sql("select contacts.name.middle from contacts, departments
where " +
+ "contacts.employer = departments.employer")
+checkScan(query2,
+ "struct," +
+"employer:struct>>",
+
"struct>>")
+checkAnswer(query2, Row("X.") :: Row("Y.") :: Nil)
+
+val query3 = sql("select contacts.employer.company.name from contacts,
departments where " +
+ "contacts.employer = departments.employer")
+checkScan(query3,
+
"struct>>",
+
"struct>>")
+checkAnswer(query3, Row("abc") :: Row(null) :: Nil)
+ }
+
+ testSchemaPruning("select one deep nested complex field after outer join") {
Review comment:
Thanks for adding the tests.
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasingSuite.scala
##
@@ -341,6 +340,90 @@ class NestedColumnAliasingSuite extends SchemaPruningTest {
.analyze
comparePlans(optimized, expected)
}
+
+ test("Nested field pruning through RepartitionByExpression") {
+val query1 = contact
+ .distribute($"id")(1)
+ .select($"name.middle")
+ .analyze
+val optimized1 = Optimize.execute(query1)
+
+val aliases1 = collectGeneratedAliases(optimized1)
+
+val expected1 = contact
+ .select('id, 'name.getField("middle").as(aliases1(0)))
+ .distribute($"id")(1)
+ .select($"${aliases1(0)}".as("middle"))
+ .analyze
+comparePlans(optimized1, expected1)
+
+
Review comment:
nit: unnecessary line break.
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaPruningSuite.scala
##
@@ -338,6 +349,93 @@ abstract class SchemaPruningSuite
}
}
+ testSchemaPruning("select one deep nested complex field after repartition") {
+val query = sql("select * from contacts")
+ .repartition(100)
+ .where("employer.company.address is not null")
+ .selectExpr("employer.id as employer_id")
+checkScan(query,
+ "struct>>")
+checkAnswer(query, Row(0) :: Nil)
+ }
+
+ testSchemaPruning("select one deep nested complex field after repartition by
expression") {
+val query1 =
[GitHub] [spark] cloud-fan commented on pull request #27066: [SPARK-31317][SQL] Add withFields method to Column
cloud-fan commented on pull request #27066:
URL: https://github.com/apache/spark/pull/27066#issuecomment-632075377
I haven't looked at the patch yet, but looking at the proposed API
```
val result = data.withColumn("a", 'a.withFields(
("b", $"a.b".withFields(
("a", $"a.b.a".withFields(
("b", lit(5
```
This is a huge improvement to the original API, but I'm thinking if we can
do better. What if we make `withColumn` support nested fields? Then we can
further simplify the API:
```
val result = data.withColumn("a.b.a", lit(5))
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28576: [SPARK-31755][SQL] allow missing year/hour when parsing date/timestamp string
AmplabJenkins removed a comment on pull request #28576: URL: https://github.com/apache/spark/pull/28576#issuecomment-632069072 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28576: [SPARK-31755][SQL] allow missing year/hour when parsing date/timestamp string
AmplabJenkins commented on pull request #28576: URL: https://github.com/apache/spark/pull/28576#issuecomment-632069072 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28576: [SPARK-31755][SQL] allow missing year/hour when parsing date/timestamp string
SparkQA commented on pull request #28576: URL: https://github.com/apache/spark/pull/28576#issuecomment-632068610 **[Test build #122932 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122932/testReport)** for PR 28576 at commit [`161223b`](https://github.com/apache/spark/commit/161223b5de92064a4cc3672702e09f868bd68870). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'
AmplabJenkins removed a comment on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632065812 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122928/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'
SparkQA removed a comment on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632032488 **[Test build #122928 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122928/testReport)** for PR 28592 at commit [`1144c03`](https://github.com/apache/spark/commit/1144c035fc55b78e004dd0d39c9b7df9a638e74e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'
AmplabJenkins removed a comment on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632065807 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'
SparkQA commented on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632065738 **[Test build #122928 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122928/testReport)** for PR 28592 at commit [`1144c03`](https://github.com/apache/spark/commit/1144c035fc55b78e004dd0d39c9b7df9a638e74e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'
AmplabJenkins commented on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632065807 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions
AmplabJenkins removed a comment on pull request #28534: URL: https://github.com/apache/spark/pull/28534#issuecomment-632059081 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122921/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data
AmplabJenkins removed a comment on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-632059000 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions
AmplabJenkins removed a comment on pull request #28534: URL: https://github.com/apache/spark/pull/28534#issuecomment-632059071 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions
SparkQA removed a comment on pull request #28534: URL: https://github.com/apache/spark/pull/28534#issuecomment-631955288 **[Test build #122921 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122921/testReport)** for PR 28534 at commit [`c1257f2`](https://github.com/apache/spark/commit/c1257f2552320141e435884629350685d5439d3e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions
AmplabJenkins commented on pull request #28534: URL: https://github.com/apache/spark/pull/28534#issuecomment-632059071 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28525: [SPARK-27562][Shuffle] Complete the verification mechanism for shuffle transmitted data
AmplabJenkins commented on pull request #28525: URL: https://github.com/apache/spark/pull/28525#issuecomment-632059000 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions
SparkQA commented on pull request #28534: URL: https://github.com/apache/spark/pull/28534#issuecomment-632058682 **[Test build #122921 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122921/testReport)** for PR 28534 at commit [`c1257f2`](https://github.com/apache/spark/commit/c1257f2552320141e435884629350685d5439d3e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
