[GitHub] [spark] AmplabJenkins commented on pull request #29722: [SPARK-32850][CORE] Simplify the RPC message flow of decommission
AmplabJenkins commented on pull request #29722: URL: https://github.com/apache/spark/pull/29722#issuecomment-693214454 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29765: [SPARK-32888][DOCS] Add user document about header flag and RDD as path for reading CSV
AmplabJenkins commented on pull request #29765: URL: https://github.com/apache/spark/pull/29765#issuecomment-693214458 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29604: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
AmplabJenkins commented on pull request #29604: URL: https://github.com/apache/spark/pull/29604#issuecomment-693214552 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29722: [SPARK-32850][CORE] Simplify the RPC message flow of decommission
AmplabJenkins removed a comment on pull request #29722: URL: https://github.com/apache/spark/pull/29722#issuecomment-693214454 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29765: [SPARK-32888][DOCS] Add user document about header flag and RDD as path for reading CSV
AmplabJenkins removed a comment on pull request #29765: URL: https://github.com/apache/spark/pull/29765#issuecomment-693214458 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on a change in pull request #29729: [SPARK-32032][SS] Avoid infinite wait in driver because of KafkaConsumer.poll(long) API
gaborgsomogyi commented on a change in pull request #29729:
URL: https://github.com/apache/spark/pull/29729#discussion_r489206957
##
File path:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
##
@@ -182,28 +170,14 @@ private[kafka010] class KafkaOffsetReader(
"If startingOffsets contains specific offsets, you must specify all
TopicPartitions.\n" +
"Use -1 for latest, -2 for earliest, if you don't care.\n" +
s"Specified: ${partitionOffsets.keySet} Assigned:
${partitions.asScala}")
- logDebug(s"Partitions assigned to consumer: $partitions. Seeking to
$partitionOffsets")
+ logDebug(s"Assigned partitions: $partitions. Seeking to
$partitionOffsets")
}
val fnRetrievePartitionOffsets: ju.Set[TopicPartition] =>
Map[TopicPartition, Long] = { _ =>
partitionOffsets
}
-val fnAssertFetchedOffsets: Map[TopicPartition, Long] => Unit = { fetched
=>
- partitionOffsets.foreach {
-case (tp, off) if off != KafkaOffsetRangeLimit.LATEST &&
- off != KafkaOffsetRangeLimit.EARLIEST =>
- if (fetched(tp) != off) {
-reportDataLoss(
Review comment:
Yeah, I was not able to create a scenario where the condition fired (see
the standalone app). If somebody can, feel free to add and we can re-evaluate
this code part.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29604: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
AmplabJenkins removed a comment on pull request #29604: URL: https://github.com/apache/spark/pull/29604#issuecomment-693214552 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
SparkQA commented on pull request #29731: URL: https://github.com/apache/spark/pull/29731#issuecomment-693216725 **[Test build #128744 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128744/testReport)** for PR 29731 at commit [`abf2f7e`](https://github.com/apache/spark/commit/abf2f7e45b60f15742bf04c47ad1ff6d0ab130dc). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
AmplabJenkins commented on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-693216812 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #25965: [SPARK-26425][SS] Add more constraint checks to avoid checkpoint corruption
AmplabJenkins commented on pull request #25965: URL: https://github.com/apache/spark/pull/25965#issuecomment-693216851 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29765: [SPARK-32888][DOCS] Add user document about header flag and RDD as path for reading CSV
SparkQA commented on pull request #29765: URL: https://github.com/apache/spark/pull/29765#issuecomment-693216741 **[Test build #128747 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128747/testReport)** for PR 29765 at commit [`19ee398`](https://github.com/apache/spark/commit/19ee3987d7a88af943dcb39c7ba22e83573521b8). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #25965: [SPARK-26425][SS] Add more constraint checks to avoid checkpoint corruption
SparkQA commented on pull request #25965: URL: https://github.com/apache/spark/pull/25965#issuecomment-693216733 **[Test build #128745 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128745/testReport)** for PR 25965 at commit [`a885212`](https://github.com/apache/spark/commit/a8852123ae978dddb5578e3411b4822fd4a1fdce). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29316: [SPARK-32508][SQL] Disallow empty part col values in partition spec before static partition writing
SparkQA commented on pull request #29316: URL: https://github.com/apache/spark/pull/29316#issuecomment-693216736 **[Test build #128735 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128735/testReport)** for PR 29316 at commit [`c8ac369`](https://github.com/apache/spark/commit/c8ac36917caf7fa67a0bca3c30a10380fd7c7306). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29739: [SPARK-32867][SQL] When explain, HiveTableRelation show limited message
SparkQA commented on pull request #29739: URL: https://github.com/apache/spark/pull/29739#issuecomment-693216722 **[Test build #128743 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128743/testReport)** for PR 29739 at commit [`883f547`](https://github.com/apache/spark/commit/883f5471ec653cd848f18c9180cb7fca6b90eb51). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29585: [SPARK-32741][SQL] Check if the same ExprId refers to the unique attribute in logical plans
SparkQA commented on pull request #29585: URL: https://github.com/apache/spark/pull/29585#issuecomment-693216740 **[Test build #128734 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128734/testReport)** for PR 29585 at commit [`fb048c7`](https://github.com/apache/spark/commit/fb048c7afcc6e366f8e840a3e0a02849a129a04c). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.
SparkQA commented on pull request #29591: URL: https://github.com/apache/spark/pull/29591#issuecomment-693216732 **[Test build #128740 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128740/testReport)** for PR 29591 at commit [`172f99a`](https://github.com/apache/spark/commit/172f99a3de38294c5f7a41340a587ca25cd29d4f). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29722: [SPARK-32850][CORE] Simplify the RPC message flow of decommission
AmplabJenkins commented on pull request #29722: URL: https://github.com/apache/spark/pull/29722#issuecomment-693216761 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29767: [SPARK-32896][SS] Add DataStreamWriter.table API
SparkQA commented on pull request #29767: URL: https://github.com/apache/spark/pull/29767#issuecomment-693216742 **[Test build #128739 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128739/testReport)** for PR 29767 at commit [`f557696`](https://github.com/apache/spark/commit/f5576969ee9afeaa35c3bb74822aad43cf9c11c2). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29739: [SPARK-32867][SQL] When explain, HiveTableRelation show limited message
AmplabJenkins commented on pull request #29739: URL: https://github.com/apache/spark/pull/29739#issuecomment-693216935 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29586: [SPARK-32743][SQL] Add distinct info at UnresolvedFunction toString
SparkQA commented on pull request #29586: URL: https://github.com/apache/spark/pull/29586#issuecomment-693216735 **[Test build #128741 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128741/testReport)** for PR 29586 at commit [`95a3e02`](https://github.com/apache/spark/commit/95a3e028b2d1c83eaeab4baf15c07870df4a66ad). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29722: [SPARK-32850][CORE] Simplify the RPC message flow of decommission
SparkQA commented on pull request #29722: URL: https://github.com/apache/spark/pull/29722#issuecomment-693216726 **[Test build #128748 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128748/testReport)** for PR 29722 at commit [`290d2c0`](https://github.com/apache/spark/commit/290d2c08a1adfce3c0c4afe1cb8c9e214b25ea3d). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29765: [SPARK-32888][DOCS] Add user document about header flag and RDD as path for reading CSV
AmplabJenkins commented on pull request #29765: URL: https://github.com/apache/spark/pull/29765#issuecomment-693216775 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29769: [SPARK-32804][Launcher][FOLLOWUP] Fix SparkSubmitCommandBuilderSuite test failure without jars
SparkQA commented on pull request #29769: URL: https://github.com/apache/spark/pull/29769#issuecomment-693216734 **[Test build #128742 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128742/testReport)** for PR 29769 at commit [`e2b0c6b`](https://github.com/apache/spark/commit/e2b0c6bc13cb0cd702f14cc6410e5d65045fe97f). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
SparkQA commented on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-693216720 **[Test build #128746 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128746/testReport)** for PR 28841 at commit [`1c8384c`](https://github.com/apache/spark/commit/1c8384cebe05fbc1ebb3c86660e57508b031c6f9). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class ModifiedBeforeFilter(thresholdTime: Long, val timeZoneId: String)` * `class ModifiedAfterFilter(thresholdTime: Long, val timeZoneId: String)` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29739: [SPARK-32867][SQL] When explain, HiveTableRelation show limited message
AmplabJenkins removed a comment on pull request #29739: URL: https://github.com/apache/spark/pull/29739#issuecomment-693216935 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29722: [SPARK-32850][CORE] Simplify the RPC message flow of decommission
SparkQA removed a comment on pull request #29722: URL: https://github.com/apache/spark/pull/29722#issuecomment-693213903 **[Test build #128748 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128748/testReport)** for PR 29722 at commit [`290d2c0`](https://github.com/apache/spark/commit/290d2c08a1adfce3c0c4afe1cb8c9e214b25ea3d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29604: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
SparkQA commented on pull request #29604: URL: https://github.com/apache/spark/pull/29604#issuecomment-693217111 **[Test build #128749 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128749/testReport)** for PR 29604 at commit [`8778412`](https://github.com/apache/spark/commit/8778412afa9a50e9dace13f140bb7c253fc0dfc7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
SparkQA removed a comment on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-693202858 **[Test build #128746 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128746/testReport)** for PR 28841 at commit [`1c8384c`](https://github.com/apache/spark/commit/1c8384cebe05fbc1ebb3c86660e57508b031c6f9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29585: [SPARK-32741][SQL] Check if the same ExprId refers to the unique attribute in logical plans
SparkQA removed a comment on pull request #29585: URL: https://github.com/apache/spark/pull/29585#issuecomment-693135147 **[Test build #128734 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128734/testReport)** for PR 29585 at commit [`fb048c7`](https://github.com/apache/spark/commit/fb048c7afcc6e366f8e840a3e0a02849a129a04c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29769: [SPARK-32804][Launcher][FOLLOWUP] Fix SparkSubmitCommandBuilderSuite test failure without jars
AmplabJenkins commented on pull request #29769: URL: https://github.com/apache/spark/pull/29769#issuecomment-693217235 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29769: [SPARK-32804][Launcher][FOLLOWUP] Fix SparkSubmitCommandBuilderSuite test failure without jars
AmplabJenkins removed a comment on pull request #29769: URL: https://github.com/apache/spark/pull/29769#issuecomment-693217235 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29722: [SPARK-32850][CORE] Simplify the RPC message flow of decommission
AmplabJenkins removed a comment on pull request #29722: URL: https://github.com/apache/spark/pull/29722#issuecomment-693216761 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
SparkQA removed a comment on pull request #29731: URL: https://github.com/apache/spark/pull/29731#issuecomment-693184148 **[Test build #128744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128744/testReport)** for PR 29731 at commit [`abf2f7e`](https://github.com/apache/spark/commit/abf2f7e45b60f15742bf04c47ad1ff6d0ab130dc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29769: [SPARK-32804][Launcher][FOLLOWUP] Fix SparkSubmitCommandBuilderSuite test failure without jars
SparkQA removed a comment on pull request #29769: URL: https://github.com/apache/spark/pull/29769#issuecomment-693179326 **[Test build #128742 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128742/testReport)** for PR 29769 at commit [`e2b0c6b`](https://github.com/apache/spark/commit/e2b0c6bc13cb0cd702f14cc6410e5d65045fe97f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29765: [SPARK-32888][DOCS] Add user document about header flag and RDD as path for reading CSV
SparkQA removed a comment on pull request #29765: URL: https://github.com/apache/spark/pull/29765#issuecomment-693213988 **[Test build #128747 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128747/testReport)** for PR 29765 at commit [`19ee398`](https://github.com/apache/spark/commit/19ee3987d7a88af943dcb39c7ba22e83573521b8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
AmplabJenkins removed a comment on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-693216812 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.
SparkQA removed a comment on pull request #29591: URL: https://github.com/apache/spark/pull/29591#issuecomment-693148274 **[Test build #128740 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128740/testReport)** for PR 29591 at commit [`172f99a`](https://github.com/apache/spark/commit/172f99a3de38294c5f7a41340a587ca25cd29d4f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29767: [SPARK-32896][SS] Add DataStreamWriter.table API
SparkQA removed a comment on pull request #29767: URL: https://github.com/apache/spark/pull/29767#issuecomment-693148256 **[Test build #128739 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128739/testReport)** for PR 29767 at commit [`f557696`](https://github.com/apache/spark/commit/f5576969ee9afeaa35c3bb74822aad43cf9c11c2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29739: [SPARK-32867][SQL] When explain, HiveTableRelation show limited message
SparkQA removed a comment on pull request #29739: URL: https://github.com/apache/spark/pull/29739#issuecomment-693179376 **[Test build #128743 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128743/testReport)** for PR 29739 at commit [`883f547`](https://github.com/apache/spark/commit/883f5471ec653cd848f18c9180cb7fca6b90eb51). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29316: [SPARK-32508][SQL] Disallow empty part col values in partition spec before static partition writing
SparkQA removed a comment on pull request #29316: URL: https://github.com/apache/spark/pull/29316#issuecomment-693135193 **[Test build #128735 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128735/testReport)** for PR 29316 at commit [`c8ac369`](https://github.com/apache/spark/commit/c8ac36917caf7fa67a0bca3c30a10380fd7c7306). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #25965: [SPARK-26425][SS] Add more constraint checks to avoid checkpoint corruption
AmplabJenkins removed a comment on pull request #25965: URL: https://github.com/apache/spark/pull/25965#issuecomment-693216851 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29586: [SPARK-32743][SQL] Add distinct info at UnresolvedFunction toString
SparkQA removed a comment on pull request #29586: URL: https://github.com/apache/spark/pull/29586#issuecomment-693176982 **[Test build #128741 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128741/testReport)** for PR 29586 at commit [`95a3e02`](https://github.com/apache/spark/commit/95a3e028b2d1c83eaeab4baf15c07870df4a66ad). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29765: [SPARK-32888][DOCS] Add user document about header flag and RDD as path for reading CSV
AmplabJenkins removed a comment on pull request #29765: URL: https://github.com/apache/spark/pull/29765#issuecomment-693216775 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29586: [SPARK-32743][SQL] Add distinct info at UnresolvedFunction toString
AmplabJenkins commented on pull request #29586: URL: https://github.com/apache/spark/pull/29586#issuecomment-693217286 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #25965: [SPARK-26425][SS] Add more constraint checks to avoid checkpoint corruption
SparkQA removed a comment on pull request #25965: URL: https://github.com/apache/spark/pull/25965#issuecomment-693197739 **[Test build #128745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128745/testReport)** for PR 25965 at commit [`a885212`](https://github.com/apache/spark/commit/a8852123ae978dddb5578e3411b4822fd4a1fdce). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29767: [SPARK-32896][SS] Add DataStreamWriter.table API
AmplabJenkins commented on pull request #29767: URL: https://github.com/apache/spark/pull/29767#issuecomment-693217397 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29586: [SPARK-32743][SQL] Add distinct info at UnresolvedFunction toString
AmplabJenkins removed a comment on pull request #29586: URL: https://github.com/apache/spark/pull/29586#issuecomment-693217286 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
AmplabJenkins commented on pull request #29731: URL: https://github.com/apache/spark/pull/29731#issuecomment-693217476 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29769: [SPARK-32804][Launcher][FOLLOWUP] Fix SparkSubmitCommandBuilderSuite test failure without jars
AmplabJenkins removed a comment on pull request #29769: URL: https://github.com/apache/spark/pull/29769#issuecomment-693217239 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128742/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29765: [SPARK-32888][DOCS] Add user document about header flag and RDD as path for reading CSV
AmplabJenkins removed a comment on pull request #29765: URL: https://github.com/apache/spark/pull/29765#issuecomment-693216783 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128747/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #25965: [SPARK-26425][SS] Add more constraint checks to avoid checkpoint corruption
AmplabJenkins removed a comment on pull request #25965: URL: https://github.com/apache/spark/pull/25965#issuecomment-693216861 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128745/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29604: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
SparkQA commented on pull request #29604: URL: https://github.com/apache/spark/pull/29604#issuecomment-693217771 **[Test build #128749 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128749/testReport)** for PR 29604 at commit [`8778412`](https://github.com/apache/spark/commit/8778412afa9a50e9dace13f140bb7c253fc0dfc7). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29585: [SPARK-32741][SQL] Check if the same ExprId refers to the unique attribute in logical plans
AmplabJenkins commented on pull request #29585: URL: https://github.com/apache/spark/pull/29585#issuecomment-693217590 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29585: [SPARK-32741][SQL] Check if the same ExprId refers to the unique attribute in logical plans
AmplabJenkins removed a comment on pull request #29585: URL: https://github.com/apache/spark/pull/29585#issuecomment-693217590 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29767: [SPARK-32896][SS] Add DataStreamWriter.table API
AmplabJenkins removed a comment on pull request #29767: URL: https://github.com/apache/spark/pull/29767#issuecomment-693217397 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29604: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
AmplabJenkins commented on pull request #29604: URL: https://github.com/apache/spark/pull/29604#issuecomment-693217787 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29722: [SPARK-32850][CORE] Simplify the RPC message flow of decommission
AmplabJenkins removed a comment on pull request #29722: URL: https://github.com/apache/spark/pull/29722#issuecomment-693216769 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128748/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
AmplabJenkins removed a comment on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-693216829 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128746/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29739: [SPARK-32867][SQL] When explain, HiveTableRelation show limited message
AmplabJenkins removed a comment on pull request #29739: URL: https://github.com/apache/spark/pull/29739#issuecomment-693216947 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128743/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
AmplabJenkins removed a comment on pull request #29731: URL: https://github.com/apache/spark/pull/29731#issuecomment-693217476 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29585: [SPARK-32741][SQL] Check if the same ExprId refers to the unique attribute in logical plans
AmplabJenkins removed a comment on pull request #29585: URL: https://github.com/apache/spark/pull/29585#issuecomment-693217598 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128734/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29604: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
AmplabJenkins removed a comment on pull request #29604: URL: https://github.com/apache/spark/pull/29604#issuecomment-693217787 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
AmplabJenkins removed a comment on pull request #29731: URL: https://github.com/apache/spark/pull/29731#issuecomment-693217485 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128744/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29767: [SPARK-32896][SS] Add DataStreamWriter.table API
AmplabJenkins removed a comment on pull request #29767: URL: https://github.com/apache/spark/pull/29767#issuecomment-693217408 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128739/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29604: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
SparkQA removed a comment on pull request #29604: URL: https://github.com/apache/spark/pull/29604#issuecomment-693217111 **[Test build #128749 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128749/testReport)** for PR 29604 at commit [`8778412`](https://github.com/apache/spark/commit/8778412afa9a50e9dace13f140bb7c253fc0dfc7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
wangyum commented on pull request #29731: URL: https://github.com/apache/spark/pull/29731#issuecomment-693218201 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29586: [SPARK-32743][SQL] Add distinct info at UnresolvedFunction toString
AmplabJenkins removed a comment on pull request #29586: URL: https://github.com/apache/spark/pull/29586#issuecomment-693217294 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128741/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29316: [SPARK-32508][SQL] Disallow empty part col values in partition spec before static partition writing
AmplabJenkins commented on pull request #29316: URL: https://github.com/apache/spark/pull/29316#issuecomment-693218330 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.
AmplabJenkins removed a comment on pull request #29591: URL: https://github.com/apache/spark/pull/29591#issuecomment-693218508 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29604: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
AmplabJenkins removed a comment on pull request #29604: URL: https://github.com/apache/spark/pull/29604#issuecomment-693217800 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128749/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29316: [SPARK-32508][SQL] Disallow empty part col values in partition spec before static partition writing
AmplabJenkins removed a comment on pull request #29316: URL: https://github.com/apache/spark/pull/29316#issuecomment-693218330 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.
AmplabJenkins commented on pull request #29591: URL: https://github.com/apache/spark/pull/29591#issuecomment-693218508 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.
AmplabJenkins removed a comment on pull request #29591: URL: https://github.com/apache/spark/pull/29591#issuecomment-693218513 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128740/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on a change in pull request #29729: [SPARK-32032][SS] Avoid infinite wait in driver because of KafkaConsumer.poll(long) API
gaborgsomogyi commented on a change in pull request #29729:
URL: https://github.com/apache/spark/pull/29729#discussion_r489212011
##
File path:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
##
@@ -105,34 +109,16 @@ private[kafka010] class KafkaOffsetReader(
minPartitions.map(_ > numTopicPartitions).getOrElse(false)
}
- private def nextGroupId(): String = {
-groupId = driverGroupIdPrefix + "-" + nextId
-nextId += 1
-groupId
- }
-
override def toString(): String = consumerStrategy.toString
/**
* Closes the connection to Kafka, and cleans up state.
*/
def close(): Unit = {
-if (_consumer != null) uninterruptibleThreadRunner.runUninterruptibly {
stopConsumer() }
+if (_admin != null) uninterruptibleThreadRunner.runUninterruptibly {
stopAdmin() }
uninterruptibleThreadRunner.shutdown()
}
- /**
- * @return The Set of TopicPartitions for a given topic
- */
- def fetchTopicPartitions(): Set[TopicPartition] =
uninterruptibleThreadRunner.runUninterruptibly {
-assert(Thread.currentThread().isInstanceOf[UninterruptibleThread])
Review comment:
I've just finished the testing of `UninterruptibleThread` removal from
`KafkaOffsetReader`. We can put it here if we agree. Both side we can come up
w/ pros and cons.
> get rid of using UninterruptibleThread altogether.
Not sure it's possible or at least not super easy. In executor side it's
used because of multiple reasons. Please see:
https://github.com/apache/spark/blob/2e3aa2f0232a539346da3df8a20cd8e7c2b7dd4f/core/src/main/scala/org/apache/spark/executor/Executor.scala#L101-L104
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29316: [SPARK-32508][SQL] Disallow empty part col values in partition spec before static partition writing
AmplabJenkins removed a comment on pull request #29316: URL: https://github.com/apache/spark/pull/29316#issuecomment-693218342 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128735/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
SparkQA commented on pull request #29731: URL: https://github.com/apache/spark/pull/29731#issuecomment-693220302 **[Test build #128750 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128750/testReport)** for PR 29731 at commit [`abf2f7e`](https://github.com/apache/spark/commit/abf2f7e45b60f15742bf04c47ad1ff6d0ab130dc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
AmplabJenkins commented on pull request #29731: URL: https://github.com/apache/spark/pull/29731#issuecomment-693220879 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
AmplabJenkins removed a comment on pull request #29731: URL: https://github.com/apache/spark/pull/29731#issuecomment-693220879 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on a change in pull request #29729: [SPARK-32032][SS] Avoid infinite wait in driver because of KafkaConsumer.poll(long) API
gaborgsomogyi commented on a change in pull request #29729:
URL: https://github.com/apache/spark/pull/29729#discussion_r489215473
##
File path:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
##
@@ -105,34 +109,16 @@ private[kafka010] class KafkaOffsetReader(
minPartitions.map(_ > numTopicPartitions).getOrElse(false)
}
- private def nextGroupId(): String = {
-groupId = driverGroupIdPrefix + "-" + nextId
-nextId += 1
-groupId
- }
-
override def toString(): String = consumerStrategy.toString
/**
* Closes the connection to Kafka, and cleans up state.
*/
def close(): Unit = {
-if (_consumer != null) uninterruptibleThreadRunner.runUninterruptibly {
stopConsumer() }
+if (_admin != null) uninterruptibleThreadRunner.runUninterruptibly {
stopAdmin() }
uninterruptibleThreadRunner.shutdown()
}
- /**
- * @return The Set of TopicPartitions for a given topic
- */
- def fetchTopicPartitions(): Set[TopicPartition] =
uninterruptibleThreadRunner.runUninterruptibly {
-assert(Thread.currentThread().isInstanceOf[UninterruptibleThread])
Review comment:
My rationale related removing here vs removing in separate PR is the
following: I personally don't see a reason why the removal could cause any
issues but to be defensive we can put it into a separate PR. This way if any
issue appears for mystical reason we can rollback that PR independently. Does
this make sense?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #29729: [SPARK-32032][SS] Avoid infinite wait in driver because of KafkaConsumer.poll(long) API
HeartSaVioR commented on a change in pull request #29729:
URL: https://github.com/apache/spark/pull/29729#discussion_r489215803
##
File path:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
##
@@ -105,34 +109,16 @@ private[kafka010] class KafkaOffsetReader(
minPartitions.map(_ > numTopicPartitions).getOrElse(false)
}
- private def nextGroupId(): String = {
-groupId = driverGroupIdPrefix + "-" + nextId
-nextId += 1
-groupId
- }
-
override def toString(): String = consumerStrategy.toString
/**
* Closes the connection to Kafka, and cleans up state.
*/
def close(): Unit = {
-if (_consumer != null) uninterruptibleThreadRunner.runUninterruptibly {
stopConsumer() }
+if (_admin != null) uninterruptibleThreadRunner.runUninterruptibly {
stopAdmin() }
uninterruptibleThreadRunner.shutdown()
}
- /**
- * @return The Set of TopicPartitions for a given topic
- */
- def fetchTopicPartitions(): Set[TopicPartition] =
uninterruptibleThreadRunner.runUninterruptibly {
-assert(Thread.currentThread().isInstanceOf[UninterruptibleThread])
Review comment:
Sorry I meant Kafka related, driver side only.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #29729: [SPARK-32032][SS] Avoid infinite wait in driver because of KafkaConsumer.poll(long) API
HeartSaVioR commented on a change in pull request #29729:
URL: https://github.com/apache/spark/pull/29729#discussion_r489215803
##
File path:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
##
@@ -105,34 +109,16 @@ private[kafka010] class KafkaOffsetReader(
minPartitions.map(_ > numTopicPartitions).getOrElse(false)
}
- private def nextGroupId(): String = {
-groupId = driverGroupIdPrefix + "-" + nextId
-nextId += 1
-groupId
- }
-
override def toString(): String = consumerStrategy.toString
/**
* Closes the connection to Kafka, and cleans up state.
*/
def close(): Unit = {
-if (_consumer != null) uninterruptibleThreadRunner.runUninterruptibly {
stopConsumer() }
+if (_admin != null) uninterruptibleThreadRunner.runUninterruptibly {
stopAdmin() }
uninterruptibleThreadRunner.shutdown()
}
- /**
- * @return The Set of TopicPartitions for a given topic
- */
- def fetchTopicPartitions(): Set[TopicPartition] =
uninterruptibleThreadRunner.runUninterruptibly {
-assert(Thread.currentThread().isInstanceOf[UninterruptibleThread])
Review comment:
Sorry I meant Kafka related, driver side only. (In executors we still
leverage KafkaConsumer so not 100% sure we can get rid of it. Though still
better to get rid if possible.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #29729: [SPARK-32032][SS] Avoid infinite wait in driver because of KafkaConsumer.poll(long) API
HeartSaVioR commented on a change in pull request #29729:
URL: https://github.com/apache/spark/pull/29729#discussion_r489217921
##
File path:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
##
@@ -105,34 +109,16 @@ private[kafka010] class KafkaOffsetReader(
minPartitions.map(_ > numTopicPartitions).getOrElse(false)
}
- private def nextGroupId(): String = {
-groupId = driverGroupIdPrefix + "-" + nextId
-nextId += 1
-groupId
- }
-
override def toString(): String = consumerStrategy.toString
/**
* Closes the connection to Kafka, and cleans up state.
*/
def close(): Unit = {
-if (_consumer != null) uninterruptibleThreadRunner.runUninterruptibly {
stopConsumer() }
+if (_admin != null) uninterruptibleThreadRunner.runUninterruptibly {
stopAdmin() }
uninterruptibleThreadRunner.shutdown()
}
- /**
- * @return The Set of TopicPartitions for a given topic
- */
- def fetchTopicPartitions(): Set[TopicPartition] =
uninterruptibleThreadRunner.runUninterruptibly {
-assert(Thread.currentThread().isInstanceOf[UninterruptibleThread])
Review comment:
Having a separate PR is OK for me - that would avoid unnecessary hold on
this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on a change in pull request #29729: [SPARK-32032][SS] Avoid infinite wait in driver because of KafkaConsumer.poll(long) API
gaborgsomogyi commented on a change in pull request #29729:
URL: https://github.com/apache/spark/pull/29729#discussion_r489218487
##
File path:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
##
@@ -105,34 +109,16 @@ private[kafka010] class KafkaOffsetReader(
minPartitions.map(_ > numTopicPartitions).getOrElse(false)
}
- private def nextGroupId(): String = {
-groupId = driverGroupIdPrefix + "-" + nextId
-nextId += 1
-groupId
- }
-
override def toString(): String = consumerStrategy.toString
/**
* Closes the connection to Kafka, and cleans up state.
*/
def close(): Unit = {
-if (_consumer != null) uninterruptibleThreadRunner.runUninterruptibly {
stopConsumer() }
+if (_admin != null) uninterruptibleThreadRunner.runUninterruptibly {
stopAdmin() }
uninterruptibleThreadRunner.shutdown()
}
- /**
- * @return The Set of TopicPartitions for a given topic
- */
- def fetchTopicPartitions(): Set[TopicPartition] =
uninterruptibleThreadRunner.runUninterruptibly {
-assert(Thread.currentThread().isInstanceOf[UninterruptibleThread])
Review comment:
I agree w/ this way.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #25965: [SPARK-26425][SS] Add more constraint checks to avoid checkpoint corruption
HeartSaVioR commented on pull request #25965: URL: https://github.com/apache/spark/pull/25965#issuecomment-693225068 retest this, please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
HeartSaVioR commented on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-693225135 retest this, please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #29767: [SPARK-32896][SS] Add DataStreamWriter.table API
HeartSaVioR commented on pull request #29767: URL: https://github.com/apache/spark/pull/29767#issuecomment-693225169 retest this, please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
SparkQA commented on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-693226793 **[Test build #128752 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128752/testReport)** for PR 28841 at commit [`1c8384c`](https://github.com/apache/spark/commit/1c8384cebe05fbc1ebb3c86660e57508b031c6f9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29767: [SPARK-32896][SS] Add DataStreamWriter.table API
SparkQA commented on pull request #29767: URL: https://github.com/apache/spark/pull/29767#issuecomment-693226706 **[Test build #128751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128751/testReport)** for PR 29767 at commit [`f557696`](https://github.com/apache/spark/commit/f5576969ee9afeaa35c3bb74822aad43cf9c11c2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #25965: [SPARK-26425][SS] Add more constraint checks to avoid checkpoint corruption
SparkQA commented on pull request #25965: URL: https://github.com/apache/spark/pull/25965#issuecomment-693226799 **[Test build #128753 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128753/testReport)** for PR 25965 at commit [`a885212`](https://github.com/apache/spark/commit/a8852123ae978dddb5578e3411b4822fd4a1fdce). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #25965: [SPARK-26425][SS] Add more constraint checks to avoid checkpoint corruption
AmplabJenkins commented on pull request #25965: URL: https://github.com/apache/spark/pull/25965#issuecomment-693227427 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
AmplabJenkins commented on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-693227460 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29767: [SPARK-32896][SS] Add DataStreamWriter.table API
AmplabJenkins commented on pull request #29767: URL: https://github.com/apache/spark/pull/29767#issuecomment-693227405 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on a change in pull request #29729: [SPARK-32032][SS] Avoid infinite wait in driver because of KafkaConsumer.poll(long) API
gaborgsomogyi commented on a change in pull request #29729:
URL: https://github.com/apache/spark/pull/29729#discussion_r489221735
##
File path:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
##
@@ -213,64 +187,68 @@ private[kafka010] class KafkaOffsetReader(
assert(partitions.asScala == partitionTimestamps.keySet,
"If starting/endingOffsetsByTimestamp contains specific offsets, you
must specify all " +
s"topics. Specified: ${partitionTimestamps.keySet} Assigned:
${partitions.asScala}")
- logDebug(s"Partitions assigned to consumer: $partitions. Seeking to
$partitionTimestamps")
+ logDebug(s"Assigned partitions: $partitions. Seeking to
$partitionTimestamps")
}
val fnRetrievePartitionOffsets: ju.Set[TopicPartition] =>
Map[TopicPartition, Long] = { _ => {
-val converted = partitionTimestamps.map { case (tp, timestamp) =>
- tp -> java.lang.Long.valueOf(timestamp)
+val listOffsetsParams = partitionTimestamps.map { case (tp, timestamp)
=>
+ tp -> OffsetSpec.forTimestamp(timestamp)
}.asJava
+admin.listOffsets(listOffsetsParams,
listOffsetsOptions).all().get().asScala.map {
+ case (tp, offsetSpec) =>
+if (failsOnNoMatchingOffset) {
+ assert(offsetSpec.offset() !=
OffsetFetchResponse.INVALID_OFFSET, "No offset " +
+s"matched from request of topic-partition $tp and timestamp " +
+s"${partitionTimestamps(tp)}.")
+}
-val offsetForTime: ju.Map[TopicPartition, OffsetAndTimestamp] =
- consumer.offsetsForTimes(converted)
-
-offsetForTime.asScala.map { case (tp, offsetAndTimestamp) =>
- if (failsOnNoMatchingOffset) {
-assert(offsetAndTimestamp != null, "No offset matched from request
of " +
- s"topic-partition $tp and timestamp ${partitionTimestamps(tp)}.")
- }
-
- if (offsetAndTimestamp == null) {
-tp -> KafkaOffsetRangeLimit.LATEST
- } else {
-tp -> offsetAndTimestamp.offset()
- }
+if (offsetSpec.offset() == OffsetFetchResponse.INVALID_OFFSET) {
+ tp -> KafkaOffsetRangeLimit.LATEST
+} else {
+ tp -> offsetSpec.offset()
+}
}.toMap
}
}
-val fnAssertFetchedOffsets: Map[TopicPartition, Long] => Unit = { _ => }
-
-fetchSpecificOffsets0(fnAssertParametersWithPartitions,
fnRetrievePartitionOffsets,
- fnAssertFetchedOffsets)
+fetchSpecificOffsets0(fnAssertParametersWithPartitions,
fnRetrievePartitionOffsets)
}
private def fetchSpecificOffsets0(
fnAssertParametersWithPartitions: ju.Set[TopicPartition] => Unit,
- fnRetrievePartitionOffsets: ju.Set[TopicPartition] =>
Map[TopicPartition, Long],
- fnAssertFetchedOffsets: Map[TopicPartition, Long] => Unit):
KafkaSourceOffset = {
+ fnRetrievePartitionOffsets: ju.Set[TopicPartition] =>
Map[TopicPartition, Long]
Review comment:
It mustn't be unless the code you've pinpointed can fire up somehow:
https://github.com/apache/spark/pull/29729#discussion_r488831782
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29767: [SPARK-32896][SS] Add DataStreamWriter.table API
AmplabJenkins removed a comment on pull request #29767: URL: https://github.com/apache/spark/pull/29767#issuecomment-693227405 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
AmplabJenkins removed a comment on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-693227460 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #25965: [SPARK-26425][SS] Add more constraint checks to avoid checkpoint corruption
AmplabJenkins removed a comment on pull request #25965: URL: https://github.com/apache/spark/pull/25965#issuecomment-693227427 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on a change in pull request #29729: [SPARK-32032][SS] Avoid infinite wait in driver because of KafkaConsumer.poll(long) API
gaborgsomogyi commented on a change in pull request #29729:
URL: https://github.com/apache/spark/pull/29729#discussion_r489222628
##
File path:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
##
@@ -182,28 +170,14 @@ private[kafka010] class KafkaOffsetReader(
"If startingOffsets contains specific offsets, you must specify all
TopicPartitions.\n" +
"Use -1 for latest, -2 for earliest, if you don't care.\n" +
s"Specified: ${partitionOffsets.keySet} Assigned:
${partitions.asScala}")
- logDebug(s"Partitions assigned to consumer: $partitions. Seeking to
$partitionOffsets")
+ logDebug(s"Assigned partitions: $partitions. Seeking to
$partitionOffsets")
}
val fnRetrievePartitionOffsets: ju.Set[TopicPartition] =>
Map[TopicPartition, Long] = { _ =>
partitionOffsets
}
-val fnAssertFetchedOffsets: Map[TopicPartition, Long] => Unit = { fetched
=>
- partitionOffsets.foreach {
-case (tp, off) if off != KafkaOffsetRangeLimit.LATEST &&
- off != KafkaOffsetRangeLimit.EARLIEST =>
- if (fetched(tp) != off) {
-reportDataLoss(
Review comment:
As mentioned in the doc already but putting it here as well this
condition was never covered w/ testcases. Maybe never worked?! Not sure...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29604: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
cloud-fan commented on a change in pull request #29604:
URL: https://github.com/apache/spark/pull/29604#discussion_r489227880
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
##
@@ -476,7 +476,7 @@ case class Lag(input: Expression, offset: Expression,
default: Expression)
abstract class AggregateWindowFunction extends DeclarativeAggregate with
WindowFunction {
self: Product =>
- override val frame = SpecifiedWindowFrame(RowFrame, UnboundedPreceding,
CurrentRow)
+ override def frame: WindowFrame = SpecifiedWindowFrame(RowFrame,
UnboundedPreceding, CurrentRow)
Review comment:
is it a necessary change?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29585: [SPARK-32741][SQL] Check if the same ExprId refers to the unique attribute in logical plans
maropu commented on pull request #29585: URL: https://github.com/apache/spark/pull/29585#issuecomment-693232942 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29585: [SPARK-32741][SQL] Check if the same ExprId refers to the unique attribute in logical plans
SparkQA commented on pull request #29585: URL: https://github.com/apache/spark/pull/29585#issuecomment-693233512 **[Test build #128754 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128754/testReport)** for PR 29585 at commit [`fb048c7`](https://github.com/apache/spark/commit/fb048c7afcc6e366f8e840a3e0a02849a129a04c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
