[GitHub] [spark] AmplabJenkins removed a comment on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
AmplabJenkins removed a comment on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-524197015 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14672/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
AmplabJenkins removed a comment on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-524197003 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
AmplabJenkins commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-524197015 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14672/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
AmplabJenkins commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-524197003 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
cloud-fan commented on a change in pull request #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#discussion_r316996274 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala ## @@ -258,6 +258,7 @@ case class Cast(child: Expression, dataType: DataType, timeZoneId: Option[String private lazy val dateFormatter = DateFormatter() private lazy val timestampFormatter = TimestampFormatter.getFractionFormatter(zoneId) + private val failOnIntegerOverflow = SQLConf.get.failOnIntegralTypeOverflow Review comment: to be consistent, shall we also rename it to `failOnIntegralTypeOverflow`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
beliefer commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-524196353 @HyukjinKwon Thanks for your review and help me to promote this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25565: [SPARK-28025][SS][BRANCH-2.4] Fix FileContextBasedCheckpointFileManager leaking c…
AmplabJenkins removed a comment on issue #25565: [SPARK-28025][SS][BRANCH-2.4] Fix FileContextBasedCheckpointFileManager leaking c… URL: https://github.com/apache/spark/pull/25565#issuecomment-524194847 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
SparkQA commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-524195487 **[Test build #109621 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109621/testReport)** for PR 25001 at commit [`10f42f4`](https://github.com/apache/spark/commit/10f42f423c4af46d83eb3c678723600da7386e8e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25565: [SPARK-28025][SS][BRANCH-2.4] Fix FileContextBasedCheckpointFileManager leaking c…
SparkQA commented on issue #25565: [SPARK-28025][SS][BRANCH-2.4] Fix FileContextBasedCheckpointFileManager leaking c… URL: https://github.com/apache/spark/pull/25565#issuecomment-524195479 **[Test build #109619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109619/testReport)** for PR 25565 at commit [`c40f8a1`](https://github.com/apache/spark/commit/c40f8a1a6ba12dd51e270e71594450c8e0df8dce). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
SparkQA commented on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize URL: https://github.com/apache/spark/pull/25309#issuecomment-524195489 **[Test build #109620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109620/testReport)** for PR 25309 at commit [`40ad336`](https://github.com/apache/spark/commit/40ad336fe1bda961d223b8a9bc666fd636c6b99b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
AmplabJenkins removed a comment on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize URL: https://github.com/apache/spark/pull/25309#issuecomment-524194899 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109618/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
beliefer commented on a change in pull request #25001: [SPARK-28083][SQL]
Support LIKE ... ESCAPE syntax
URL: https://github.com/apache/spark/pull/25001#discussion_r316994509
##
File path: sql/core/src/main/scala/org/apache/spark/sql/Column.scala
##
@@ -822,6 +822,16 @@ class Column(val expr: Expression) extends Logging {
*/
def like(literal: String): Column = withExpr { Like(expr, lit(literal).expr)
}
+ /**
+ * SQL like expression. Returns a boolean column based on a SQL LIKE ESCAPE
match.
+ *
+ * @group expr_ops
+ * @since 3.0.0
+ */
+ def like(literal: String, escapeStr: String): Column = withExpr {
+Like(expr, lit(literal).expr, Option(escapeStr))
+ }
+
Review comment:
OK.I will remove it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25565: [SPARK-28025][SS][BRANCH-2.4] Fix FileContextBasedCheckpointFileManager leaking c…
AmplabJenkins removed a comment on issue #25565: [SPARK-28025][SS][BRANCH-2.4] Fix FileContextBasedCheckpointFileManager leaking c… URL: https://github.com/apache/spark/pull/25565#issuecomment-524194676 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
SparkQA removed a comment on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize URL: https://github.com/apache/spark/pull/25309#issuecomment-524189709 **[Test build #109618 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109618/testReport)** for PR 25309 at commit [`8f37ca1`](https://github.com/apache/spark/commit/8f37ca13c2bd803e75151fe5e701c2f1a80d4478). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
LuciferYang commented on a change in pull request #25309:
[SPARK-28577][YARN]Resource capability requested for each executor add
offHeapMemorySize
URL: https://github.com/apache/spark/pull/25309#discussion_r316994188
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
##
@@ -184,4 +183,18 @@ object YarnSparkHadoopUtil {
ConverterUtils.toContainerId(containerIdString)
}
+ /**
+ * Convert MEMORY_OFFHEAP_SIZE to MB Unit, return 0 if
MEMORY_OFFHEAP_ENABLED is false.
+ */
+ def executorOffHeapMemorySizeAsMb(sparkConf: SparkConf): Int = {
+if (sparkConf.get(MEMORY_OFFHEAP_ENABLED)) {
+ val sizeInMB =
+sparkConf.getSizeAsMb(MEMORY_OFFHEAP_SIZE.key,
MEMORY_OFFHEAP_SIZE.defaultValueString).toInt
Review comment:
@tgravescs Refer to api comments,
We have to use `s"${sparkConf.get(MEMORY_OFFHEAP_SIZE)}B"` as input to call
`Utils.byteStringAsMb(str: String)` because `byte string should 50b, 100k, or
250m`,looks a little strange, It feels like the old version looks more
comfortable.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
LuciferYang commented on a change in pull request #25309:
[SPARK-28577][YARN]Resource capability requested for each executor add
offHeapMemorySize
URL: https://github.com/apache/spark/pull/25309#discussion_r316994188
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
##
@@ -184,4 +183,18 @@ object YarnSparkHadoopUtil {
ConverterUtils.toContainerId(containerIdString)
}
+ /**
+ * Convert MEMORY_OFFHEAP_SIZE to MB Unit, return 0 if
MEMORY_OFFHEAP_ENABLED is false.
+ */
+ def executorOffHeapMemorySizeAsMb(sparkConf: SparkConf): Int = {
+if (sparkConf.get(MEMORY_OFFHEAP_ENABLED)) {
+ val sizeInMB =
+sparkConf.getSizeAsMb(MEMORY_OFFHEAP_SIZE.key,
MEMORY_OFFHEAP_SIZE.defaultValueString).toInt
Review comment:
@tgravescs Refer to api comments,
We have to use `s"${sparkConf.get(MEMORY_OFFHEAP_SIZE)}B"` as input to call
`Utils.byteStringAsMb(str: String)` because `byte string should 50b, 100k, or
250m`,looks a little strange.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
AmplabJenkins removed a comment on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize URL: https://github.com/apache/spark/pull/25309#issuecomment-524194894 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
AmplabJenkins commented on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize URL: https://github.com/apache/spark/pull/25309#issuecomment-524194894 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
AmplabJenkins commented on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize URL: https://github.com/apache/spark/pull/25309#issuecomment-524194899 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109618/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
LuciferYang commented on a change in pull request #25309:
[SPARK-28577][YARN]Resource capability requested for each executor add
offHeapMemorySize
URL: https://github.com/apache/spark/pull/25309#discussion_r316994188
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
##
@@ -184,4 +183,18 @@ object YarnSparkHadoopUtil {
ConverterUtils.toContainerId(containerIdString)
}
+ /**
+ * Convert MEMORY_OFFHEAP_SIZE to MB Unit, return 0 if
MEMORY_OFFHEAP_ENABLED is false.
+ */
+ def executorOffHeapMemorySizeAsMb(sparkConf: SparkConf): Int = {
+if (sparkConf.get(MEMORY_OFFHEAP_ENABLED)) {
+ val sizeInMB =
+sparkConf.getSizeAsMb(MEMORY_OFFHEAP_SIZE.key,
MEMORY_OFFHEAP_SIZE.defaultValueString).toInt
Review comment:
@tgravescs ref api comments,
We have to use `s"${sparkConf.get(MEMORY_OFFHEAP_SIZE)}B"` as input to call
`Utils.byteStringAsMb(str: String)` because `byte string should 50b, 100k, or
250m`,looks a little strange.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
SparkQA commented on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize URL: https://github.com/apache/spark/pull/25309#issuecomment-524194871 **[Test build #109618 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109618/testReport)** for PR 25309 at commit [`8f37ca1`](https://github.com/apache/spark/commit/8f37ca13c2bd803e75151fe5e701c2f1a80d4478). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25565: [SPARK-28025][SS][BRANCH-2.4] Fix FileContextBasedCheckpointFileManager leaking c…
AmplabJenkins commented on issue #25565: [SPARK-28025][SS][BRANCH-2.4] Fix FileContextBasedCheckpointFileManager leaking c… URL: https://github.com/apache/spark/pull/25565#issuecomment-524194847 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25565: [SPARK-28025][SS][BRANCH-2.4] Fix FileContextBasedCheckpointFileManager leaking c…
AmplabJenkins commented on issue #25565: [SPARK-28025][SS][BRANCH-2.4] Fix FileContextBasedCheckpointFileManager leaking c… URL: https://github.com/apache/spark/pull/25565#issuecomment-524194676 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #25565: [SPARK-28025][SS][BRANCH-2.4] Fix FileContextBasedCheckpointFileManager leaking c…
HeartSaVioR commented on issue #25565: [SPARK-28025][SS][BRANCH-2.4] Fix FileContextBasedCheckpointFileManager leaking c… URL: https://github.com/apache/spark/pull/25565#issuecomment-524194578 cc. @zsxwing This is ported version of #25488 for branch-2.4 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR opened a new pull request #25565: [SPARK-28025][SS][BRANCH-2.4] Fix FileContextBasedCheckpointFileManager leaking c…
HeartSaVioR opened a new pull request #25565: [SPARK-28025][SS][BRANCH-2.4] Fix FileContextBasedCheckpointFileManager leaking c… URL: https://github.com/apache/spark/pull/25565 ### What changes were proposed in this pull request? This PR fixes the leak of crc files from CheckpointFileManager when FileContextBasedCheckpointFileManager is being used. Spark hits the Hadoop bug, [HADOOP-16255](https://issues.apache.org/jira/browse/HADOOP-16255) which seems to be a long-standing issue. This is there're two `renameInternal` methods: ``` public void renameInternal(Path src, Path dst) public void renameInternal(final Path src, final Path dst, boolean overwrite) ``` which should be overridden to handle all cases but ChecksumFs only overrides method with 2 params, so when latter is called FilterFs.renameInternal(...) is called instead, and it will do rename with RawLocalFs as underlying filesystem. The bug is related to FileContext, so FileSystemBasedCheckpointFileManager is not affected. [SPARK-17475](https://issues.apache.org/jira/browse/SPARK-17475) took a workaround for this bug, but [SPARK-23966](https://issues.apache.org/jira/browse/SPARK-23966) seemed to bring regression. This PR deletes crc file as "best-effort" when renaming, as failing to delete crc file is not that critical to fail the task. ### Why are the changes needed? This PR prevents crc files not being cleaned up even purging batches. Too many files in same directory often hurts performance, as well as each crc file occupies more space than its own size so possible to occupy nontrivial amount of space when batches go up to 10+. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Some unit tests are modified to check leakage of crc files. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #25488: [SPARK-28025][SS] Fix FileContextBasedCheckpointFileManager leaking crc files
HeartSaVioR commented on issue #25488: [SPARK-28025][SS] Fix FileContextBasedCheckpointFileManager leaking crc files URL: https://github.com/apache/spark/pull/25488#issuecomment-524193744 Thanks for the quick reviewing and merging! I'll raise PR for branch-2.4 soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-524191105 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14671/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-524191100 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
AmplabJenkins commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-524191100 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
AmplabJenkins commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-524191105 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14671/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging
dongjoon-hyun commented on issue #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging URL: https://github.com/apache/spark/pull/25564#issuecomment-524190957 Thank you for review and approval, @cloud-fan ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
LuciferYang commented on a change in pull request #25309:
[SPARK-28577][YARN]Resource capability requested for each executor add
offHeapMemorySize
URL: https://github.com/apache/spark/pull/25309#discussion_r316989995
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
##
@@ -184,4 +183,18 @@ object YarnSparkHadoopUtil {
ConverterUtils.toContainerId(containerIdString)
}
+ /**
+ * Convert MEMORY_OFFHEAP_SIZE to MB Unit, return 0 if
MEMORY_OFFHEAP_ENABLED is false.
+ */
+ def executorOffHeapMemorySizeAsMb(sparkConf: SparkConf): Int = {
+if (sparkConf.get(MEMORY_OFFHEAP_ENABLED)) {
+ val sizeInMB =
+sparkConf.getSizeAsMb(MEMORY_OFFHEAP_SIZE.key,
MEMORY_OFFHEAP_SIZE.defaultValueString).toInt
+ require(sizeInMB > 0,
+s"${MEMORY_OFFHEAP_SIZE.key} must be > 0 when
${MEMORY_OFFHEAP_ENABLED.key} == true")
Review comment:
OK~ I will add a new JIRA to discuss this issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
LuciferYang commented on a change in pull request #25309:
[SPARK-28577][YARN]Resource capability requested for each executor add
offHeapMemorySize
URL: https://github.com/apache/spark/pull/25309#discussion_r316989995
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
##
@@ -184,4 +183,18 @@ object YarnSparkHadoopUtil {
ConverterUtils.toContainerId(containerIdString)
}
+ /**
+ * Convert MEMORY_OFFHEAP_SIZE to MB Unit, return 0 if
MEMORY_OFFHEAP_ENABLED is false.
+ */
+ def executorOffHeapMemorySizeAsMb(sparkConf: SparkConf): Int = {
+if (sparkConf.get(MEMORY_OFFHEAP_ENABLED)) {
+ val sizeInMB =
+sparkConf.getSizeAsMb(MEMORY_OFFHEAP_SIZE.key,
MEMORY_OFFHEAP_SIZE.defaultValueString).toInt
+ require(sizeInMB > 0,
+s"${MEMORY_OFFHEAP_SIZE.key} must be > 0 when
${MEMORY_OFFHEAP_ENABLED.key} == true")
Review comment:
OK~ I will add a new jira to discuss this issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
LuciferYang commented on a change in pull request #25309:
[SPARK-28577][YARN]Resource capability requested for each executor add
offHeapMemorySize
URL: https://github.com/apache/spark/pull/25309#discussion_r316989260
##
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
##
@@ -184,4 +183,18 @@ object YarnSparkHadoopUtil {
ConverterUtils.toContainerId(containerIdString)
}
+ /**
+ * Convert MEMORY_OFFHEAP_SIZE to MB Unit, return 0 if
MEMORY_OFFHEAP_ENABLED is false.
+ */
+ def executorOffHeapMemorySizeAsMb(sparkConf: SparkConf): Int = {
+if (sparkConf.get(MEMORY_OFFHEAP_ENABLED)) {
+ val sizeInMB =
+sparkConf.getSizeAsMb(MEMORY_OFFHEAP_SIZE.key,
MEMORY_OFFHEAP_SIZE.defaultValueString).toInt
Review comment:
[8f37ca1](https://github.com/apache/spark/pull/25309/commits/8f37ca13c2bd803e75151fe5e701c2f1a80d4478)
change to use
`Utils.byteStringAsMb(sparkConf.get(MEMORY_OFFHEAP_SIZE).toString).toInt` to
conversion config value to sizeInMB.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize
SparkQA commented on issue #25309: [SPARK-28577][YARN]Resource capability requested for each executor add offHeapMemorySize URL: https://github.com/apache/spark/pull/25309#issuecomment-524189709 **[Test build #109618 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109618/testReport)** for PR 25309 at commit [`8f37ca1`](https://github.com/apache/spark/commit/8f37ca13c2bd803e75151fe5e701c2f1a80d4478). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
SparkQA commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-524189678 **[Test build #109617 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109617/testReport)** for PR 25461 at commit [`f1c64e1`](https://github.com/apache/spark/commit/f1c64e1af10acd4a521ea4577d78070377ebd13a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25535: [SPARK-28835][SQL][TEST] Add TPCDSSchema trait
dongjoon-hyun commented on issue #25535: [SPARK-28835][SQL][TEST] Add TPCDSSchema trait URL: https://github.com/apache/spark/pull/25535#issuecomment-524188660 Since this is merged already, I'll switch SPARK-28835 and SPARK-28836 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
AmplabJenkins removed a comment on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-524188411 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
AmplabJenkins removed a comment on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-524188414 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109608/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
AmplabJenkins commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-524188411 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
AmplabJenkins commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-524188414 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109608/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25535: [SPARK-28835][SQL][TEST] Add TPCDSSchema trait
dongjoon-hyun commented on issue #25535: [SPARK-28835][SQL][TEST] Add TPCDSSchema trait URL: https://github.com/apache/spark/pull/25535#issuecomment-524188168 Oops. @dbaliafroozeh . It seems that you use a wrong JIRA id here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
cloud-fan closed pull request #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25247 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
SparkQA removed a comment on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-524148840 **[Test build #109608 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109608/testReport)** for PR 25001 at commit [`aa0f2f7`](https://github.com/apache/spark/commit/aa0f2f7469637c9bd5d16e8c64afae34cab9d274). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax
SparkQA commented on issue #25001: [SPARK-28083][SQL] Support LIKE ... ESCAPE syntax URL: https://github.com/apache/spark/pull/25001#issuecomment-524187981 **[Test build #109608 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109608/testReport)** for PR 25001 at commit [`aa0f2f7`](https://github.com/apache/spark/commit/aa0f2f7469637c9bd5d16e8c64afae34cab9d274). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #25535: [SPARK-28835][SQL][TEST] Add TPCDSSchema trait
dongjoon-hyun closed pull request #25535: [SPARK-28835][SQL][TEST] Add TPCDSSchema trait URL: https://github.com/apache/spark/pull/25535 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on issue #25563: Revert "[SPARK-25474][SQL] Support `spark.sql.statistics.fallBackToHdfs` in data source tables"
wangyum commented on issue #25563: Revert "[SPARK-25474][SQL] Support `spark.sql.statistics.fallBackToHdfs` in data source tables" URL: https://github.com/apache/spark/pull/25563#issuecomment-524187453 @shahidki31 Could you raise a PR to document it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25535: [SPARK-28835][SQL][TEST] Add TPCDSSchema trait
dongjoon-hyun commented on a change in pull request #25535:
[SPARK-28835][SQL][TEST] Add TPCDSSchema trait
URL: https://github.com/apache/spark/pull/25535#discussion_r316967767
##
File path: sql/core/src/test/scala/org/apache/spark/sql/TPCDSSchema.scala
##
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+trait TPCDSSchema {
+
+ private val tableColumns = Map(
+"call_center" ->
Review comment:
~Thanks. Could you include the link to the PR description, too? It's always
good to explain the way of your thought explicitly to the reviewers. It will be
included to the commit log.~ Never mind.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zsxwing commented on issue #25488: [SPARK-28025][SS] Fix FileContextBasedCheckpointFileManager leaking crc files
zsxwing commented on issue #25488: [SPARK-28025][SS] Fix FileContextBasedCheckpointFileManager leaking crc files URL: https://github.com/apache/spark/pull/25488#issuecomment-524186178 @HeartSaVioR Thanks! I merged to master. Could you submit a backport PR for branch-2.4? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] asfgit closed pull request #25488: [SPARK-28025][SS] Fix FileContextBasedCheckpointFileManager leaking crc files
asfgit closed pull request #25488: [SPARK-28025][SS] Fix FileContextBasedCheckpointFileManager leaking crc files URL: https://github.com/apache/spark/pull/25488 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
cloud-fan commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25247#issuecomment-524186032 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on a change in pull request #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
dilipbiswal commented on a change in pull request #25525:
[SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
URL: https://github.com/apache/spark/pull/25525#discussion_r316985567
##
File path: docs/sql-ref-syntax-dml-insert-into.md
##
@@ -0,0 +1,65 @@
+---
+layout: global
+title: INSERT INTO
+displayTitle: INSERT INTO
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+### Description
+
+The `INSERT INTO` statement inserts new rows into a table. The inserted rows
can be specified by value expressions, or resulted from a query.
+
+### Syntax
+{% highlight sql %}
+INSERT INTO [TABLE] [db_name.]table_name [partition_spec] value_clause | query
+
+partition_spec:
+: (part_col_name1[=val1] [, part_col_name2[=val2], ...])
+
+value_clause:
+: VALUES values_row [, values_row ...]
+
+values_row:
+: (val1 [, val2, ...])
+
+{% endhighlight %}
+
+### Examples
+{% highlight sql %}
+ CREATE TABLE employees (name VARCHAR(64), age INT, salary DECIMAL(9,2))
+ USING PARQUET PARTITIONED BY (age)
+
+ INSERT INTO employees
+ VALUES ('Amy Smith', 35, 20.00)
+
+ INSERT INTO employees PARTITION (age = 35)
+ SELECT * FROM candidates WHERE name = "Bob Doe"
+{% endhighlight %}
+
+### Parameters
+
+ ***table_name***:
+The name of an existing table.
+
+ ***partition_spec***:
+Partition column specification.
+
+ ***value_clause***:
+Specify the values to be inserted.
+
+ ***query***:
+A `SELECT` statement that provides the rows to be inserted.
Review comment:
@huaxingao would `produces` sound better ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on a change in pull request #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
dilipbiswal commented on a change in pull request #25525:
[SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
URL: https://github.com/apache/spark/pull/25525#discussion_r316985216
##
File path: docs/sql-ref-syntax-dml-insert-into.md
##
@@ -0,0 +1,65 @@
+---
+layout: global
+title: INSERT INTO
+displayTitle: INSERT INTO
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+### Description
+
+The `INSERT INTO` statement inserts new rows into a table. The inserted rows
can be specified by value expressions, or resulted from a query.
+
+### Syntax
+{% highlight sql %}
+INSERT INTO [TABLE] [db_name.]table_name [partition_spec] value_clause | query
Review comment:
@huaxingao Can you please check the grammar. I think we allow more cases ..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on a change in pull request #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
dilipbiswal commented on a change in pull request #25525:
[SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
URL: https://github.com/apache/spark/pull/25525#discussion_r316985359
##
File path: docs/sql-ref-syntax-dml-insert-into.md
##
@@ -0,0 +1,65 @@
+---
+layout: global
+title: INSERT INTO
+displayTitle: INSERT INTO
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+### Description
+
+The `INSERT INTO` statement inserts new rows into a table. The inserted rows
can be specified by value expressions, or resulted from a query.
+
+### Syntax
+{% highlight sql %}
+INSERT INTO [TABLE] [db_name.]table_name [partition_spec] value_clause | query
+
+partition_spec:
+: (part_col_name1[=val1] [, part_col_name2[=val2], ...])
+
+value_clause:
+: VALUES values_row [, values_row ...]
+
+values_row:
+: (val1 [, val2, ...])
+
+{% endhighlight %}
+
+### Examples
+{% highlight sql %}
+ CREATE TABLE employees (name VARCHAR(64), age INT, salary DECIMAL(9,2))
+ USING PARQUET PARTITIONED BY (age)
+
+ INSERT INTO employees
+ VALUES ('Amy Smith', 35, 20.00)
+
+ INSERT INTO employees PARTITION (age = 35)
+ SELECT * FROM candidates WHERE name = "Bob Doe"
+{% endhighlight %}
Review comment:
@huaxingao If we determine that we allow more syntax flavors .. could we
please add one test for each ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging
cloud-fan commented on issue #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging URL: https://github.com/apache/spark/pull/25564#issuecomment-524184239 thanks for fixing it! LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum closed pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation
wangyum closed pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation URL: https://github.com/apache/spark/pull/24715 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging
AmplabJenkins removed a comment on issue #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging URL: https://github.com/apache/spark/pull/25564#issuecomment-524182465 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14670/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging
AmplabJenkins removed a comment on issue #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging URL: https://github.com/apache/spark/pull/25564#issuecomment-524182462 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging
SparkQA commented on issue #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging URL: https://github.com/apache/spark/pull/25564#issuecomment-524182921 **[Test build #109616 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109616/testReport)** for PR 25564 at commit [`0b0e02e`](https://github.com/apache/spark/commit/0b0e02e2c9ccd011a797eefe56bab254cb8babfa). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging
dongjoon-hyun commented on a change in pull request #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging URL: https://github.com/apache/spark/pull/25564#discussion_r316983236 ## File path: dev/merge_spark_pr.py ## @@ -495,7 +495,24 @@ def main(): else: title = pr["title"] -body = pr["body"] +modified_body = re.sub(re.compile(r'\n?', re.DOTALL), '', pr["body"]).lstrip() Review comment: `lstrip()` is used to remove the first PR template comments and new lines. `strip()` has false positive because it will modify all PR description (which doesn't have comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging
dongjoon-hyun commented on issue #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging URL: https://github.com/apache/spark/pull/25564#issuecomment-524182631 Could you review this PR, @gatorsmile , @cloud-fan , @HyukjinKwon ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging
AmplabJenkins commented on issue #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging URL: https://github.com/apache/spark/pull/25564#issuecomment-524182462 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging
AmplabJenkins commented on issue #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging URL: https://github.com/apache/spark/pull/25564#issuecomment-524182465 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14670/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #25453: [SPARK-28730][SQL] Configurable type coercion policy for table insertion
cloud-fan commented on issue #25453: [SPARK-28730][SQL] Configurable type coercion policy for table insertion URL: https://github.com/apache/spark/pull/25453#issuecomment-524181937 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #25453: [SPARK-28730][SQL] Configurable type coercion policy for table insertion
cloud-fan closed pull request #25453: [SPARK-28730][SQL] Configurable type coercion policy for table insertion URL: https://github.com/apache/spark/pull/25453 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25464: [SPARK-28746][SQL] Add partitionby hint for sql queries
SparkQA commented on issue #25464: [SPARK-28746][SQL] Add partitionby hint for sql queries URL: https://github.com/apache/spark/pull/25464#issuecomment-524181306 **[Test build #109615 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109615/testReport)** for PR 25464 at commit [`b0ae689`](https://github.com/apache/spark/commit/b0ae689fe86f36b6620d5709adeb429c514f40d4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging
dongjoon-hyun opened a new pull request #25564: [SPARK-28857][INFRA] Clean up the comments of PR template during merging URL: https://github.com/apache/spark/pull/25564 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce any user-facing change? No. (only for committers) ### How was this patch tested? Manually with Python2/Python3. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25464: [SPARK-28746][SQL] Add partitionby hint for sql queries
AmplabJenkins removed a comment on issue #25464: [SPARK-28746][SQL] Add partitionby hint for sql queries URL: https://github.com/apache/spark/pull/25464#issuecomment-524180963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14669/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25464: [SPARK-28746][SQL] Add partitionby hint for sql queries
AmplabJenkins commented on issue #25464: [SPARK-28746][SQL] Add partitionby hint for sql queries URL: https://github.com/apache/spark/pull/25464#issuecomment-524180960 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25464: [SPARK-28746][SQL] Add partitionby hint for sql queries
AmplabJenkins commented on issue #25464: [SPARK-28746][SQL] Add partitionby hint for sql queries URL: https://github.com/apache/spark/pull/25464#issuecomment-524180963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14669/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25464: [SPARK-28746][SQL] Add partitionby hint for sql queries
AmplabJenkins removed a comment on issue #25464: [SPARK-28746][SQL] Add partitionby hint for sql queries URL: https://github.com/apache/spark/pull/25464#issuecomment-524180960 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25556: [SPARK-28853][SQL] Support conf to organize file partitions by file path
AmplabJenkins commented on issue #25556: [SPARK-28853][SQL] Support conf to organize file partitions by file path URL: https://github.com/apache/spark/pull/25556#issuecomment-524180405 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109605/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25563: Revert "[SPARK-25474][SQL] Support `spark.sql.statistics.fallBackToHdfs` in data source tables"
dongjoon-hyun commented on issue #25563: Revert "[SPARK-25474][SQL] Support `spark.sql.statistics.fallBackToHdfs` in data source tables" URL: https://github.com/apache/spark/pull/25563#issuecomment-524180456 cc @shahidki31 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25556: [SPARK-28853][SQL] Support conf to organize file partitions by file path
AmplabJenkins removed a comment on issue #25556: [SPARK-28853][SQL] Support conf to organize file partitions by file path URL: https://github.com/apache/spark/pull/25556#issuecomment-524180405 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109605/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25556: [SPARK-28853][SQL] Support conf to organize file partitions by file path
AmplabJenkins commented on issue #25556: [SPARK-28853][SQL] Support conf to organize file partitions by file path URL: https://github.com/apache/spark/pull/25556#issuecomment-524180401 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25556: [SPARK-28853][SQL] Support conf to organize file partitions by file path
AmplabJenkins removed a comment on issue #25556: [SPARK-28853][SQL] Support conf to organize file partitions by file path URL: https://github.com/apache/spark/pull/25556#issuecomment-524180401 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25535: [SPARK-28835][SQL][TEST] Add TPCDSSchema trait
AmplabJenkins removed a comment on issue #25535: [SPARK-28835][SQL][TEST] Add TPCDSSchema trait URL: https://github.com/apache/spark/pull/25535#issuecomment-524180126 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25535: [SPARK-28835][SQL][TEST] Add TPCDSSchema trait
AmplabJenkins commented on issue #25535: [SPARK-28835][SQL][TEST] Add TPCDSSchema trait URL: https://github.com/apache/spark/pull/25535#issuecomment-524180128 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109606/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25535: [SPARK-28835][SQL][TEST] Add TPCDSSchema trait
AmplabJenkins removed a comment on issue #25535: [SPARK-28835][SQL][TEST] Add TPCDSSchema trait URL: https://github.com/apache/spark/pull/25535#issuecomment-524180128 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109606/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25535: [SPARK-28835][SQL][TEST] Add TPCDSSchema trait
AmplabJenkins commented on issue #25535: [SPARK-28835][SQL][TEST] Add TPCDSSchema trait URL: https://github.com/apache/spark/pull/25535#issuecomment-524180126 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25556: [SPARK-28853][SQL] Support conf to organize file partitions by file path
SparkQA commented on issue #25556: [SPARK-28853][SQL] Support conf to organize file partitions by file path URL: https://github.com/apache/spark/pull/25556#issuecomment-524180071 **[Test build #109605 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109605/testReport)** for PR 25556 at commit [`9aca171`](https://github.com/apache/spark/commit/9aca1719235985bb9068745941734c3158de68c2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25556: [SPARK-28853][SQL] Support conf to organize file partitions by file path
SparkQA removed a comment on issue #25556: [SPARK-28853][SQL] Support conf to organize file partitions by file path URL: https://github.com/apache/spark/pull/25556#issuecomment-52416 **[Test build #109605 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109605/testReport)** for PR 25556 at commit [`9aca171`](https://github.com/apache/spark/commit/9aca1719235985bb9068745941734c3158de68c2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25535: [SPARK-28835][SQL][TEST] Add TPCDSSchema trait
SparkQA removed a comment on issue #25535: [SPARK-28835][SQL][TEST] Add TPCDSSchema trait URL: https://github.com/apache/spark/pull/25535#issuecomment-524144471 **[Test build #109606 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109606/testReport)** for PR 25535 at commit [`1ca8607`](https://github.com/apache/spark/commit/1ca86079cfd31a9344ee3059360e04af05d4f4cf). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25535: [SPARK-28835][SQL][TEST] Add TPCDSSchema trait
SparkQA commented on issue #25535: [SPARK-28835][SQL][TEST] Add TPCDSSchema trait URL: https://github.com/apache/spark/pull/25535#issuecomment-524179800 **[Test build #109606 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109606/testReport)** for PR 25535 at commit [`1ca8607`](https://github.com/apache/spark/commit/1ca86079cfd31a9344ee3059360e04af05d4f4cf). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on a change in pull request #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
dilipbiswal commented on a change in pull request #25525:
[SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
URL: https://github.com/apache/spark/pull/25525#discussion_r316979104
##
File path: docs/sql-ref-syntax-dml-insert-into.md
##
@@ -0,0 +1,65 @@
+---
+layout: global
+title: INSERT INTO
+displayTitle: INSERT INTO
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+### Description
+
+The `INSERT INTO` statement inserts new rows into a table. The inserted rows
can be specified by value expressions, or resulted from a query.
+
+### Syntax
+{% highlight sql %}
+INSERT INTO [TABLE] [db_name.]table_name [partition_spec] value_clause | query
Review comment:
@huaxingao
Should it be { value_clause | query }
or
( value_clause | query )
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on a change in pull request #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
dilipbiswal commented on a change in pull request #25525:
[SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
URL: https://github.com/apache/spark/pull/25525#discussion_r316979104
##
File path: docs/sql-ref-syntax-dml-insert-into.md
##
@@ -0,0 +1,65 @@
+---
+layout: global
+title: INSERT INTO
+displayTitle: INSERT INTO
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+### Description
+
+The `INSERT INTO` statement inserts new rows into a table. The inserted rows
can be specified by value expressions, or resulted from a query.
+
+### Syntax
+{% highlight sql %}
+INSERT INTO [TABLE] [db_name.]table_name [partition_spec] value_clause | query
Review comment:
@huaxingao
Should it be { value_clause | query }
or
( value_clause | query ) ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25520: [SPARK-28621][SQL] Make spark.sql.crossJoin.enabled default value true
AmplabJenkins removed a comment on issue #25520: [SPARK-28621][SQL] Make spark.sql.crossJoin.enabled default value true URL: https://github.com/apache/spark/pull/25520#issuecomment-524177587 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109607/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25520: [SPARK-28621][SQL] Make spark.sql.crossJoin.enabled default value true
AmplabJenkins removed a comment on issue #25520: [SPARK-28621][SQL] Make spark.sql.crossJoin.enabled default value true URL: https://github.com/apache/spark/pull/25520#issuecomment-524177576 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25520: [SPARK-28621][SQL] Make spark.sql.crossJoin.enabled default value true
AmplabJenkins commented on issue #25520: [SPARK-28621][SQL] Make spark.sql.crossJoin.enabled default value true URL: https://github.com/apache/spark/pull/25520#issuecomment-524177587 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109607/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25520: [SPARK-28621][SQL] Make spark.sql.crossJoin.enabled default value true
AmplabJenkins commented on issue #25520: [SPARK-28621][SQL] Make spark.sql.crossJoin.enabled default value true URL: https://github.com/apache/spark/pull/25520#issuecomment-524177576 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25520: [SPARK-28621][SQL] Make spark.sql.crossJoin.enabled default value true
SparkQA removed a comment on issue #25520: [SPARK-28621][SQL] Make spark.sql.crossJoin.enabled default value true URL: https://github.com/apache/spark/pull/25520#issuecomment-524147391 **[Test build #109607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109607/testReport)** for PR 25520 at commit [`47c7186`](https://github.com/apache/spark/commit/47c7186eb58b93ec1c82a41db59c8e58ed028ad8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25520: [SPARK-28621][SQL] Make spark.sql.crossJoin.enabled default value true
SparkQA commented on issue #25520: [SPARK-28621][SQL] Make spark.sql.crossJoin.enabled default value true URL: https://github.com/apache/spark/pull/25520#issuecomment-524177403 **[Test build #109607 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109607/testReport)** for PR 25520 at commit [`47c7186`](https://github.com/apache/spark/commit/47c7186eb58b93ec1c82a41db59c8e58ed028ad8). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] JoshRosen commented on issue #25503: [SPARK-28702][SQL] Display useful error message (instead of NPE) for invalid Dataset operations
JoshRosen commented on issue #25503: [SPARK-28702][SQL] Display useful error message (instead of NPE) for invalid Dataset operations URL: https://github.com/apache/spark/pull/25503#issuecomment-524175550 Merged to master. Thanks @shivusondur! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] JoshRosen closed pull request #25503: [SPARK-28702][SQL] Display useful error message (instead of NPE) for invalid Dataset operations
JoshRosen closed pull request #25503: [SPARK-28702][SQL] Display useful error message (instead of NPE) for invalid Dataset operations URL: https://github.com/apache/spark/pull/25503 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25488: [SPARK-28025][SS] Fix FileContextBasedCheckpointFileManager leaking crc files
AmplabJenkins commented on issue #25488: [SPARK-28025][SS] Fix FileContextBasedCheckpointFileManager leaking crc files URL: https://github.com/apache/spark/pull/25488#issuecomment-524174912 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25488: [SPARK-28025][SS] Fix FileContextBasedCheckpointFileManager leaking crc files
AmplabJenkins commented on issue #25488: [SPARK-28025][SS] Fix FileContextBasedCheckpointFileManager leaking crc files URL: https://github.com/apache/spark/pull/25488#issuecomment-524174917 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109604/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25488: [SPARK-28025][SS] Fix FileContextBasedCheckpointFileManager leaking crc files
AmplabJenkins removed a comment on issue #25488: [SPARK-28025][SS] Fix FileContextBasedCheckpointFileManager leaking crc files URL: https://github.com/apache/spark/pull/25488#issuecomment-524174917 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109604/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25488: [SPARK-28025][SS] Fix FileContextBasedCheckpointFileManager leaking crc files
AmplabJenkins removed a comment on issue #25488: [SPARK-28025][SS] Fix FileContextBasedCheckpointFileManager leaking crc files URL: https://github.com/apache/spark/pull/25488#issuecomment-524174912 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25488: [SPARK-28025][SS] Fix FileContextBasedCheckpointFileManager leaking crc files
SparkQA removed a comment on issue #25488: [SPARK-28025][SS] Fix FileContextBasedCheckpointFileManager leaking crc files URL: https://github.com/apache/spark/pull/25488#issuecomment-524137571 **[Test build #109604 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109604/testReport)** for PR 25488 at commit [`376b7eb`](https://github.com/apache/spark/commit/376b7eb102532c87e27e2008866f868fa87951b8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25488: [SPARK-28025][SS] Fix FileContextBasedCheckpointFileManager leaking crc files
SparkQA commented on issue #25488: [SPARK-28025][SS] Fix FileContextBasedCheckpointFileManager leaking crc files URL: https://github.com/apache/spark/pull/25488#issuecomment-524174582 **[Test build #109604 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109604/testReport)** for PR 25488 at commit [`376b7eb`](https://github.com/apache/spark/commit/376b7eb102532c87e27e2008866f868fa87951b8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
