[GitHub] [spark] SparkQA commented on pull request #30912: [SPARK-32968][SQL] Prune unnecessary columns from CsvToStructs
SparkQA commented on pull request #30912: URL: https://github.com/apache/spark/pull/30912#issuecomment-751622345 **[Test build #133433 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133433/testReport)** for PR 30912 at commit [`55fb78f`](https://github.com/apache/spark/commit/55fb78fbcc59be2d8569093f70ddb2f329127b03). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30923: [SPARK-33909][SQL] Check rand functions seed is legal at analyer side
cloud-fan commented on a change in pull request #30923:
URL: https://github.com/apache/spark/pull/30923#discussion_r549251683
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##
@@ -215,6 +215,10 @@ trait CheckAnalysis extends PredicateHelper with
LookupCatalog {
case s: SubqueryExpression =>
checkSubqueryExpression(operator, s)
s
+
+ case e: ExpressionWithRandomSeed if !e.seedExpression.foldable =>
+failAnalysis(
+ s"Input argument to ${e.prettyName} must be an integer, long or
null literal.")
Review comment:
Then the error message here should just say "Input argument to
${e.prettyName} must be a constant"?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30841: [SPARK-28191][SS] New data source - state - reader part
SparkQA commented on pull request #30841: URL: https://github.com/apache/spark/pull/30841#issuecomment-751622626 **[Test build #133434 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133434/testReport)** for PR 30841 at commit [`a495f6d`](https://github.com/apache/spark/commit/a495f6d56411f2f3bb1e271babe9efad008b3959). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30942: [SPARK-32685][SQL] When specify serde, default filed.delim is '\t'
cloud-fan commented on a change in pull request #30942:
URL: https://github.com/apache/spark/pull/30942#discussion_r549251997
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala
##
@@ -505,7 +505,12 @@ class SparkSqlAstBuilder extends AstBuilder {
} else {
None
}
- (Seq.empty, Option(name), props.toSeq, recordHandler)
+ val finalProps = if (!props.contains("field.delim")) {
Review comment:
is this case sensitive?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30941: [SPARK-33923][SQL][TESTS] Fix some tests with AQE enabled
dongjoon-hyun commented on pull request #30941: URL: https://github.com/apache/spark/pull/30941#issuecomment-751623252 This PR is changing test cases only and Jenkins passed Scala/Java/Python tests already. Merged to master/3.1. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #30941: [SPARK-33923][SQL][TESTS] Fix some tests with AQE enabled
dongjoon-hyun closed pull request #30941: URL: https://github.com/apache/spark/pull/30941 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
AmplabJenkins commented on pull request #28422: URL: https://github.com/apache/spark/pull/28422#issuecomment-751623610 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133435/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] StefanXiepj commented on a change in pull request #30785: SPARK-33755: Allow creating orc table when row format separator is defined
StefanXiepj commented on a change in pull request #30785:
URL: https://github.com/apache/spark/pull/30785#discussion_r549252522
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala
##
@@ -153,4 +153,18 @@ class HiveOrcSourceSuite extends OrcSuite with
TestHiveSingleton {
}
}
}
+
+ test("SPARK-33755: Allow creating orc table when row format separator is
defined") {
+withTable("row_format_orc") {
+ sql(
+s"""CREATE TABLE row_format_orc(
+ | intField INT,
+ | stringField STRING
+ |)
+ |ROW FORMAT DELIMITED FIELDS TERMINATED BY '002'
Review comment:
thx very much, i'll close it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] StefanXiepj closed pull request #30785: SPARK-33755: Allow creating orc table when row format separator is defined
StefanXiepj closed pull request #30785: URL: https://github.com/apache/spark/pull/30785 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
SparkQA commented on pull request #28422: URL: https://github.com/apache/spark/pull/28422#issuecomment-751623594 **[Test build #133435 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133435/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30905: [SPARK-33890][SQL] Improve the implement of trim/trimleft/trimright
cloud-fan commented on a change in pull request #30905:
URL: https://github.com/apache/spark/pull/30905#discussion_r549253006
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##
@@ -756,6 +756,52 @@ trait String2TrimExpression extends Expression with
ImplicitCastInputTypes {
override def nullable: Boolean = children.exists(_.nullable)
override def foldable: Boolean = children.forall(_.foldable)
+ protected def doEval(input: InternalRow, srcString: UTF8String): Any
+
+ override def eval(input: InternalRow): Any = {
+val srcString = srcStr.eval(input).asInstanceOf[UTF8String]
+if (srcString == null) {
+ null
+} else {
+ doEval(input, srcString)
+}
+ }
+
+ protected val trimMethod: String
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+val evals = children.map(_.genCode(ctx))
+val srcString = evals(0)
+
+if (evals.length == 1) {
+ ev.copy(evals.map(_.code) :+ code"""
Review comment:
nit: let's use the
```
"""
|xxx
|xxx""".stripMargin
```
syntax
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##
@@ -756,6 +756,52 @@ trait String2TrimExpression extends Expression with
ImplicitCastInputTypes {
override def nullable: Boolean = children.exists(_.nullable)
override def foldable: Boolean = children.forall(_.foldable)
+ protected def doEval(input: InternalRow, srcString: UTF8String): Any
+
+ override def eval(input: InternalRow): Any = {
+val srcString = srcStr.eval(input).asInstanceOf[UTF8String]
+if (srcString == null) {
+ null
+} else {
+ doEval(input, srcString)
+}
+ }
+
+ protected val trimMethod: String
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+val evals = children.map(_.genCode(ctx))
+val srcString = evals(0)
+
+if (evals.length == 1) {
+ ev.copy(evals.map(_.code) :+ code"""
+boolean ${ev.isNull} = false;
+UTF8String ${ev.value} = null;
+if (${srcString.isNull}) {
+ ${ev.isNull} = true;
+} else {
+ ${ev.value} = ${srcString.value}.$trimMethod();
+}""")
+} else {
+ val trimString = evals(1)
+ val getTrimFunction =
+s"""
+if (${trimString.isNull}) {
Review comment:
ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] turboFei commented on a change in pull request #29982: [SPARK-33100][SQL] Ignore a semicolon inside a bracketed comment in spark-sql
turboFei commented on a change in pull request #29982:
URL: https://github.com/apache/spark/pull/29982#discussion_r549253131
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
##
@@ -573,4 +573,16 @@ class CliSuite extends SparkFunSuite with
BeforeAndAfterAll with Logging {
// the date formatter for `java.sql.LocalDate` must output negative years
with sign.
runCliWithin(1.minute)("SELECT MAKE_DATE(-44, 3, 15);" -> "-0044-03-15")
}
+
+ test("SPARK-33100: Ignore a semicolon inside a bracketed comment in
spark-sql") {
+runCliWithin(1.minute)("/* SELECT 'test';*/ SELECT 'test';" -> "test" )
+runCliWithin(1.minute)(";;/* SELECT 'test';*/ SELECT 'test';" -> "test" )
+runCliWithin(1.minute)("/* SELECT 'test';*/;; SELECT 'test';" -> "test" )
+runCliWithin(1.minute)("SELECT 'test'; -- SELECT 'test';" -> "")
+runCliWithin(1.minute)("SELECT 'test'; /* SELECT 'test';*/" -> "")
+runCliWithin(1.minute)("/*$meta chars{^\\;}*/ SELECT 'test';" -> "test")
+runCliWithin(1.minute)("/*\nmulti-line\n*/ SELECT 'test';" -> "test")
+runCliWithin(1.minute)("/*/* multi-level bracketed*/ SELECT 'test';" ->
"test")
+runCliWithin(1.minute)("SELECT /*+ COALESCE(3) */ * FROM (SELECT
'test')t;" -> "test")
Review comment:
thanks, has added
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30905: [SPARK-33890][SQL] Improve the implement of trim/trimleft/trimright
cloud-fan commented on a change in pull request #30905:
URL: https://github.com/apache/spark/pull/30905#discussion_r549253255
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##
@@ -756,6 +756,52 @@ trait String2TrimExpression extends Expression with
ImplicitCastInputTypes {
override def nullable: Boolean = children.exists(_.nullable)
override def foldable: Boolean = children.forall(_.foldable)
+ protected def doEval(input: InternalRow, srcString: UTF8String): Any
+
+ override def eval(input: InternalRow): Any = {
+val srcString = srcStr.eval(input).asInstanceOf[UTF8String]
+if (srcString == null) {
+ null
+} else {
+ doEval(input, srcString)
+}
+ }
+
+ protected val trimMethod: String
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+val evals = children.map(_.genCode(ctx))
+val srcString = evals(0)
+
+if (evals.length == 1) {
+ ev.copy(evals.map(_.code) :+ code"""
+boolean ${ev.isNull} = false;
+UTF8String ${ev.value} = null;
+if (${srcString.isNull}) {
+ ${ev.isNull} = true;
+} else {
+ ${ev.value} = ${srcString.value}.$trimMethod();
+}""")
+} else {
+ val trimString = evals(1)
+ val getTrimFunction =
Review comment:
can we inline it and use `if... else if... else`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] turboFei commented on pull request #29982: [SPARK-33100][SQL] Ignore a semicolon inside a bracketed comment in spark-sql
turboFei commented on pull request #29982: URL: https://github.com/apache/spark/pull/29982#issuecomment-751624262 test passed, gentle ping @maropu This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30942: [SPARK-32685][SQL] When specify serde, default filed.delim is '\t'
AngersZh commented on a change in pull request #30942:
URL: https://github.com/apache/spark/pull/30942#discussion_r549253481
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala
##
@@ -505,7 +505,12 @@ class SparkSqlAstBuilder extends AstBuilder {
} else {
None
}
- (Seq.empty, Option(name), props.toSeq, recordHandler)
+ val finalProps = if (!props.contains("field.delim")) {
Review comment:
> is this case sensitive?
Yes, case sensitive
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30905: [SPARK-33890][SQL] Improve the implement of trim/trimleft/trimright
cloud-fan commented on a change in pull request #30905:
URL: https://github.com/apache/spark/pull/30905#discussion_r549253565
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##
@@ -756,6 +756,52 @@ trait String2TrimExpression extends Expression with
ImplicitCastInputTypes {
override def nullable: Boolean = children.exists(_.nullable)
override def foldable: Boolean = children.forall(_.foldable)
+ protected def doEval(input: InternalRow, srcString: UTF8String): Any
Review comment:
how about `doEval(srcString: UTF8String, trimString:
Option[UTF8String])`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30905: [SPARK-33890][SQL] Improve the implement of trim/trimleft/trimright
cloud-fan commented on a change in pull request #30905:
URL: https://github.com/apache/spark/pull/30905#discussion_r549253565
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##
@@ -756,6 +756,52 @@ trait String2TrimExpression extends Expression with
ImplicitCastInputTypes {
override def nullable: Boolean = children.exists(_.nullable)
override def foldable: Boolean = children.forall(_.foldable)
+ protected def doEval(input: InternalRow, srcString: UTF8String): Any
Review comment:
how about 2 methods:
```
doEval(srcString: UTF8String)
doEval(srcString: UTF8String, trimString: UTF8String)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
AmplabJenkins commented on pull request #28422: URL: https://github.com/apache/spark/pull/28422#issuecomment-751625093 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38026/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29982: [SPARK-33100][SQL] Ignore a semicolon inside a bracketed comment in spark-sql
AmplabJenkins removed a comment on pull request #29982: URL: https://github.com/apache/spark/pull/29982#issuecomment-751621035 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38018/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30827: [SPARK-33827][SS] Unload inactive state store as soon as possible
AmplabJenkins removed a comment on pull request #30827: URL: https://github.com/apache/spark/pull/30827#issuecomment-751621032 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133424/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
SparkQA removed a comment on pull request #28422: URL: https://github.com/apache/spark/pull/28422#issuecomment-751622016 **[Test build #133435 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133435/testReport)** for PR 28422 at commit [`b7d94f7`](https://github.com/apache/spark/commit/b7d94f77860ca9343505f86e4688c30890fbb6e8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28422: [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input files
AmplabJenkins removed a comment on pull request #28422: URL: https://github.com/apache/spark/pull/28422#issuecomment-751623610 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30928: [SPARK-33912][SQL] Refactor DependencyUtils ivy property parameter
AmplabJenkins removed a comment on pull request #30928: URL: https://github.com/apache/spark/pull/30928#issuecomment-751621031 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38023/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30944: [SPARK-33907][SQL][3.1][FOLLOWUP] Add test for corrupt record column
AmplabJenkins removed a comment on pull request #30944: URL: https://github.com/apache/spark/pull/30944#issuecomment-751604685 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133429/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
AmplabJenkins removed a comment on pull request #30943: URL: https://github.com/apache/spark/pull/30943#issuecomment-751621034 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38020/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30942: [SPARK-32685][SQL] When specify serde, default filed.delim is '\t'
SparkQA commented on pull request #30942: URL: https://github.com/apache/spark/pull/30942#issuecomment-751625398 **[Test build #133421 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133421/testReport)** for PR 30942 at commit [`79e78f3`](https://github.com/apache/spark/commit/79e78f34b31a3ccdfab40b0a0e95ff9d8fd70f14). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30942: [SPARK-32685][SQL] When specify serde, default filed.delim is '\t'
SparkQA removed a comment on pull request #30942: URL: https://github.com/apache/spark/pull/30942#issuecomment-751565754 **[Test build #133421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133421/testReport)** for PR 30942 at commit [`79e78f3`](https://github.com/apache/spark/commit/79e78f34b31a3ccdfab40b0a0e95ff9d8fd70f14). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on pull request #30827: [SPARK-33827][SS] Unload inactive state store as soon as possible
viirya commented on pull request #30827: URL: https://github.com/apache/spark/pull/30827#issuecomment-751625629 Thanks guys! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on pull request #30944: [SPARK-33907][SQL][3.1][FOLLOWUP] Add test for corrupt record column
viirya commented on pull request #30944: URL: https://github.com/apache/spark/pull/30944#issuecomment-751625919 Thanks @dongjoon-hyun @HyukjinKwon This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30942: [SPARK-32685][SQL] When specify serde, default filed.delim is '\t'
AmplabJenkins commented on pull request #30942: URL: https://github.com/apache/spark/pull/30942#issuecomment-751625971 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133421/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30944: [SPARK-33907][SQL][3.1][FOLLOWUP] Add test for corrupt record column
AmplabJenkins commented on pull request #30944: URL: https://github.com/apache/spark/pull/30944#issuecomment-751626188 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38022/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30944: [SPARK-33907][SQL][3.1][FOLLOWUP] Add test for corrupt record column
SparkQA commented on pull request #30944: URL: https://github.com/apache/spark/pull/30944#issuecomment-751626176 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38022/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30942: [SPARK-32685][SQL] When specify serde, default filed.delim is '\t'
AmplabJenkins removed a comment on pull request #30942: URL: https://github.com/apache/spark/pull/30942#issuecomment-751625971 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133421/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30944: [SPARK-33907][SQL][3.1][FOLLOWUP] Add test for corrupt record column
AmplabJenkins removed a comment on pull request #30944: URL: https://github.com/apache/spark/pull/30944#issuecomment-751626188 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38022/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #30923: [SPARK-33909][SQL] Check rand functions seed is legal at analyer side
ulysses-you commented on a change in pull request #30923:
URL: https://github.com/apache/spark/pull/30923#discussion_r549256759
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##
@@ -215,6 +215,10 @@ trait CheckAnalysis extends PredicateHelper with
LookupCatalog {
case s: SubqueryExpression =>
checkSubqueryExpression(operator, s)
s
+
+ case e: ExpressionWithRandomSeed if !e.seedExpression.foldable =>
+failAnalysis(
+ s"Input argument to ${e.prettyName} must be an integer, long or
null literal.")
Review comment:
yeah, updated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #30942: [SPARK-32685][SQL] When specify serde, default filed.delim is '\t'
AngersZh commented on pull request #30942: URL: https://github.com/apache/spark/pull/30942#issuecomment-751628889 The code has been modified to make it easier for it to add other possible default parameters in specified Hive serde mode. FYI @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #30941: [SPARK-33923][SQL][TESTS] Fix some tests with AQE enabled
Ngone51 commented on pull request #30941: URL: https://github.com/apache/spark/pull/30941#issuecomment-751628871 thanks all! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30941: [SPARK-33923][SQL][TESTS] Fix some tests with AQE enabled
SparkQA commented on pull request #30941: URL: https://github.com/apache/spark/pull/30941#issuecomment-751628925 **[Test build #133422 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133422/testReport)** for PR 30941 at commit [`d6be7c4`](https://github.com/apache/spark/commit/d6be7c4b577f5b47b9ad8f965eb46f5e48f4b36b). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class BucketedReadSuite extends QueryTest with SQLTestUtils with AdaptiveSparkPlanHelper ` * `class BucketedReadWithHiveSupportSuite extends BucketedReadSuite with TestHiveSingleton ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30941: [SPARK-33923][SQL][TESTS] Fix some tests with AQE enabled
SparkQA removed a comment on pull request #30941: URL: https://github.com/apache/spark/pull/30941#issuecomment-751566077 **[Test build #133422 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133422/testReport)** for PR 30941 at commit [`d6be7c4`](https://github.com/apache/spark/commit/d6be7c4b577f5b47b9ad8f965eb46f5e48f4b36b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #30942: [SPARK-32685][SQL] When specify serde, default filed.delim is '\t'
cloud-fan commented on pull request #30942: URL: https://github.com/apache/spark/pull/30942#issuecomment-751629073 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #30942: [SPARK-32685][SQL] When specify serde, default filed.delim is '\t'
cloud-fan closed pull request #30942: URL: https://github.com/apache/spark/pull/30942 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30912: [SPARK-32968][SQL] Prune unnecessary columns from CsvToStructs
SparkQA commented on pull request #30912: URL: https://github.com/apache/spark/pull/30912#issuecomment-751629279 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38021/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #30905: [SPARK-33890][SQL] Improve the implement of trim/trimleft/trimright
beliefer commented on a change in pull request #30905:
URL: https://github.com/apache/spark/pull/30905#discussion_r549260063
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##
@@ -756,6 +756,52 @@ trait String2TrimExpression extends Expression with
ImplicitCastInputTypes {
override def nullable: Boolean = children.exists(_.nullable)
override def foldable: Boolean = children.forall(_.foldable)
+ protected def doEval(input: InternalRow, srcString: UTF8String): Any
+
+ override def eval(input: InternalRow): Any = {
+val srcString = srcStr.eval(input).asInstanceOf[UTF8String]
+if (srcString == null) {
+ null
+} else {
+ doEval(input, srcString)
+}
+ }
+
+ protected val trimMethod: String
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+val evals = children.map(_.genCode(ctx))
+val srcString = evals(0)
+
+if (evals.length == 1) {
+ ev.copy(evals.map(_.code) :+ code"""
Review comment:
OK
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #30905: [SPARK-33890][SQL] Improve the implement of trim/trimleft/trimright
beliefer commented on a change in pull request #30905:
URL: https://github.com/apache/spark/pull/30905#discussion_r549261519
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
##
@@ -756,6 +756,52 @@ trait String2TrimExpression extends Expression with
ImplicitCastInputTypes {
override def nullable: Boolean = children.exists(_.nullable)
override def foldable: Boolean = children.forall(_.foldable)
+ protected def doEval(input: InternalRow, srcString: UTF8String): Any
+
+ override def eval(input: InternalRow): Any = {
+val srcString = srcStr.eval(input).asInstanceOf[UTF8String]
+if (srcString == null) {
+ null
+} else {
+ doEval(input, srcString)
+}
+ }
+
+ protected val trimMethod: String
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+val evals = children.map(_.genCode(ctx))
+val srcString = evals(0)
+
+if (evals.length == 1) {
+ ev.copy(evals.map(_.code) :+ code"""
+boolean ${ev.isNull} = false;
+UTF8String ${ev.value} = null;
+if (${srcString.isNull}) {
+ ${ev.isNull} = true;
+} else {
+ ${ev.value} = ${srcString.value}.$trimMethod();
+}""")
+} else {
+ val trimString = evals(1)
+ val getTrimFunction =
Review comment:
Good idea.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ScrapCodes commented on pull request #30472: [SPARK-32221][k8s] Avoid possible errors due to incorrect file size or type supplied in spark conf.
ScrapCodes commented on pull request #30472: URL: https://github.com/apache/spark/pull/30472#issuecomment-751632890 Hi @dongjoon-hyun, I am sorry I was not well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon opened a new pull request #30945: [SPARK-33925][CORE] Remove unused SecurityManager in Utils.fetchFile
HyukjinKwon opened a new pull request #30945: URL: https://github.com/apache/spark/pull/30945 ### What changes were proposed in this pull request? This is kind of a followup of https://github.com/apache/spark/pull/24033. The first and last usage of that argument `SecurityManager` was removed in https://github.com/apache/spark/pull/24033. After that, we don't need to pass `SecurityManager` anymore in `Utils.fetchFile` and related code paths. This PR proposes to remove it out. ### Why are the changes needed? For better readability of codes. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Manually complied. GitHub Actions and Jenkins build should test it out as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30945: [SPARK-33925][CORE] Remove unused SecurityManager in Utils.fetchFile
HyukjinKwon commented on pull request #30945: URL: https://github.com/apache/spark/pull/30945#issuecomment-751634700 cc @vanzin, @gaborgsomogyi, @skonto FYI This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30935: [SPARK-33859][SQL] Support V2 ALTER TABLE .. RENAME PARTITION
MaxGekk commented on a change in pull request #30935:
URL: https://github.com/apache/spark/pull/30935#discussion_r549265253
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/AlterTableRenamePartitionSuiteBase.scala
##
@@ -17,8 +17,149 @@
package org.apache.spark.sql.execution.command
-import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
+import org.apache.spark.sql.catalyst.analysis.{NoSuchPartitionException,
PartitionAlreadyExistsException}
+import org.apache.spark.sql.internal.SQLConf
trait AlterTableRenamePartitionSuiteBase extends QueryTest with
DDLCommandTestUtils {
override val command = "ALTER TABLE .. RENAME PARTITION"
+
+ protected def createSinglePartTable(t: String): Unit = {
+sql(s"CREATE TABLE $t (id bigint, data string) $defaultUsing PARTITIONED
BY (id)")
+sql(s"INSERT INTO $t PARTITION (id = 1) SELECT 'abc'")
+ }
+
+ test("rename without explicitly specifying database") {
+withNamespaceAndTable("ns", "tbl") { t =>
+ createSinglePartTable(t)
+ checkPartitions(t, Map("id" -> "1"))
+
+ sql(s"ALTER TABLE $t PARTITION (id = 1) RENAME TO PARTITION (id = 2)")
Review comment:
ok. I will rewrite the test slightly to not use any database in the
table name
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30924: [SPARK-33804][CORE] Fix compilation warnings about 'view bounds are deprecated'
SparkQA commented on pull request #30924: URL: https://github.com/apache/spark/pull/30924#issuecomment-751635996 **[Test build #133426 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133426/testReport)** for PR 30924 at commit [`8e626f8`](https://github.com/apache/spark/commit/8e626f8ff8807dc85f2d4c4a6a888f8abd264dfb). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class SequenceFileRDDFunctions[K: IsWritable: ClassTag, V: IsWritable: ClassTag](` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30841: [SPARK-28191][SS] New data source - state - reader part
SparkQA commented on pull request #30841: URL: https://github.com/apache/spark/pull/30841#issuecomment-751636556 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38025/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30941: [SPARK-33923][SQL][TESTS] Fix some tests with AQE enabled
AmplabJenkins commented on pull request #30941: URL: https://github.com/apache/spark/pull/30941#issuecomment-751637943 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133422/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30912: [SPARK-32968][SQL] Prune unnecessary columns from CsvToStructs
AmplabJenkins commented on pull request #30912: URL: https://github.com/apache/spark/pull/30912#issuecomment-751637946 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38021/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30924: [SPARK-33804][CORE] Fix compilation warnings about 'view bounds are deprecated'
AmplabJenkins commented on pull request #30924: URL: https://github.com/apache/spark/pull/30924#issuecomment-751637944 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133426/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30935: [SPARK-33859][SQL] Support V2 ALTER TABLE .. RENAME PARTITION
MaxGekk commented on a change in pull request #30935:
URL: https://github.com/apache/spark/pull/30935#discussion_r549268524
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/AlterTableRenamePartitionSuiteBase.scala
##
@@ -17,8 +17,149 @@
package org.apache.spark.sql.execution.command
-import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
+import org.apache.spark.sql.catalyst.analysis.{NoSuchPartitionException,
PartitionAlreadyExistsException}
+import org.apache.spark.sql.internal.SQLConf
trait AlterTableRenamePartitionSuiteBase extends QueryTest with
DDLCommandTestUtils {
override val command = "ALTER TABLE .. RENAME PARTITION"
+
+ protected def createSinglePartTable(t: String): Unit = {
+sql(s"CREATE TABLE $t (id bigint, data string) $defaultUsing PARTITIONED
BY (id)")
+sql(s"INSERT INTO $t PARTITION (id = 1) SELECT 'abc'")
+ }
+
+ test("rename without explicitly specifying database") {
+withNamespaceAndTable("ns", "tbl") { t =>
+ createSinglePartTable(t)
+ checkPartitions(t, Map("id" -> "1"))
+
+ sql(s"ALTER TABLE $t PARTITION (id = 1) RENAME TO PARTITION (id = 2)")
+ checkPartitions(t, Map("id" -> "2"))
+ checkAnswer(sql(s"SELECT id, data FROM $t"), Row(2, "abc"))
+}
+ }
+
+ test("table to alter does not exist") {
+withNamespace(s"$catalog.ns") {
+ sql(s"CREATE NAMESPACE $catalog.ns")
+ val errMsg = intercept[AnalysisException] {
+sql(s"ALTER TABLE $catalog.ns.no_tbl PARTITION (id=1) RENAME TO
PARTITION (id=2)")
+ }.getMessage
+ assert(errMsg.contains("Table not found"))
+}
+ }
+
+ test("partition to rename does not exist") {
+withNamespaceAndTable("ns", "tbl") { t =>
+ createSinglePartTable(t)
+ checkPartitions(t, Map("id" -> "1"))
+ val errMsg = intercept[NoSuchPartitionException] {
+sql(s"ALTER TABLE $t PARTITION (id = 3) RENAME TO PARTITION (id = 2)")
+ }.getMessage
+ assert(errMsg.contains("Partition not found in table"))
+}
+ }
+
+ test("target partition exists") {
+withNamespaceAndTable("ns", "tbl") { t =>
+ createSinglePartTable(t)
+ sql(s"INSERT INTO $t PARTITION (id = 2) SELECT 'def'")
+ checkPartitions(t, Map("id" -> "1"), Map("id" -> "2"))
+ val errMsg = intercept[PartitionAlreadyExistsException] {
+sql(s"ALTER TABLE $t PARTITION (id = 1) RENAME TO PARTITION (id = 2)")
+ }.getMessage
+ assert(errMsg.contains("Partition already exists"))
+}
+ }
+
+ test("single part partition") {
+withNamespaceAndTable("ns", "tbl") { t =>
+ createSinglePartTable(t)
+ checkPartitions(t, Map("id" -> "1"))
+
+ sql(s"ALTER TABLE $t PARTITION (id = 1) RENAME TO PARTITION (id = 2)")
+ checkPartitions(t, Map("id" -> "2"))
+ checkAnswer(sql(s"SELECT id, data FROM $t"), Row(2, "abc"))
+}
+ }
+
+ test("multi part partition") {
+withNamespaceAndTable("ns", "tbl") { t =>
+ createWideTable(t)
+ checkPartitions(t,
+Map(
+ "year" -> "2016",
+ "month" -> "3",
+ "hour" -> "10",
+ "minute" -> "10",
+ "sec" -> "10",
+ "extra" -> "1"),
+Map(
+ "year" -> "2016",
+ "month" -> "4",
+ "hour" -> "10",
+ "minute" -> "10",
+ "sec" -> "10",
+ "extra" -> "1"))
+
+ sql(s"""
+|ALTER TABLE $t
+|PARTITION (
+| year = 2016, month = 3, hour = 10, minute = 10, sec = 10, extra = 1
+|) RENAME TO PARTITION (
+| year = 2016, month = 3, hour = 10, minute = 10, sec = 123, extra = 1
+|)""".stripMargin)
+ checkPartitions(t,
+Map(
+ "year" -> "2016",
+ "month" -> "3",
+ "hour" -> "10",
+ "minute" -> "10",
+ "sec" -> "123",
+ "extra" -> "1"),
+Map(
+ "year" -> "2016",
+ "month" -> "4",
+ "hour" -> "10",
+ "minute" -> "10",
+ "sec" -> "10",
+ "extra" -> "1"))
+ checkAnswer(sql(s"SELECT month, sec, price FROM $t"), Row(3, 123, 3))
+}
+ }
+
+ test("with location") {
+withNamespaceAndTable("ns", "tbl") { t =>
+ createSinglePartTable(t)
+ sql(s"ALTER TABLE $t ADD PARTITION (id = 2) LOCATION 'loc1'")
+ sql(s"INSERT INTO $t PARTITION (id = 2) SELECT 'def'")
+ checkPartitions(t, Map("id" -> "1"), Map("id" -> "2"))
+
+ sql(s"ALTER TABLE $t PARTITION (id = 2) RENAME TO PARTITION (id = 3)")
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
[GitHub] [spark] SparkQA commented on pull request #30945: [SPARK-33925][CORE] Remove unused SecurityManager in Utils.fetchFile
SparkQA commented on pull request #30945: URL: https://github.com/apache/spark/pull/30945#issuecomment-751639051 **[Test build #133436 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133436/testReport)** for PR 30945 at commit [`56e7707`](https://github.com/apache/spark/commit/56e7707c24f77ee534fdcd61b8df3e89627f0877). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30923: [SPARK-33909][SQL] Check rand functions seed is legal at analyer side
SparkQA commented on pull request #30923: URL: https://github.com/apache/spark/pull/30923#issuecomment-751639068 **[Test build #133438 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133438/testReport)** for PR 30923 at commit [`28b4d21`](https://github.com/apache/spark/commit/28b4d215cf6efc96c72277fa16b5bbd37463ad3a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30905: [SPARK-33890][SQL] Improve the implement of trim/trimleft/trimright
SparkQA commented on pull request #30905: URL: https://github.com/apache/spark/pull/30905#issuecomment-751639091 **[Test build #133439 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133439/testReport)** for PR 30905 at commit [`7d32023`](https://github.com/apache/spark/commit/7d32023231c0d2e5b17a66450461622c7c69a3e3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30935: [SPARK-33859][SQL] Support V2 ALTER TABLE .. RENAME PARTITION
SparkQA commented on pull request #30935: URL: https://github.com/apache/spark/pull/30935#issuecomment-751639130 **[Test build #133437 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133437/testReport)** for PR 30935 at commit [`e81ab1f`](https://github.com/apache/spark/commit/e81ab1fc41ba16745ab80d56888d10344e345aa0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30945: [SPARK-33925][CORE] Remove unused SecurityManager in Utils.fetchFile
AmplabJenkins commented on pull request #30945: URL: https://github.com/apache/spark/pull/30945#issuecomment-751639631 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38027/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30841: [SPARK-28191][SS] New data source - state - reader part
SparkQA commented on pull request #30841: URL: https://github.com/apache/spark/pull/30841#issuecomment-751639925 **[Test build #133434 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133434/testReport)** for PR 30841 at commit [`a495f6d`](https://github.com/apache/spark/commit/a495f6d56411f2f3bb1e271babe9efad008b3959). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30841: [SPARK-28191][SS] New data source - state - reader part
AmplabJenkins commented on pull request #30841: URL: https://github.com/apache/spark/pull/30841#issuecomment-751640090 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133434/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30945: [SPARK-33925][CORE] Remove unused SecurityManager in Utils.fetchFile
AmplabJenkins commented on pull request #30945: URL: https://github.com/apache/spark/pull/30945#issuecomment-751641410 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133436/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30945: [SPARK-33925][CORE] Remove unused SecurityManager in Utils.fetchFile
SparkQA commented on pull request #30945: URL: https://github.com/apache/spark/pull/30945#issuecomment-751641396 **[Test build #133436 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133436/testReport)** for PR 30945 at commit [`56e7707`](https://github.com/apache/spark/commit/56e7707c24f77ee534fdcd61b8df3e89627f0877). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30915: [SPARK-33899][SQL] Fix assert failure in v1 SHOW TABLES/VIEWS on `spark_catalog`
cloud-fan commented on a change in pull request #30915:
URL: https://github.com/apache/spark/pull/30915#discussion_r549271216
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowTablesSuite.scala
##
@@ -100,6 +100,17 @@ trait ShowTablesSuiteBase extends
command.ShowTablesSuiteBase {
}
}
}
+
+ test("no database specified") {
+Seq(
+ s"SHOW TABLES IN $catalog",
+ s"SHOW TABLE EXTENDED IN $catalog LIKE '*tbl'").foreach { showTableCmd =>
+ val errMsg = intercept[AnalysisException] {
+sql(showTableCmd)
+ }.getMessage
+ assert(errMsg.contains("multi-part identifier cannot be empty"))
Review comment:
Since it may affect many tests, let's update the error message in a new
PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30915: [SPARK-33899][SQL] Fix assert failure in v1 SHOW TABLES/VIEWS on `spark_catalog`
MaxGekk commented on a change in pull request #30915:
URL: https://github.com/apache/spark/pull/30915#discussion_r549271323
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowTablesSuite.scala
##
@@ -100,6 +100,17 @@ trait ShowTablesSuiteBase extends
command.ShowTablesSuiteBase {
}
}
}
+
+ test("no database specified") {
+Seq(
+ s"SHOW TABLES IN $catalog",
+ s"SHOW TABLE EXTENDED IN $catalog LIKE '*tbl'").foreach { showTableCmd =>
+ val errMsg = intercept[AnalysisException] {
+sql(showTableCmd)
+ }.getMessage
+ assert(errMsg.contains("multi-part identifier cannot be empty"))
Review comment:
I would prefer to make such changes separately if you don't mind because
such change will cause modification that are unrelated to this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #30915: [SPARK-33899][SQL] Fix assert failure in v1 SHOW TABLES/VIEWS on `spark_catalog`
cloud-fan commented on pull request #30915: URL: https://github.com/apache/spark/pull/30915#issuecomment-751641664 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #30915: [SPARK-33899][SQL] Fix assert failure in v1 SHOW TABLES/VIEWS on `spark_catalog`
cloud-fan closed pull request #30915: URL: https://github.com/apache/spark/pull/30915 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30841: [SPARK-28191][SS] New data source - state - reader part
SparkQA removed a comment on pull request #30841: URL: https://github.com/apache/spark/pull/30841#issuecomment-751622626 **[Test build #133434 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133434/testReport)** for PR 30841 at commit [`a495f6d`](https://github.com/apache/spark/commit/a495f6d56411f2f3bb1e271babe9efad008b3959). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30841: [SPARK-28191][SS] New data source - state - reader part
AmplabJenkins removed a comment on pull request #30841: URL: https://github.com/apache/spark/pull/30841#issuecomment-751640090 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133434/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30924: [SPARK-33804][CORE] Fix compilation warnings about 'view bounds are deprecated'
AmplabJenkins removed a comment on pull request #30924: URL: https://github.com/apache/spark/pull/30924#issuecomment-751637944 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133426/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30912: [SPARK-32968][SQL] Prune unnecessary columns from CsvToStructs
AmplabJenkins removed a comment on pull request #30912: URL: https://github.com/apache/spark/pull/30912#issuecomment-751637946 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38021/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30941: [SPARK-33923][SQL][TESTS] Fix some tests with AQE enabled
AmplabJenkins removed a comment on pull request #30941: URL: https://github.com/apache/spark/pull/30941#issuecomment-751637943 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133422/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30924: [SPARK-33804][CORE] Fix compilation warnings about 'view bounds are deprecated'
SparkQA removed a comment on pull request #30924: URL: https://github.com/apache/spark/pull/30924#issuecomment-75156 **[Test build #133426 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133426/testReport)** for PR 30924 at commit [`8e626f8`](https://github.com/apache/spark/commit/8e626f8ff8807dc85f2d4c4a6a888f8abd264dfb). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30945: [SPARK-33925][CORE] Remove unused SecurityManager in Utils.fetchFile
SparkQA removed a comment on pull request #30945: URL: https://github.com/apache/spark/pull/30945#issuecomment-751639051 **[Test build #133436 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133436/testReport)** for PR 30945 at commit [`56e7707`](https://github.com/apache/spark/commit/56e7707c24f77ee534fdcd61b8df3e89627f0877). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30945: [SPARK-33925][CORE] Remove unused SecurityManager in Utils.fetchFile
AmplabJenkins removed a comment on pull request #30945: URL: https://github.com/apache/spark/pull/30945#issuecomment-751639631 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu opened a new pull request #30946: [SPARK-32684][SQL] Script transform hive serde mode null value keep same with hive as '\\N'
AngersZh opened a new pull request #30946: URL: https://github.com/apache/spark/pull/30946 ### What changes were proposed in this pull request? In hive script transform serde mode, NULL format default is `\\N` ``` String nullString = tbl.getProperty( serdeConstants.SERIALIZATION_NULL_FORMAT, "\\N"); nullSequence = new Text(nullString); ``` I make a mistake that in Spark's code we need to fix and keep same with hive too. So add some test case to show this issue. ### Why are the changes needed? add UT ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Added UT This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #30946: [SPARK-32684][SQL] Script transform hive serde mode null value keep same with hive as '\\N'
AngersZh commented on pull request #30946: URL: https://github.com/apache/spark/pull/30946#issuecomment-751643600 FYI @maropu @cloud-fan @HyukjinKwon This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu edited a comment on pull request #29087: [SPARK-28227][SQL] Support TRANSFORM with aggregation
AngersZh edited a comment on pull request #29087: URL: https://github.com/apache/spark/pull/29087#issuecomment-751614832 also FYI @Alfozan @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30144: [SPARK-33229][SQL] Support GROUP BY use Separate columns and CUBE/ROLLUP
SparkQA commented on pull request #30144: URL: https://github.com/apache/spark/pull/30144#issuecomment-751644388 **[Test build #133440 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133440/testReport)** for PR 30144 at commit [`c1c551c`](https://github.com/apache/spark/commit/c1c551c6f9656ec71d6da03fb8fd4c4119d66c3a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30472: [SPARK-32221][k8s] Avoid possible errors due to incorrect file size or type supplied in spark conf.
SparkQA commented on pull request #30472: URL: https://github.com/apache/spark/pull/30472#issuecomment-751644392 **[Test build #133442 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133442/testReport)** for PR 30472 at commit [`38288f1`](https://github.com/apache/spark/commit/38288f1e49ec290fb8a9cb1435e11f3bd881f0dd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30946: [SPARK-32684][SQL] Script transform hive serde mode null value keep same with hive as '\\N'
SparkQA commented on pull request #30946: URL: https://github.com/apache/spark/pull/30946#issuecomment-751645324 **[Test build #133441 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133441/testReport)** for PR 30946 at commit [`983a3e2`](https://github.com/apache/spark/commit/983a3e2d4dbb358c480fcf86b3a726291a8ce4e1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
SparkQA commented on pull request #30943: URL: https://github.com/apache/spark/pull/30943#issuecomment-751645526 **[Test build #133430 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133430/testReport)** for PR 30943 at commit [`c82bb15`](https://github.com/apache/spark/commit/c82bb15172847ed39e3b9b59ae1642c32529e300). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
SparkQA removed a comment on pull request #30943: URL: https://github.com/apache/spark/pull/30943#issuecomment-751605734 **[Test build #133430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133430/testReport)** for PR 30943 at commit [`c82bb15`](https://github.com/apache/spark/commit/c82bb15172847ed39e3b9b59ae1642c32529e300). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30841: [SPARK-28191][SS] New data source - state - reader part
SparkQA commented on pull request #30841: URL: https://github.com/apache/spark/pull/30841#issuecomment-751645925 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38025/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #30919: [SPARK-33904][SQL] Recognize `spark_catalog` in `saveAsTable()` and `insertInto()`
MaxGekk commented on pull request #30919: URL: https://github.com/apache/spark/pull/30919#issuecomment-751646048 @cloud-fan @HyukjinKwon Could you review this PR, please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30905: [SPARK-33890][SQL] Improve the implement of trim/trimleft/trimright
SparkQA commented on pull request #30905: URL: https://github.com/apache/spark/pull/30905#issuecomment-751646126 **[Test build #133439 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133439/testReport)** for PR 30905 at commit [`7d32023`](https://github.com/apache/spark/commit/7d32023231c0d2e5b17a66450461622c7c69a3e3). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30905: [SPARK-33890][SQL] Improve the implement of trim/trimleft/trimright
SparkQA removed a comment on pull request #30905: URL: https://github.com/apache/spark/pull/30905#issuecomment-751639091 **[Test build #133439 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133439/testReport)** for PR 30905 at commit [`7d32023`](https://github.com/apache/spark/commit/7d32023231c0d2e5b17a66450461622c7c69a3e3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] saikocat commented on pull request #30902: [SPARK-33888][SQL] JDBC SQL TIME type represents incorrectly as TimestampType, it should be physical Int in millis
saikocat commented on pull request #30902: URL: https://github.com/apache/spark/pull/30902#issuecomment-751647327 @cloud-fan I have addressed the code review comments. Took awhile cos of a bug in metadata builder not being build. I will use the metadata field to store a key `logicaltimetype` boolean and check for this in pattern matching. Also fixed a bug in the metadata not being build/propagate for `StructField` in `getSchema()`. Ideally another MR but this fix is required for the extra field in metadata. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30935: [SPARK-33859][SQL] Support V2 ALTER TABLE .. RENAME PARTITION
SparkQA commented on pull request #30935: URL: https://github.com/apache/spark/pull/30935#issuecomment-751647947 **[Test build #133437 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133437/testReport)** for PR 30935 at commit [`e81ab1f`](https://github.com/apache/spark/commit/e81ab1fc41ba16745ab80d56888d10344e345aa0). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30935: [SPARK-33859][SQL] Support V2 ALTER TABLE .. RENAME PARTITION
SparkQA removed a comment on pull request #30935: URL: https://github.com/apache/spark/pull/30935#issuecomment-751639130 **[Test build #133437 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133437/testReport)** for PR 30935 at commit [`e81ab1f`](https://github.com/apache/spark/commit/e81ab1fc41ba16745ab80d56888d10344e345aa0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30472: [SPARK-32221][k8s] Avoid possible errors due to incorrect file size or type supplied in spark conf.
SparkQA commented on pull request #30472: URL: https://github.com/apache/spark/pull/30472#issuecomment-751648508 **[Test build #133442 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133442/testReport)** for PR 30472 at commit [`38288f1`](https://github.com/apache/spark/commit/38288f1e49ec290fb8a9cb1435e11f3bd881f0dd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30472: [SPARK-32221][k8s] Avoid possible errors due to incorrect file size or type supplied in spark conf.
SparkQA removed a comment on pull request #30472: URL: https://github.com/apache/spark/pull/30472#issuecomment-751644392 **[Test build #133442 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133442/testReport)** for PR 30472 at commit [`38288f1`](https://github.com/apache/spark/commit/38288f1e49ec290fb8a9cb1435e11f3bd881f0dd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30905: [SPARK-33890][SQL] Improve the implement of trim/trimleft/trimright
SparkQA commented on pull request #30905: URL: https://github.com/apache/spark/pull/30905#issuecomment-751652975 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38030/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30935: [SPARK-33859][SQL] Support V2 ALTER TABLE .. RENAME PARTITION
SparkQA commented on pull request #30935: URL: https://github.com/apache/spark/pull/30935#issuecomment-751653279 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38028/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30912: [SPARK-32968][SQL] Prune unnecessary columns from CsvToStructs
AmplabJenkins commented on pull request #30912: URL: https://github.com/apache/spark/pull/30912#issuecomment-751656052 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38024/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30905: [SPARK-33890][SQL] Improve the implement of trim/trimleft/trimright
AmplabJenkins commented on pull request #30905: URL: https://github.com/apache/spark/pull/30905#issuecomment-751656047 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133439/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30841: [SPARK-28191][SS] New data source - state - reader part
AmplabJenkins commented on pull request #30841: URL: https://github.com/apache/spark/pull/30841#issuecomment-751656046 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38025/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
AmplabJenkins commented on pull request #30943: URL: https://github.com/apache/spark/pull/30943#issuecomment-751656048 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133430/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30472: [SPARK-32221][k8s] Avoid possible errors due to incorrect file size or type supplied in spark conf.
AmplabJenkins commented on pull request #30472: URL: https://github.com/apache/spark/pull/30472#issuecomment-751656050 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133442/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30935: [SPARK-33859][SQL] Support V2 ALTER TABLE .. RENAME PARTITION
AmplabJenkins commented on pull request #30935: URL: https://github.com/apache/spark/pull/30935#issuecomment-751656049 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133437/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
