[GitHub] [spark] ulysses-you commented on a change in pull request #26365: [SPARK-29720][CORE] Add linux condition to make ProcfsMetricsGetter more complete
ulysses-you commented on a change in pull request #26365: [SPARK-29720][CORE] Add linux condition to make ProcfsMetricsGetter more complete URL: https://github.com/apache/spark/pull/26365#discussion_r341836148 ## File path: core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ## @@ -62,13 +64,14 @@ private[spark] class ProcfsMetricsGetter(procfsDir: String = "/proc/") extends L SparkEnv.get.conf.get(config.EVENT_LOG_STAGE_EXECUTOR_METRICS) val shouldLogStageExecutorProcessTreeMetrics = SparkEnv.get.conf.get(config.EVENT_LOG_PROCESS_TREE_METRICS) - procDirExists.get && shouldLogStageExecutorProcessTreeMetrics && shouldLogStageExecutorMetrics + procDirExists.get && shouldLogStageExecutorProcessTreeMetrics && +shouldLogStageExecutorMetrics && Shell.LINUX Review comment: OK. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26373: [SPARK-29729][BUILD][test-hadoop3.2][test-java11] Upgrade ASM to 7.2
AmplabJenkins commented on issue #26373: [SPARK-29729][BUILD][test-hadoop3.2][test-java11] Upgrade ASM to 7.2 URL: https://github.com/apache/spark/pull/26373#issuecomment-549107686 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26373: [SPARK-29729][BUILD][test-hadoop3.2][test-java11] Upgrade ASM to 7.2
AmplabJenkins removed a comment on issue #26373: [SPARK-29729][BUILD][test-hadoop3.2][test-java11] Upgrade ASM to 7.2 URL: https://github.com/apache/spark/pull/26373#issuecomment-549107687 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18066/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26373: [SPARK-29729][BUILD][test-hadoop3.2][test-java11] Upgrade ASM to 7.2
AmplabJenkins commented on issue #26373: [SPARK-29729][BUILD][test-hadoop3.2][test-java11] Upgrade ASM to 7.2 URL: https://github.com/apache/spark/pull/26373#issuecomment-549107687 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18066/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26373: [SPARK-29729][BUILD][test-hadoop3.2][test-java11] Upgrade ASM to 7.2
AmplabJenkins removed a comment on issue #26373: [SPARK-29729][BUILD][test-hadoop3.2][test-java11] Upgrade ASM to 7.2 URL: https://github.com/apache/spark/pull/26373#issuecomment-549107686 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26107: [SPARK-28885][SQL] Follow ANSI store assignment rules in table insertion by default
dongjoon-hyun commented on a change in pull request #26107: [SPARK-28885][SQL]
Follow ANSI store assignment rules in table insertion by default
URL: https://github.com/apache/spark/pull/26107#discussion_r341835985
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##
@@ -143,7 +143,19 @@ class SQLQueryTestSuite extends QueryTest with
SharedSparkSession {
/** List of test cases to ignore, in lower cases. */
protected def blackList: Set[String] = Set(
-"blacklist.sql" // Do NOT remove this one. It is here to test the
blacklist functionality.
+"blacklist.sql", // Do NOT remove this one. It is here to test the
blacklist functionality.
+// SPARK-28885 String value is not allowed to be stored as numeric type
with
+// ANSI store assignment policy.
+"postgreSQL/numeric.sql",
+"postgreSQL/int2.sql",
+"postgreSQL/int4.sql",
+"postgreSQL/int8.sql",
+"postgreSQL/float4.sql",
+"postgreSQL/float8.sql",
+// SPARK-28885 String value is not allowed to be stored as date/timestamp
type with
+// ANSI store assignment policy.
+"postgreSQL/date.sql",
Review comment:
I'm +1 for @gengliangwang 's suggestion `enabling those tests with that
modification`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26373: [SPARK-29729][BUILD][test-hadoop3.2][test-java11] Upgrade ASM to 7.2
SparkQA commented on issue #26373: [SPARK-29729][BUILD][test-hadoop3.2][test-java11] Upgrade ASM to 7.2 URL: https://github.com/apache/spark/pull/26373#issuecomment-549107634 **[Test build #113152 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113152/testReport)** for PR 26373 at commit [`4076e42`](https://github.com/apache/spark/commit/4076e427ed2d4d196453453275359b95101921bb). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26373: [SPARK-29729][BUILD][test-hadoop3.2][test-java11] Upgrade ASM to 7.2
dongjoon-hyun commented on issue #26373: [SPARK-29729][BUILD][test-hadoop3.2][test-java11] Upgrade ASM to 7.2 URL: https://github.com/apache/spark/pull/26373#issuecomment-549107570 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26107: [SPARK-28885][SQL] Follow ANSI store assignment rules in table insertion by default
maropu commented on a change in pull request #26107: [SPARK-28885][SQL] Follow
ANSI store assignment rules in table insertion by default
URL: https://github.com/apache/spark/pull/26107#discussion_r341835437
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##
@@ -143,7 +143,19 @@ class SQLQueryTestSuite extends QueryTest with
SharedSparkSession {
/** List of test cases to ignore, in lower cases. */
protected def blackList: Set[String] = Set(
-"blacklist.sql" // Do NOT remove this one. It is here to test the
blacklist functionality.
+"blacklist.sql", // Do NOT remove this one. It is here to test the
blacklist functionality.
+// SPARK-28885 String value is not allowed to be stored as numeric type
with
+// ANSI store assignment policy.
+"postgreSQL/numeric.sql",
+"postgreSQL/int2.sql",
+"postgreSQL/int4.sql",
+"postgreSQL/int8.sql",
+"postgreSQL/float4.sql",
+"postgreSQL/float8.sql",
+// SPARK-28885 String value is not allowed to be stored as date/timestamp
type with
+// ANSI store assignment policy.
+"postgreSQL/date.sql",
Review comment:
> I am not sure about whether there is a guideline in porting the PgSQL test
files. Such modifications are allowed, right?
Yea, I think that's ok. cc: @wangyum @dongjoon-hyun @HyukjinKwon
If you have no time to do, I'll check next week.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26107: [SPARK-28885][SQL] Follow ANSI store assignment rules in table insertion by default
maropu commented on a change in pull request #26107: [SPARK-28885][SQL] Follow
ANSI store assignment rules in table insertion by default
URL: https://github.com/apache/spark/pull/26107#discussion_r341835437
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##
@@ -143,7 +143,19 @@ class SQLQueryTestSuite extends QueryTest with
SharedSparkSession {
/** List of test cases to ignore, in lower cases. */
protected def blackList: Set[String] = Set(
-"blacklist.sql" // Do NOT remove this one. It is here to test the
blacklist functionality.
+"blacklist.sql", // Do NOT remove this one. It is here to test the
blacklist functionality.
+// SPARK-28885 String value is not allowed to be stored as numeric type
with
+// ANSI store assignment policy.
+"postgreSQL/numeric.sql",
+"postgreSQL/int2.sql",
+"postgreSQL/int4.sql",
+"postgreSQL/int8.sql",
+"postgreSQL/float4.sql",
+"postgreSQL/float8.sql",
+// SPARK-28885 String value is not allowed to be stored as date/timestamp
type with
+// ANSI store assignment policy.
+"postgreSQL/date.sql",
Review comment:
> I am not sure about whether there is a guideline in porting the PgSQL test
files. Such modifications are allowed, right?
Yea, I think that's ok. cc: @wangyum @dongjoon-hyun @HyukjinKwon
If you have no time to do, I'll will check next week.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #26107: [SPARK-28885][SQL] Follow ANSI store assignment rules in table insertion by default
gengliangwang commented on a change in pull request #26107: [SPARK-28885][SQL]
Follow ANSI store assignment rules in table insertion by default
URL: https://github.com/apache/spark/pull/26107#discussion_r341835365
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##
@@ -143,7 +143,19 @@ class SQLQueryTestSuite extends QueryTest with
SharedSparkSession {
/** List of test cases to ignore, in lower cases. */
protected def blackList: Set[String] = Set(
-"blacklist.sql" // Do NOT remove this one. It is here to test the
blacklist functionality.
+"blacklist.sql", // Do NOT remove this one. It is here to test the
blacklist functionality.
+// SPARK-28885 String value is not allowed to be stored as numeric type
with
+// ANSI store assignment policy.
+"postgreSQL/numeric.sql",
+"postgreSQL/int2.sql",
+"postgreSQL/int4.sql",
+"postgreSQL/int8.sql",
+"postgreSQL/float4.sql",
+"postgreSQL/float8.sql",
+// SPARK-28885 String value is not allowed to be stored as date/timestamp
type with
+// ANSI store assignment policy.
+"postgreSQL/date.sql",
Review comment:
How about changing the timestamp/date values from string literal to
timestamp/date literal in those sql files, just as
https://github.com/apache/spark/pull/26107/files#diff-431a4d1f056a06e853da8a60c46e9ca0R68
I am not sure about whether there is a guideline in porting the PgSQL test
files. Such modifications are allowed, right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #26373: [SPARK-29729][BUILD] Upgrade ASM to 7.2
maropu commented on issue #26373: [SPARK-29729][BUILD] Upgrade ASM to 7.2 URL: https://github.com/apache/spark/pull/26373#issuecomment-549106180 Yea, I've also checked all the unit tests passed except for that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26107: [SPARK-28885][SQL] Follow ANSI store assignment rules in table insertion by default
maropu commented on a change in pull request #26107: [SPARK-28885][SQL] Follow
ANSI store assignment rules in table insertion by default
URL: https://github.com/apache/spark/pull/26107#discussion_r341835227
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##
@@ -143,7 +143,19 @@ class SQLQueryTestSuite extends QueryTest with
SharedSparkSession {
/** List of test cases to ignore, in lower cases. */
protected def blackList: Set[String] = Set(
-"blacklist.sql" // Do NOT remove this one. It is here to test the
blacklist functionality.
+"blacklist.sql", // Do NOT remove this one. It is here to test the
blacklist functionality.
+// SPARK-28885 String value is not allowed to be stored as numeric type
with
+// ANSI store assignment policy.
+"postgreSQL/numeric.sql",
+"postgreSQL/int2.sql",
+"postgreSQL/int4.sql",
+"postgreSQL/int8.sql",
+"postgreSQL/float4.sql",
+"postgreSQL/float8.sql",
+// SPARK-28885 String value is not allowed to be stored as date/timestamp
type with
+// ANSI store assignment policy.
+"postgreSQL/date.sql",
Review comment:
But, before this pr's been merged, we tested the PgSQL tests in the LEGACY
mode? Is my understanding wrong? Personally, I think we need to explicitly file
issues in jira (Or, comment out them?) if we have inaccurate tests in `pgSQL/`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26107: [SPARK-28885][SQL] Follow ANSI store assignment rules in table insertion by default
maropu commented on a change in pull request #26107: [SPARK-28885][SQL] Follow
ANSI store assignment rules in table insertion by default
URL: https://github.com/apache/spark/pull/26107#discussion_r341835227
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##
@@ -143,7 +143,19 @@ class SQLQueryTestSuite extends QueryTest with
SharedSparkSession {
/** List of test cases to ignore, in lower cases. */
protected def blackList: Set[String] = Set(
-"blacklist.sql" // Do NOT remove this one. It is here to test the
blacklist functionality.
+"blacklist.sql", // Do NOT remove this one. It is here to test the
blacklist functionality.
+// SPARK-28885 String value is not allowed to be stored as numeric type
with
+// ANSI store assignment policy.
+"postgreSQL/numeric.sql",
+"postgreSQL/int2.sql",
+"postgreSQL/int4.sql",
+"postgreSQL/int8.sql",
+"postgreSQL/float4.sql",
+"postgreSQL/float8.sql",
+// SPARK-28885 String value is not allowed to be stored as date/timestamp
type with
+// ANSI store assignment policy.
+"postgreSQL/date.sql",
Review comment:
But, before this pr's been merged, we tested the PgSQL tests in the LEGACY
mode? Is my understanding wrong? Personally, I think we need to explicitly file
issues in jira if we have inaccurate tests in `pgSQL/`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang closed pull request #26336: [SPARK-29504][WEBUI] Toggle full job description on click
gengliangwang closed pull request #26336: [SPARK-29504][WEBUI] Toggle full job description on click URL: https://github.com/apache/spark/pull/26336 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #26336: [SPARK-29504][WEBUI] Toggle full job description on click
gengliangwang commented on issue #26336: [SPARK-29504][WEBUI] Toggle full job description on click URL: https://github.com/apache/spark/pull/26336#issuecomment-549105821 Hi @PavithraRamachandran , After building and running test manually with the changes, I found it doesn't work well: on double click the class `description-input` is removed. In the original PR https://github.com/apache/spark/pull/25484, the file `additional-metrics.js` is not removed. I am closing this PR now. Sorry I didn't look close into https://github.com/apache/spark/pull/25484 . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26107: [SPARK-28885][SQL] Follow ANSI store assignment rules in table insertion by default
maropu commented on a change in pull request #26107: [SPARK-28885][SQL] Follow
ANSI store assignment rules in table insertion by default
URL: https://github.com/apache/spark/pull/26107#discussion_r341835111
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##
@@ -143,7 +143,19 @@ class SQLQueryTestSuite extends QueryTest with
SharedSparkSession {
/** List of test cases to ignore, in lower cases. */
protected def blackList: Set[String] = Set(
-"blacklist.sql" // Do NOT remove this one. It is here to test the
blacklist functionality.
+"blacklist.sql", // Do NOT remove this one. It is here to test the
blacklist functionality.
+// SPARK-28885 String value is not allowed to be stored as numeric type
with
+// ANSI store assignment policy.
+"postgreSQL/numeric.sql",
+"postgreSQL/int2.sql",
+"postgreSQL/int4.sql",
+"postgreSQL/int8.sql",
+"postgreSQL/float4.sql",
+"postgreSQL/float8.sql",
+// SPARK-28885 String value is not allowed to be stored as date/timestamp
type with
+// ANSI store assignment policy.
+"postgreSQL/date.sql",
Review comment:
Ur, I see...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26107: [SPARK-28885][SQL] Follow ANSI store assignment rules in table insertion by default
maropu commented on a change in pull request #26107: [SPARK-28885][SQL] Follow
ANSI store assignment rules in table insertion by default
URL: https://github.com/apache/spark/pull/26107#discussion_r341835111
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##
@@ -143,7 +143,19 @@ class SQLQueryTestSuite extends QueryTest with
SharedSparkSession {
/** List of test cases to ignore, in lower cases. */
protected def blackList: Set[String] = Set(
-"blacklist.sql" // Do NOT remove this one. It is here to test the
blacklist functionality.
+"blacklist.sql", // Do NOT remove this one. It is here to test the
blacklist functionality.
+// SPARK-28885 String value is not allowed to be stored as numeric type
with
+// ANSI store assignment policy.
+"postgreSQL/numeric.sql",
+"postgreSQL/int2.sql",
+"postgreSQL/int4.sql",
+"postgreSQL/int8.sql",
+"postgreSQL/float4.sql",
+"postgreSQL/float8.sql",
+// SPARK-28885 String value is not allowed to be stored as date/timestamp
type with
+// ANSI store assignment policy.
+"postgreSQL/date.sql",
Review comment:
Ur, I see...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #26107: [SPARK-28885][SQL] Follow ANSI store assignment rules in table insertion by default
gengliangwang commented on a change in pull request #26107: [SPARK-28885][SQL]
Follow ANSI store assignment rules in table insertion by default
URL: https://github.com/apache/spark/pull/26107#discussion_r341834925
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##
@@ -143,7 +143,19 @@ class SQLQueryTestSuite extends QueryTest with
SharedSparkSession {
/** List of test cases to ignore, in lower cases. */
protected def blackList: Set[String] = Set(
-"blacklist.sql" // Do NOT remove this one. It is here to test the
blacklist functionality.
+"blacklist.sql", // Do NOT remove this one. It is here to test the
blacklist functionality.
+// SPARK-28885 String value is not allowed to be stored as numeric type
with
+// ANSI store assignment policy.
+"postgreSQL/numeric.sql",
+"postgreSQL/int2.sql",
+"postgreSQL/int4.sql",
+"postgreSQL/int8.sql",
+"postgreSQL/float4.sql",
+"postgreSQL/float8.sql",
+// SPARK-28885 String value is not allowed to be stored as date/timestamp
type with
+// ANSI store assignment policy.
+"postgreSQL/date.sql",
Review comment:
@maropu I am not sure about that.
For PgSQL, it disable inserting string values to numeric columns except for
literals. Setting `storeAssignmentPolicy=LEGACY` for all the PgSQL tests seems
inaccurate.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26373: [SPARK-29729][BUILD] Upgrade ASM to 7.2
dongjoon-hyun commented on issue #26373: [SPARK-29729][BUILD] Upgrade ASM to 7.2 URL: https://github.com/apache/spark/pull/26373#issuecomment-549105083 This passed all UTs twice successfully except the CRAN check. ``` * checking CRAN incoming feasibility ...Error in .check_package_CRAN_incoming(pkgdir) : dims [product 24] do not match the length of object [0] ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #26365: [SPARK-29720][CORE] Add linux condition to make ProcfsMetricsGetter more complete
viirya commented on a change in pull request #26365: [SPARK-29720][CORE] Add linux condition to make ProcfsMetricsGetter more complete URL: https://github.com/apache/spark/pull/26365#discussion_r341834873 ## File path: core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ## @@ -62,13 +64,14 @@ private[spark] class ProcfsMetricsGetter(procfsDir: String = "/proc/") extends L SparkEnv.get.conf.get(config.EVENT_LOG_STAGE_EXECUTOR_METRICS) val shouldLogStageExecutorProcessTreeMetrics = SparkEnv.get.conf.get(config.EVENT_LOG_PROCESS_TREE_METRICS) - procDirExists.get && shouldLogStageExecutorProcessTreeMetrics && shouldLogStageExecutorMetrics + procDirExists.get && shouldLogStageExecutorProcessTreeMetrics && +shouldLogStageExecutorMetrics && Shell.LINUX Review comment: Are we sure to support only Linux? Other Unix-like systems which have /proc? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
dongjoon-hyun closed pull request #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26107: [SPARK-28885][SQL] Follow ANSI store assignment rules in table insertion by default
maropu commented on a change in pull request #26107: [SPARK-28885][SQL] Follow
ANSI store assignment rules in table insertion by default
URL: https://github.com/apache/spark/pull/26107#discussion_r341834773
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##
@@ -143,7 +143,19 @@ class SQLQueryTestSuite extends QueryTest with
SharedSparkSession {
/** List of test cases to ignore, in lower cases. */
protected def blackList: Set[String] = Set(
-"blacklist.sql" // Do NOT remove this one. It is here to test the
blacklist functionality.
+"blacklist.sql", // Do NOT remove this one. It is here to test the
blacklist functionality.
+// SPARK-28885 String value is not allowed to be stored as numeric type
with
+// ANSI store assignment policy.
+"postgreSQL/numeric.sql",
+"postgreSQL/int2.sql",
+"postgreSQL/int4.sql",
+"postgreSQL/int8.sql",
+"postgreSQL/float4.sql",
+"postgreSQL/float8.sql",
+// SPARK-28885 String value is not allowed to be stored as date/timestamp
type with
+// ANSI store assignment policy.
+"postgreSQL/date.sql",
Review comment:
How about just setting `storeAssignmentPolicy=LEGACY` for the PgSQL tests?
They have a lots of INSERT queries having implicit casts from string literals
to numeric values.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
dongjoon-hyun commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549104725 I created SPARK-29732 first. - https://issues.apache.org/jira/browse/SPARK-29732 For the minimum version of R, could you raise an email to dev? If there is the best person to set it, I believe it is you, @shivaram , in the community. :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya edited a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
viirya edited a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549104521 It sounds fine to me as this actually blocks all PR dev. And because of weekend, CRAN sysadmins might not response quickly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
viirya commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549104521 It sounds fine to me as this actually blocks all PR dev. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
dongjoon-hyun commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549104542 Thank you all. Sure! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26365: [SPARK-29720][CORE] Add linux condition to make ProcfsMetricsGetter more complete
maropu commented on a change in pull request #26365: [SPARK-29720][CORE] Add linux condition to make ProcfsMetricsGetter more complete URL: https://github.com/apache/spark/pull/26365#discussion_r341834561 ## File path: core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ## @@ -26,6 +26,8 @@ import scala.collection.mutable import scala.collection.mutable.ArrayBuffer import scala.util.Try +import org.apache.hadoop.util.Shell Review comment: How about checking `os.name` instead of using hadoop-commons? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26365: [SPARK-29720][CORE] Add linux condition to make ProcfsMetricsGetter more complete
maropu commented on a change in pull request #26365: [SPARK-29720][CORE] Add linux condition to make ProcfsMetricsGetter more complete URL: https://github.com/apache/spark/pull/26365#discussion_r341834569 ## File path: core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ## @@ -62,13 +64,14 @@ private[spark] class ProcfsMetricsGetter(procfsDir: String = "/proc/") extends L SparkEnv.get.conf.get(config.EVENT_LOG_STAGE_EXECUTOR_METRICS) val shouldLogStageExecutorProcessTreeMetrics = SparkEnv.get.conf.get(config.EVENT_LOG_PROCESS_TREE_METRICS) - procDirExists.get && shouldLogStageExecutorProcessTreeMetrics && shouldLogStageExecutorMetrics + procDirExists.get && shouldLogStageExecutorProcessTreeMetrics && +shouldLogStageExecutorMetrics && Shell.LINUX Review comment: Let's move the os check in the first condition. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shivaram commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
shivaram commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549104172 I'm not sure running it just on releases is enough as that makes it hard to go back and figure out what change broke the CRAN test. So running it on R PRs seems necessary to me. I'm fine with unblocking the build for now if you can take care of opening a new JIRA / PR to track (1) and (2) above This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26373: [SPARK-29729][BUILD] Upgrade ASM to 7.2
AmplabJenkins removed a comment on issue #26373: [SPARK-29729][BUILD] Upgrade ASM to 7.2 URL: https://github.com/apache/spark/pull/26373#issuecomment-549103942 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113147/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26373: [SPARK-29729][BUILD] Upgrade ASM to 7.2
AmplabJenkins removed a comment on issue #26373: [SPARK-29729][BUILD] Upgrade ASM to 7.2 URL: https://github.com/apache/spark/pull/26373#issuecomment-549103941 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26373: [SPARK-29729][BUILD] Upgrade ASM to 7.2
AmplabJenkins commented on issue #26373: [SPARK-29729][BUILD] Upgrade ASM to 7.2 URL: https://github.com/apache/spark/pull/26373#issuecomment-549103941 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26373: [SPARK-29729][BUILD] Upgrade ASM to 7.2
AmplabJenkins commented on issue #26373: [SPARK-29729][BUILD] Upgrade ASM to 7.2 URL: https://github.com/apache/spark/pull/26373#issuecomment-549103942 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113147/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26373: [SPARK-29729][BUILD] Upgrade ASM to 7.2
SparkQA removed a comment on issue #26373: [SPARK-29729][BUILD] Upgrade ASM to 7.2 URL: https://github.com/apache/spark/pull/26373#issuecomment-549097424 **[Test build #113147 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113147/testReport)** for PR 26373 at commit [`4076e42`](https://github.com/apache/spark/commit/4076e427ed2d4d196453453275359b95101921bb). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
dongjoon-hyun edited a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549103727 Inside the new PR for one of (1) and (2), we can reenable this. How do you think about that, @shivaram and @viirya ? Currently, this blocks the almost whole community for 21 hours since Nov 1st midnight (PST). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26373: [SPARK-29729][BUILD] Upgrade ASM to 7.2
SparkQA commented on issue #26373: [SPARK-29729][BUILD] Upgrade ASM to 7.2 URL: https://github.com/apache/spark/pull/26373#issuecomment-549103882 **[Test build #113147 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113147/testReport)** for PR 26373 at commit [`4076e42`](https://github.com/apache/spark/commit/4076e427ed2d4d196453453275359b95101921bb). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
dongjoon-hyun edited a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549103727 Inside the new PR for one of (1) and (2), we can reenable this. How do you think about that, @shivaram and @viirya ? Currently, this blocks the whole community. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
dongjoon-hyun commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549103727 Inside the new PR for one of (1) and (2), we can reenable this. How do you think about that, @shivaram and @viirya ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
dongjoon-hyun commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549103674 I'm okay for both (1) invoking CRAN check for R PR only and (2) upgrading Jenkins R and raised the minimum R versions. However, let's unblock the PRBuilder first. Both (1) and (2) also needs some time to develop and validate. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
dongjoon-hyun commented on a change in pull request #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#discussion_r341834316 ## File path: R/run-tests.sh ## @@ -32,8 +32,10 @@ NUM_TEST_WARNING="$(grep -c -e 'Warnings ' $LOGFILE)" CRAN_CHECK_LOG_FILE=$FWDIR/cran-check.out rm -f $CRAN_CHECK_LOG_FILE -NO_TESTS=1 NO_MANUAL=1 $FWDIR/check-cran.sh 2>&1 | tee -a $CRAN_CHECK_LOG_FILE -FAILED=$((PIPESTATUS[0]||$FAILED)) +# SPARK-24152 We will add this test back as a separate Jenkins job Review comment: I guess the higher R version is going to hit this issue again after one year, but that sounds a good solution for now. How do you think about that, @shivaram and @felixcheung ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
dongjoon-hyun edited a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549103475 Thanks, @shivaram . > Do we know what the reason for the failure is? It's always happening their side changes. We don't know why this fails suddenly. As you see on SPARK-24152. This happens frequently in this year time to time . And, everytime, @viirya or @felixcheung contacted CRAN maintainers . > Also what is the work around in terms of detecting CRAN problems in a timely fashion? We can have separate jobs for `linter` and `publishing snapshot`. It's non-blocking jobs. > Can we say run this just for PRs that touch R code? That also can reduce this kind of massive outage. But, it's still be a blocker for all R PRs. CRAN check is only required only a few time when we do the release. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
dongjoon-hyun commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549103475 Thanks, @shivaram . > Do we know what the reason for the failure is? It's always happening their side changes. We don't know why this fails suddenly. As you see on SPARK-24152. This happens frequently in this year time to time . And, everytime, @viirya or @felixcheung contacted CRAN maintainers . > Also what is the work around in terms of detecting CRAN problems in a timely fashion? We can have separate jobs for `linter` and `publishing snapshot`. It's non-blocking jobs. > Can we say run this just for PRs that touch R code? That also can reduce this kind of massive outage. But, it's still be a blocker for all R PRs. CRAN check is only required only a few time when we do the release. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26376: [SPARK-29731][INFRA] Use public JIRA REST API to read-only access
AmplabJenkins removed a comment on issue #26376: [SPARK-29731][INFRA] Use public JIRA REST API to read-only access URL: https://github.com/apache/spark/pull/26376#issuecomment-549103127 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18065/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26376: [SPARK-29731][INFRA] Use public JIRA REST API to read-only access
AmplabJenkins removed a comment on issue #26376: [SPARK-29731][INFRA] Use public JIRA REST API to read-only access URL: https://github.com/apache/spark/pull/26376#issuecomment-549103125 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26376: [SPARK-29731][INFRA] Use public JIRA REST API to read-only access
SparkQA commented on issue #26376: [SPARK-29731][INFRA] Use public JIRA REST API to read-only access URL: https://github.com/apache/spark/pull/26376#issuecomment-549103421 **[Test build #113151 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113151/testReport)** for PR 26376 at commit [`877e007`](https://github.com/apache/spark/commit/877e0075f677e821f8840ef8a8afd3569e6d8626). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26366: [SPARK-29722][SQL] Non reversed keywords should be able to be used in high order functions
maropu commented on a change in pull request #26366: [SPARK-29722][SQL] Non reversed keywords should be able to be used in high order functions URL: https://github.com/apache/spark/pull/26366#discussion_r341834227 ## File path: sql/core/src/test/resources/sql-tests/inputs/higher-order-functions.sql ## @@ -12,9 +12,14 @@ select transform(zs, z -> z) as v from nested; -- Transform an array select transform(ys, y -> y * y) as v from nested; +-- use non reversed keywords +select transform(ys, left -> left * left) as v from nested; + -- Transform an array with index select transform(ys, (y, i) -> y + i) as v from nested; +-- use non reversed keywords +select transform(ys, (cost, i) -> cost + i) as v from nested; Review comment: Also, can you add some tests for error cases if ansi=true? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands
AmplabJenkins removed a comment on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26374#issuecomment-549103365 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113146/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands
AmplabJenkins commented on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26374#issuecomment-549103365 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113146/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands
AmplabJenkins removed a comment on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26374#issuecomment-549103363 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26366: [SPARK-29722][SQL] Non reversed keywords should be able to be used in high order functions
maropu commented on a change in pull request #26366: [SPARK-29722][SQL] Non reversed keywords should be able to be used in high order functions URL: https://github.com/apache/spark/pull/26366#discussion_r341834215 ## File path: sql/core/src/test/resources/sql-tests/inputs/higher-order-functions.sql ## @@ -12,9 +12,14 @@ select transform(zs, z -> z) as v from nested; -- Transform an array select transform(ys, y -> y * y) as v from nested; +-- use non reversed keywords +select transform(ys, left -> left * left) as v from nested; + -- Transform an array with index select transform(ys, (y, i) -> y + i) as v from nested; +-- use non reversed keywords +select transform(ys, (cost, i) -> cost + i) as v from nested; Review comment: Can you move the two tests above into the end of the file to make diff smaller in `sq.out`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands
AmplabJenkins commented on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26374#issuecomment-549103363 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands
SparkQA commented on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26374#issuecomment-549103315 **[Test build #113146 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113146/testReport)** for PR 26374 at commit [`b32975f`](https://github.com/apache/spark/commit/b32975f08f8a88c1c26e2c5136599468a7143301). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class AlterTableSerDePropertiesStatement(` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands
SparkQA removed a comment on issue #26374: [SPARK-29695][SQL] ALTER TABLE (SerDe properties) should look up catalog/table like v2 commands URL: https://github.com/apache/spark/pull/26374#issuecomment-549093312 **[Test build #113146 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113146/testReport)** for PR 26374 at commit [`b32975f`](https://github.com/apache/spark/commit/b32975f08f8a88c1c26e2c5136599468a7143301). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26366: [SPARK-29722][SQL] Non reversed keywords should be able to be used in high order functions
maropu commented on a change in pull request #26366: [SPARK-29722][SQL] Non
reversed keywords should be able to be used in high order functions
URL: https://github.com/apache/spark/pull/26366#discussion_r341834196
##
File path:
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
##
@@ -718,8 +718,8 @@ primaryExpression
| '(' query ')'
#subqueryExpression
| qualifiedName '(' (setQuantifier? argument+=expression (','
argument+=expression)*)? ')'
(OVER windowSpec)?
#functionCall
-| IDENTIFIER '->' expression
#lambda
-| '(' IDENTIFIER (',' IDENTIFIER)+ ')' '->' expression
#lambda
+| identifier '->' expression
#lambda
+| '(' identifier (',' identifier)+ ')' '->' expression
#lambda
Review comment:
Ur, I see. Looks reasonable to me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26376: [SPARK-29731][INFRA] Use public JIRA REST API to read-only access
AmplabJenkins commented on issue #26376: [SPARK-29731][INFRA] Use public JIRA REST API to read-only access URL: https://github.com/apache/spark/pull/26376#issuecomment-549103127 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18065/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
AmplabJenkins commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549103117 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
AmplabJenkins removed a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549103117 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
AmplabJenkins commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549103118 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113150/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26376: [SPARK-29731][INFRA] Use public JIRA REST API to read-only access
AmplabJenkins commented on issue #26376: [SPARK-29731][INFRA] Use public JIRA REST API to read-only access URL: https://github.com/apache/spark/pull/26376#issuecomment-549103125 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
AmplabJenkins removed a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549103118 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113150/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
SparkQA removed a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549102089 **[Test build #113150 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113150/testReport)** for PR 26375 at commit [`235c3af`](https://github.com/apache/spark/commit/235c3af3c79a771c97f4f80985f5f094bbbdd1c1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
SparkQA commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549103093 **[Test build #113150 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113150/testReport)** for PR 26375 at commit [`235c3af`](https://github.com/apache/spark/commit/235c3af3c79a771c97f4f80985f5f094bbbdd1c1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #26376: [SPARK-29731][INFRA] Use public JIRA REST API to read-only access
dongjoon-hyun opened a new pull request #26376: [SPARK-29731][INFRA] Use public JIRA REST API to read-only access URL: https://github.com/apache/spark/pull/26376 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce any user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
viirya commented on a change in pull request #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#discussion_r341833994 ## File path: R/run-tests.sh ## @@ -32,8 +32,10 @@ NUM_TEST_WARNING="$(grep -c -e 'Warnings ' $LOGFILE)" CRAN_CHECK_LOG_FILE=$FWDIR/cran-check.out rm -f $CRAN_CHECK_LOG_FILE -NO_TESTS=1 NO_MANUAL=1 $FWDIR/check-cran.sh 2>&1 | tee -a $CRAN_CHECK_LOG_FILE -FAILED=$((PIPESTATUS[0]||$FAILED)) +# SPARK-24152 We will add this test back as a separate Jenkins job Review comment: Do we plan to upgrade R version? I remember that a higher R version can avoid this kind of failure. cc @HyukjinKwon This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shivaram commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
shivaram commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549102618 Do we know what the reason for the failure is? Also what is the work around in terms of detecting CRAN problems in a timely fashion? Can we say run this just for PRs that touch R code? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
AmplabJenkins removed a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549102158 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18064/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
AmplabJenkins removed a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549102155 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #26371: [SPARK-27976][SQL] Add built-in Array Functions: array_append
maropu commented on a change in pull request #26371: [SPARK-27976][SQL] Add
built-in Array Functions: array_append
URL: https://github.com/apache/spark/pull/26371#discussion_r341833795
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
##
@@ -3928,3 +3928,88 @@ case class ArrayExcept(left: Expression, right:
Expression) extends ArrayBinaryL
override def prettyName: String = "array_except"
}
+
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array, element) - Returns an array of appending an element to the end
of an array
+ """,
+ examples = """
+Examples:
+ > SELECT _FUNC_(array(1, 2, 3), 3);
+ [1,2,3,3]
+ """,
+ since = "3.0.0")
+case class ArrayAppend(left: Expression, right: Expression) extends
BinaryExpression {
+
+ override def checkInputDataTypes(): TypeCheckResult = left.dataType match {
+case ArrayType(et, _) => right.dataType match {
+ case NullType =>
+TypeCheckResult.TypeCheckSuccess
+ case _: AtomicType =>
+TypeUtils.checkForSameTypeInputExpr(Seq(et, right.dataType),
s"function $prettyName")
+ case o => TypeCheckResult.TypeCheckFailure(
+s"function $prettyName not support append ${o.typeName} to
array[${et.typeName}]")
+}
+case NullType => right.dataType match {
+ case _: AtomicType =>
+TypeCheckResult.TypeCheckSuccess
+ case o => TypeCheckResult.TypeCheckFailure(
+s"function $prettyName not support append ${o.typeName} to array")
+}
+case o => TypeCheckResult.TypeCheckFailure(
+ s"function $prettyName not support append to ${o.typeName} type")
+ }
+
+ override def eval(input: InternalRow): Any = {
+val array = left.eval(input)
+val elem = right.eval(input)
+if (array == null) {
+ new GenericArrayData(Array(elem))
+} else {
+ new GenericArrayData(array.asInstanceOf[ArrayData].array ++ Array(elem))
+}
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode =
left.dataType match {
+case NullType =>
+ val (allocation, assigns, arrayData) =
+GenArrayData.genCodeToCreateArrayData(ctx, right.dataType, Seq(right),
prettyName)
+ ev.copy(
+code = code"""$allocation$assigns""",
+value = JavaCode.variable(arrayData, dataType),
+isNull = FalseLiteral)
+case ArrayType(et, _) => defineCodeGen(ctx, ev, (inputArray, ne) => {
+ val newArray = ctx.freshName("newArray")
+ val oldArraySize = ctx.freshName("oldArraySize")
+ val newArraySize = ctx.freshName("newArraySize")
+
+ val allocation =
+CodeGenerator.createArrayData(newArray, et, newArraySize,
s"$prettyName failed")
+
+ val i = ctx.freshName("i")
+ val assignment =
+CodeGenerator.createArrayAssignment(newArray, et, inputArray, i, i,
true)
+
+ val setNewValue =
+CodeGenerator.setArrayElement(newArray, et, oldArraySize, ne,
Some(s"$ne == null"))
+
+ s"""
+ |int $oldArraySize = $inputArray.numElements();
+ |int $newArraySize = $inputArray.numElements() + 1;
+ |$allocation
+ |for (int $i = 0; i < $inputArray.numElements(); $i ++) {
+ | $assignment
+ |}
+ |$setNewValue
+ |${ev.value} = $newArray;
+ |""".stripMargin
+})
+ }
+
+ override def dataType: DataType = left.dataType match {
Review comment:
Yea, thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
AmplabJenkins commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549102155 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
AmplabJenkins commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549102158 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18064/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
SparkQA commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549102089 **[Test build #113150 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113150/testReport)** for PR 26375 at commit [`235c3af`](https://github.com/apache/spark/commit/235c3af3c79a771c97f4f80985f5f094bbbdd1c1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #26371: [SPARK-27976][SQL] Add built-in Array Functions: array_append
yaooqinn commented on a change in pull request #26371: [SPARK-27976][SQL] Add
built-in Array Functions: array_append
URL: https://github.com/apache/spark/pull/26371#discussion_r341833606
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
##
@@ -3928,3 +3928,88 @@ case class ArrayExcept(left: Expression, right:
Expression) extends ArrayBinaryL
override def prettyName: String = "array_except"
}
+
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array, element) - Returns an array of appending an element to the end
of an array
+ """,
+ examples = """
+Examples:
+ > SELECT _FUNC_(array(1, 2, 3), 3);
+ [1,2,3,3]
+ """,
+ since = "3.0.0")
+case class ArrayAppend(left: Expression, right: Expression) extends
BinaryExpression {
+
+ override def checkInputDataTypes(): TypeCheckResult = left.dataType match {
+case ArrayType(et, _) => right.dataType match {
+ case NullType =>
+TypeCheckResult.TypeCheckSuccess
+ case _: AtomicType =>
+TypeUtils.checkForSameTypeInputExpr(Seq(et, right.dataType),
s"function $prettyName")
+ case o => TypeCheckResult.TypeCheckFailure(
+s"function $prettyName not support append ${o.typeName} to
array[${et.typeName}]")
+}
+case NullType => right.dataType match {
+ case _: AtomicType =>
+TypeCheckResult.TypeCheckSuccess
+ case o => TypeCheckResult.TypeCheckFailure(
+s"function $prettyName not support append ${o.typeName} to array")
+}
+case o => TypeCheckResult.TypeCheckFailure(
+ s"function $prettyName not support append to ${o.typeName} type")
+ }
+
+ override def eval(input: InternalRow): Any = {
+val array = left.eval(input)
+val elem = right.eval(input)
+if (array == null) {
+ new GenericArrayData(Array(elem))
+} else {
+ new GenericArrayData(array.asInstanceOf[ArrayData].array ++ Array(elem))
+}
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode =
left.dataType match {
+case NullType =>
+ val (allocation, assigns, arrayData) =
+GenArrayData.genCodeToCreateArrayData(ctx, right.dataType, Seq(right),
prettyName)
+ ev.copy(
+code = code"""$allocation$assigns""",
+value = JavaCode.variable(arrayData, dataType),
+isNull = FalseLiteral)
+case ArrayType(et, _) => defineCodeGen(ctx, ev, (inputArray, ne) => {
+ val newArray = ctx.freshName("newArray")
+ val oldArraySize = ctx.freshName("oldArraySize")
+ val newArraySize = ctx.freshName("newArraySize")
+
+ val allocation =
+CodeGenerator.createArrayData(newArray, et, newArraySize,
s"$prettyName failed")
+
+ val i = ctx.freshName("i")
+ val assignment =
+CodeGenerator.createArrayAssignment(newArray, et, inputArray, i, i,
true)
+
+ val setNewValue =
+CodeGenerator.setArrayElement(newArray, et, oldArraySize, ne,
Some(s"$ne == null"))
+
+ s"""
+ |int $oldArraySize = $inputArray.numElements();
+ |int $newArraySize = $inputArray.numElements() + 1;
+ |$allocation
+ |for (int $i = 0; i < $inputArray.numElements(); $i ++) {
+ | $assignment
+ |}
+ |$setNewValue
+ |${ev.value} = $newArray;
+ |""".stripMargin
+})
+ }
+
+ override def dataType: DataType = left.dataType match {
Review comment:
I got your point here. thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
AmplabJenkins removed a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549101515 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113149/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
SparkQA commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549101507 **[Test build #113149 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113149/testReport)** for PR 26375 at commit [`0f2ce08`](https://github.com/apache/spark/commit/0f2ce08b673ac33ae53400e22b376553d193016b). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
AmplabJenkins removed a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549101513 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
AmplabJenkins commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549101513 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
SparkQA removed a comment on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549100115 **[Test build #113149 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113149/testReport)** for PR 26375 at commit [`0f2ce08`](https://github.com/apache/spark/commit/0f2ce08b673ac33ae53400e22b376553d193016b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh
AmplabJenkins commented on issue #26375: [SPARK-24152][R][TESTS] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549101515 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113149/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on issue #26365: [SPARK-29720][CORE] Add linux condition to make ProcfsMetricsGetter more complete
ulysses-you commented on issue #26365: [SPARK-29720][CORE] Add linux condition to make ProcfsMetricsGetter more complete URL: https://github.com/apache/spark/pull/26365#issuecomment-549101205 This catch just confirm `echo $PPID ` is support for current os . But not sure whether current os support `porc`. E.g. mac os, `echo $PPID ` can return the right pid. Add Linux condition is just to avoid that other os has the same dir like `/proc/$pid/stat` but in different use case. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #26375: [SPARK-24152][R] Disable check-cran from run-tests.sh
dongjoon-hyun edited a comment on issue #26375: [SPARK-24152][R] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549100188 @shivaram , @felixcheung , @HyukjinKwon , @viirya , @srowen . I made a PR to recover our PRBuilder and 4 Jenkins jobs. How do you think about this? In fact, CRAN failure occurs more frequently than PG power shutdown in this year. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26375: [SPARK-24152][R] Disable check-cran from run-tests.sh
AmplabJenkins commented on issue #26375: [SPARK-24152][R] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549100202 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18063/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26375: [SPARK-24152][R] Disable check-cran from run-tests.sh
AmplabJenkins removed a comment on issue #26375: [SPARK-24152][R] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549100200 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26375: [SPARK-24152][R] Disable check-cran from run-tests.sh
AmplabJenkins commented on issue #26375: [SPARK-24152][R] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549100200 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26375: [SPARK-24152][R] Disable check-cran from run-tests.sh
AmplabJenkins removed a comment on issue #26375: [SPARK-24152][R] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549100202 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18063/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26375: [SPARK-24152][R] Disable check-cran from run-tests.sh
dongjoon-hyun commented on issue #26375: [SPARK-24152][R] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549100188 @shivaram , @felixcheung , @HyukjinKwon , @viirya . I made a PR to recover our PRBuilder and 4 Jenkins jobs. How do you think about this? In fact, CRAN failure occurs more frequently than PG power shutdown in this year. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26375: [SPARK-24152][R] Disable check-cran from run-tests.sh
SparkQA commented on issue #26375: [SPARK-24152][R] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375#issuecomment-549100115 **[Test build #113149 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113149/testReport)** for PR 26375 at commit [`0f2ce08`](https://github.com/apache/spark/commit/0f2ce08b673ac33ae53400e22b376553d193016b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #26375: [SPARK-24152][R] Disable check-cran from run-tests.sh
dongjoon-hyun opened a new pull request #26375: [SPARK-24152][R] Disable check-cran from run-tests.sh URL: https://github.com/apache/spark/pull/26375 ### What changes were proposed in this pull request? This ### Why are the changes needed? ### Does this PR introduce any user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26366: [SPARK-29722][SQL] Non reversed keywords should be able to be used in high order functions
AmplabJenkins commented on issue #26366: [SPARK-29722][SQL] Non reversed keywords should be able to be used in high order functions URL: https://github.com/apache/spark/pull/26366#issuecomment-549099383 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26366: [SPARK-29722][SQL] Non reversed keywords should be able to be used in high order functions
AmplabJenkins removed a comment on issue #26366: [SPARK-29722][SQL] Non reversed keywords should be able to be used in high order functions URL: https://github.com/apache/spark/pull/26366#issuecomment-549099385 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18062/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26366: [SPARK-29722][SQL] Non reversed keywords should be able to be used in high order functions
AmplabJenkins commented on issue #26366: [SPARK-29722][SQL] Non reversed keywords should be able to be used in high order functions URL: https://github.com/apache/spark/pull/26366#issuecomment-549099385 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18062/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26366: [SPARK-29722][SQL] Non reversed keywords should be able to be used in high order functions
AmplabJenkins removed a comment on issue #26366: [SPARK-29722][SQL] Non reversed keywords should be able to be used in high order functions URL: https://github.com/apache/spark/pull/26366#issuecomment-549099383 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on issue #26366: [SPARK-29722][SQL] Non reversed keywords should be able to be used in high order functions
yaooqinn commented on issue #26366: [SPARK-29722][SQL] Non reversed keywords should be able to be used in high order functions URL: https://github.com/apache/spark/pull/26366#issuecomment-549099269 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26366: [SPARK-29722][SQL] Non reversed keywords should be able to be used in high order functions
SparkQA commented on issue #26366: [SPARK-29722][SQL] Non reversed keywords should be able to be used in high order functions URL: https://github.com/apache/spark/pull/26366#issuecomment-549099301 **[Test build #113148 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113148/testReport)** for PR 26366 at commit [`2895b3b`](https://github.com/apache/spark/commit/2895b3b07426855220336908ad50c253a9848efe). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #26371: [SPARK-27976][SQL] Add built-in Array Functions: array_append
yaooqinn commented on a change in pull request #26371: [SPARK-27976][SQL] Add
built-in Array Functions: array_append
URL: https://github.com/apache/spark/pull/26371#discussion_r341832868
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
##
@@ -3928,3 +3928,88 @@ case class ArrayExcept(left: Expression, right:
Expression) extends ArrayBinaryL
override def prettyName: String = "array_except"
}
+
+@ExpressionDescription(
+ usage = """
+ _FUNC_(array, element) - Returns an array of appending an element to the end
of an array
+ """,
+ examples = """
+Examples:
+ > SELECT _FUNC_(array(1, 2, 3), 3);
+ [1,2,3,3]
+ """,
+ since = "3.0.0")
+case class ArrayAppend(left: Expression, right: Expression) extends
BinaryExpression {
+
+ override def checkInputDataTypes(): TypeCheckResult = left.dataType match {
+case ArrayType(et, _) => right.dataType match {
+ case NullType =>
+TypeCheckResult.TypeCheckSuccess
+ case _: AtomicType =>
+TypeUtils.checkForSameTypeInputExpr(Seq(et, right.dataType),
s"function $prettyName")
+ case o => TypeCheckResult.TypeCheckFailure(
+s"function $prettyName not support append ${o.typeName} to
array[${et.typeName}]")
+}
+case NullType => right.dataType match {
+ case _: AtomicType =>
+TypeCheckResult.TypeCheckSuccess
+ case o => TypeCheckResult.TypeCheckFailure(
+s"function $prettyName not support append ${o.typeName} to array")
+}
+case o => TypeCheckResult.TypeCheckFailure(
+ s"function $prettyName not support append to ${o.typeName} type")
+ }
+
+ override def eval(input: InternalRow): Any = {
+val array = left.eval(input)
+val elem = right.eval(input)
+if (array == null) {
+ new GenericArrayData(Array(elem))
+} else {
+ new GenericArrayData(array.asInstanceOf[ArrayData].array ++ Array(elem))
+}
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode =
left.dataType match {
+case NullType =>
+ val (allocation, assigns, arrayData) =
+GenArrayData.genCodeToCreateArrayData(ctx, right.dataType, Seq(right),
prettyName)
+ ev.copy(
+code = code"""$allocation$assigns""",
+value = JavaCode.variable(arrayData, dataType),
+isNull = FalseLiteral)
+case ArrayType(et, _) => defineCodeGen(ctx, ev, (inputArray, ne) => {
+ val newArray = ctx.freshName("newArray")
+ val oldArraySize = ctx.freshName("oldArraySize")
+ val newArraySize = ctx.freshName("newArraySize")
+
+ val allocation =
+CodeGenerator.createArrayData(newArray, et, newArraySize,
s"$prettyName failed")
+
+ val i = ctx.freshName("i")
+ val assignment =
+CodeGenerator.createArrayAssignment(newArray, et, inputArray, i, i,
true)
+
+ val setNewValue =
+CodeGenerator.setArrayElement(newArray, et, oldArraySize, ne,
Some(s"$ne == null"))
+
+ s"""
+ |int $oldArraySize = $inputArray.numElements();
+ |int $newArraySize = $inputArray.numElements() + 1;
+ |$allocation
+ |for (int $i = 0; i < $inputArray.numElements(); $i ++) {
+ | $assignment
+ |}
+ |$setNewValue
+ |${ev.value} = $newArray;
+ |""".stripMargin
+})
+ }
+
+ override def dataType: DataType = left.dataType match {
Review comment:
all checked in `checkInputDataTypes`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26256: [SPARK-29605][SQL] Optimize string to interval casting
AmplabJenkins removed a comment on issue #26256: [SPARK-29605][SQL] Optimize string to interval casting URL: https://github.com/apache/spark/pull/26256#issuecomment-549098815 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113145/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26256: [SPARK-29605][SQL] Optimize string to interval casting
AmplabJenkins commented on issue #26256: [SPARK-29605][SQL] Optimize string to interval casting URL: https://github.com/apache/spark/pull/26256#issuecomment-549098813 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26256: [SPARK-29605][SQL] Optimize string to interval casting
AmplabJenkins removed a comment on issue #26256: [SPARK-29605][SQL] Optimize string to interval casting URL: https://github.com/apache/spark/pull/26256#issuecomment-549098813 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26256: [SPARK-29605][SQL] Optimize string to interval casting
AmplabJenkins commented on issue #26256: [SPARK-29605][SQL] Optimize string to interval casting URL: https://github.com/apache/spark/pull/26256#issuecomment-549098815 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113145/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26256: [SPARK-29605][SQL] Optimize string to interval casting
SparkQA removed a comment on issue #26256: [SPARK-29605][SQL] Optimize string to interval casting URL: https://github.com/apache/spark/pull/26256#issuecomment-549090317 **[Test build #113145 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113145/testReport)** for PR 26256 at commit [`9dfb45d`](https://github.com/apache/spark/commit/9dfb45d582d25e8e07a226e34f55840345e1b78c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
