[GitHub] [spark] SparkQA commented on pull request #33238: [SPARK-36020][SQL][3.1] Check logical link in remove redundant projects
SparkQA commented on pull request #33238: URL: https://github.com/apache/spark/pull/33238#issuecomment-875305184 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45251/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33239: [WIP][SPARK-36030][SQL] Support DS v2 metrics at writing path
SparkQA commented on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-875304144 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45250/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range
SparkQA commented on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-875303768 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45247/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33243: [SPARK-36016][SQL] Support TimestampNTZType in expression ApproxCountDistinctForIntervals
SparkQA commented on pull request #33243: URL: https://github.com/apache/spark/pull/33243#issuecomment-875303096 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45249/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33238: [SPARK-36020][SQL][3.1] Check logical link in remove redundant projects
SparkQA commented on pull request #33238: URL: https://github.com/apache/spark/pull/33238#issuecomment-875301851 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45248/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range
SparkQA removed a comment on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-875237620 **[Test build #140730 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140730/testReport)** for PR 32959 at commit [`d9f8af4`](https://github.com/apache/spark/commit/d9f8af4977a794661fc1bd3188100d7eee642e47). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range
SparkQA commented on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-875301018 **[Test build #140730 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140730/testReport)** for PR 32959 at commit [`d9f8af4`](https://github.com/apache/spark/commit/d9f8af4977a794661fc1bd3188100d7eee642e47). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33242: [SPARK-35735][SQL][FOLLOWUP] Fix case minute to second regex can cover by hour to minute
SparkQA commented on pull request #33242: URL: https://github.com/apache/spark/pull/33242#issuecomment-875297418 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45243/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33164: [SPARK-35958][CORE] Refactor SparkError.scala to SparkThrowable.java
SparkQA removed a comment on pull request #33164: URL: https://github.com/apache/spark/pull/33164#issuecomment-875234189 **[Test build #140729 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140729/testReport)** for PR 33164 at commit [`d6f93b7`](https://github.com/apache/spark/commit/d6f93b77e498702756b4ef31119122166c43fda2). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33164: [SPARK-35958][CORE] Refactor SparkError.scala to SparkThrowable.java
SparkQA commented on pull request #33164: URL: https://github.com/apache/spark/pull/33164#issuecomment-875293430 **[Test build #140729 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140729/testReport)** for PR 33164 at commit [`d6f93b7`](https://github.com/apache/spark/commit/d6f93b77e498702756b4ef31119122166c43fda2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33240: [SPARK-35906][SQL][FOLLOWUP] Recursive remove sort if the maximum number of rows less than or equal to 1
HyukjinKwon closed pull request #33240: URL: https://github.com/apache/spark/pull/33240 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33231: [SPARK-36021][SQL] Parse interval literals should support more than 2 digits
SparkQA commented on pull request #33231: URL: https://github.com/apache/spark/pull/33231#issuecomment-875293070 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45245/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33240: [SPARK-35906][SQL][FOLLOWUP] Recursive remove sort if the maximum number of rows less than or equal to 1
HyukjinKwon commented on pull request #33240: URL: https://github.com/apache/spark/pull/33240#issuecomment-875293047 Merged to master and branch-3.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions
SparkQA commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-875293024 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45246/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame
SparkQA removed a comment on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-875188042 **[Test build #140723 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140723/testReport)** for PR 33214 at commit [`52a9a70`](https://github.com/apache/spark/commit/52a9a70fa3e5b720b41e2ff4e9177a5d201b471f). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame
SparkQA commented on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-875291908 **[Test build #140723 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140723/testReport)** for PR 33214 at commit [`52a9a70`](https://github.com/apache/spark/commit/52a9a70fa3e5b720b41e2ff4e9177a5d201b471f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33238: [SPARK-36020][SQL][3.1] Check logical link in remove redundant projects
SparkQA commented on pull request #33238: URL: https://github.com/apache/spark/pull/33238#issuecomment-875282707 **[Test build #140741 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140741/testReport)** for PR 33238 at commit [`b8ef9a8`](https://github.com/apache/spark/commit/b8ef9a813e9e3d2b4c6b1e18322d2efa075eafd3). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33239: [WIP][SPARK-36030][SQL] Support DS v2 metrics at writing path
SparkQA commented on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-875282623 **[Test build #140740 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140740/testReport)** for PR 33239 at commit [`05800b4`](https://github.com/apache/spark/commit/05800b4f4a28887f6f2070cbb1d55cbf4ea08fd1). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33238: [SPARK-36020][SQL][3.1] Check logical link in remove redundant projects
SparkQA removed a comment on pull request #33238: URL: https://github.com/apache/spark/pull/33238#issuecomment-875215847 **[Test build #140724 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140724/testReport)** for PR 33238 at commit [`4fdb921`](https://github.com/apache/spark/commit/4fdb92106c846db283d5b1a5f26a8b331b10b7bf). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33238: [SPARK-36020][SQL][3.1] Check logical link in remove redundant projects
AmplabJenkins removed a comment on pull request #33238: URL: https://github.com/apache/spark/pull/33238#issuecomment-875281167 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140724/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33238: [SPARK-36020][SQL][3.1] Check logical link in remove redundant projects
AmplabJenkins commented on pull request #33238: URL: https://github.com/apache/spark/pull/33238#issuecomment-875281167 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140724/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33241: [SPARK-36017][SQL] Support TimestampNTZType in expression ApproximatePercentile
AmplabJenkins removed a comment on pull request #33241: URL: https://github.com/apache/spark/pull/33241#issuecomment-875280594 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45244/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range
AmplabJenkins removed a comment on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-875280599 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45241/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33164: [SPARK-35958][CORE] Refactor SparkError.scala to SparkThrowable.java
AmplabJenkins removed a comment on pull request #33164: URL: https://github.com/apache/spark/pull/33164#issuecomment-875280597 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45239/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions
AmplabJenkins removed a comment on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-875280593 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140736/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33239: [WIP][SPARK-36030][SQL] Support DS v2 metrics at writing path
AmplabJenkins removed a comment on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-875280595 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45238/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33240: [SPARK-35906][SQL][FOLLOWUP] Recursive remove sort if the maximum number of rows less than or equal to 1
AmplabJenkins removed a comment on pull request #33240: URL: https://github.com/apache/spark/pull/33240#issuecomment-875280596 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45240/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33238: [SPARK-36020][SQL][3.1] Check logical link in remove redundant projects
SparkQA commented on pull request #33238: URL: https://github.com/apache/spark/pull/33238#issuecomment-875280746 **[Test build #140724 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140724/testReport)** for PR 33238 at commit [`4fdb921`](https://github.com/apache/spark/commit/4fdb92106c846db283d5b1a5f26a8b331b10b7bf). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33164: [SPARK-35958][CORE] Refactor SparkError.scala to SparkThrowable.java
AmplabJenkins commented on pull request #33164: URL: https://github.com/apache/spark/pull/33164#issuecomment-875280597 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45239/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions
AmplabJenkins commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-875280593 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140736/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33240: [SPARK-35906][SQL][FOLLOWUP] Recursive remove sort if the maximum number of rows less than or equal to 1
AmplabJenkins commented on pull request #33240: URL: https://github.com/apache/spark/pull/33240#issuecomment-875280596 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45240/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33241: [SPARK-36017][SQL] Support TimestampNTZType in expression ApproximatePercentile
AmplabJenkins commented on pull request #33241: URL: https://github.com/apache/spark/pull/33241#issuecomment-875280594 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45244/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33239: [WIP][SPARK-36030][SQL] Support DS v2 metrics at writing path
AmplabJenkins commented on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-875280595 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45238/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range
AmplabJenkins commented on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-875280599 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45241/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33242: [SPARK-35735][SQL][FOLLOWUP] Fix case minute to second regex can cover by hour to minute
SparkQA commented on pull request #33242: URL: https://github.com/apache/spark/pull/33242#issuecomment-875280189 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45243/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] linhongliu-db commented on a change in pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range
linhongliu-db commented on a change in pull request #32959:
URL: https://github.com/apache/spark/pull/32959#discussion_r665015489
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##
@@ -249,70 +249,107 @@ object DateTimeUtils {
* the input string can't be parsed as timestamp, the result
timestamp segments are empty.
*/
def parseTimestampString(s: UTF8String): (Array[Int], Option[ZoneId],
Boolean) = {
-if (s == null) {
+def isValidDigits(segment: Int, digits: Int): Boolean = {
+ // A Long is able to represent a timestamp within [+-]200 thousand years
+ val maxDigitsYear = 6
+ // For the nanosecond part, more than 6 digits is allowed, but will be
truncated.
+ segment == 6 || (segment == 0 && digits > 0 && digits <= maxDigitsYear)
||
+(segment != 0 && segment != 6 && digits <= 2)
Review comment:
segments except year are allowed to have 0 digits before this PR. so I
didn't do zero checks for these segments.
for example, before this PR, the below query is valid:
```sql
select cast('12::' as timestamp); -- output: 2021-07-07 12:00:00
select cast('T' as timestamp); -- output: 2021-07-07 00:00:00
```
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##
@@ -249,70 +249,107 @@ object DateTimeUtils {
* the input string can't be parsed as timestamp, the result
timestamp segments are empty.
*/
def parseTimestampString(s: UTF8String): (Array[Int], Option[ZoneId],
Boolean) = {
-if (s == null) {
+def isValidDigits(segment: Int, digits: Int): Boolean = {
+ // A Long is able to represent a timestamp within [+-]200 thousand years
+ val maxDigitsYear = 6
+ // For the nanosecond part, more than 6 digits is allowed, but will be
truncated.
+ segment == 6 || (segment == 0 && digits > 0 && digits <= maxDigitsYear)
||
+(segment != 0 && segment != 6 && digits <= 2)
Review comment:
segments except year are allowed to have 0 digits before this PR. so I
didn't do zero checks for these segments.
for example, before and after this PR, the below query is valid:
```sql
select cast('12::' as timestamp); -- output: 2021-07-07 12:00:00
select cast('T' as timestamp); -- output: 2021-07-07 00:00:00
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33231: [SPARK-36021][SQL] Parse interval literals should support more than 2 digits
SparkQA commented on pull request #33231: URL: https://github.com/apache/spark/pull/33231#issuecomment-875276219 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45245/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions
SparkQA commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-875276165 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45246/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33241: [SPARK-36017][SQL] Support TimestampNTZType in expression ApproximatePercentile
SparkQA commented on pull request #33241: URL: https://github.com/apache/spark/pull/33241#issuecomment-875273128 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45244/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range
SparkQA commented on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-875272342 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45241/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32872: [SPARK-35639][SQL] Make hasCoalescedPartition return true if something was actually coalesced
ulysses-you commented on a change in pull request #32872:
URL: https://github.com/apache/spark/pull/32872#discussion_r665031937
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##
@@ -1567,7 +1567,7 @@ class AdaptiveQueryExecSuite
assert(!smj.head.isSkewJoin)
// Both sides are coalesced.
val customReaders = collect(smj.head) {
- case c: CustomShuffleReaderExec if c.hasCoalescedPartition => c
+ case c: CustomShuffleReaderExec => c
Review comment:
I think here have two things:
1. hasCoalescedPartition; `CustomShuffleReaderExec` has some
`CoalescedPartitionSpec`
2. coalesced; `CoalescedPartitionSpec` has some coalesced partition (exists
`startReducerIndex + 1 != endReducerIndex`)
The `stringArgs` method we want to change is for option 2, so I think is it
better to create new method `coalesced` to make them clearly ? Then we do not
need to change the `hasCoalescedPartition` method. cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #33238: [SPARK-36020][SQL][3.1] Check logical link in remove redundant projects
ulysses-you commented on a change in pull request #33238:
URL: https://github.com/apache/spark/pull/33238#discussion_r665033163
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##
@@ -1481,4 +1481,26 @@ class AdaptiveQueryExecSuite
}
}
}
+
+ test("SPARK-36020: Check logical link in remove redundant projects") {
+withTempView("t") {
+ spark.range(10).selectExpr("id % 10 as key", "cast(id * 2 as int) as a",
+"cast(id * 3 as int) as b", "array(id, id + 1, id + 3) as
c").createOrReplaceTempView("t")
+ withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true") {
+val query =
+ """
+|WITH tt AS (
+| SELECT key, a, b, explode(c) AS c FROM t
+|)
+|SELECT t1.key, t1.c, t2.key, t2.c
+|FROM (SELECT a, b, c, key FROM tt WHERE a > 1) t1
+|JOIN (SELECT a, b, c, key FROM tt) t2
+| ON t1.key = t2.key
+|""".stripMargin
+val (origin, adaptive) = runAdaptiveAndVerifyResult(query)
Review comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions
SparkQA removed a comment on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-875257101 **[Test build #140736 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140736/testReport)** for PR 33223 at commit [`df5e08b`](https://github.com/apache/spark/commit/df5e08bde6ee967235605de9f4c5971dcda04f27). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions
SparkQA commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-875268934 **[Test build #140736 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140736/testReport)** for PR 33223 at commit [`df5e08b`](https://github.com/apache/spark/commit/df5e08bde6ee967235605de9f4c5971dcda04f27). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #32872: [SPARK-35639][SQL] Make hasCoalescedPartition return true if something was actually coalesced
ulysses-you commented on a change in pull request #32872:
URL: https://github.com/apache/spark/pull/32872#discussion_r665031937
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##
@@ -1567,7 +1567,7 @@ class AdaptiveQueryExecSuite
assert(!smj.head.isSkewJoin)
// Both sides are coalesced.
val customReaders = collect(smj.head) {
- case c: CustomShuffleReaderExec if c.hasCoalescedPartition => c
+ case c: CustomShuffleReaderExec => c
Review comment:
I think here have two things:
1. hasCoalescedPartition; `CustomShuffleReaderExec` has some
`CoalescedPartitionSpec`
2. coalesced; `CoalescedPartitionSpec` has some coalesced partition (exists
`startReducerIndex + 1 != endReducerIndex`)
The `stringArgs` method we want to change is for option 2, so I think is it
better to create new method `coalesced` to make them clearly ? cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33240: [SPARK-35906][SQL][FOLLOWUP] Recursive remove sort if the maximum number of rows less than or equal to 1
SparkQA commented on pull request #33240: URL: https://github.com/apache/spark/pull/33240#issuecomment-875268157 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45240/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions
HyukjinKwon closed pull request #33223: URL: https://github.com/apache/spark/pull/33223 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions
HyukjinKwon commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-875266531 Merged to master and branch-3.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #33235: [SPARK-36028][SQL] Allow Project to host outer references in scalar subqueries
cloud-fan closed pull request #33235: URL: https://github.com/apache/spark/pull/33235 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions
HyukjinKwon commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-875266206 Python linter passed. Let me get this in! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #33235: [SPARK-36028][SQL] Allow Project to host outer references in scalar subqueries
cloud-fan commented on pull request #33235: URL: https://github.com/apache/spark/pull/33235#issuecomment-875266259 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33164: [SPARK-35958][CORE] Refactor SparkError.scala to SparkThrowable.java
SparkQA commented on pull request #33164: URL: https://github.com/apache/spark/pull/33164#issuecomment-875266249 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45239/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33239: [WIP][SPARK-36030][SQL] Support DS v2 metrics at writing path
SparkQA commented on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-875265372 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45238/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33232: [SPARK-36027][SQL] Add the code change to pushdown filter in case of typedFilter
HyukjinKwon commented on a change in pull request #33232:
URL: https://github.com/apache/spark/pull/33232#discussion_r665029120
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##
@@ -1442,6 +1442,12 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
pushDownPredicate(filter, u.child) { predicate =>
u.withNewChildren(Seq(Filter(predicate, u.child)))
}
+
+// Push down filter predicates in case filter having child as TypedFilter.
+// In this scenario inorder to push the filter predicates there is need to
+// to push Filter beneath the TypedFilter.
+case Filter(condition, typeFilter @ TypedFilter(_, _, _, _, _)) =>
+ typeFilter.copy(child = Filter(condition, typeFilter.child))
Review comment:
The only thing makes me worried is `TypedFilter` takes an arbitrary
function that can depend on, e.g, number of records being filtered or any sort
of global variable for the task. Once the order is switched, the logic can be
broken. e.g.)
```scala
object RecordFilteredPerTask {
var count = 0
}
df.filter({ v =>
RecordFilteredPerTask.count += 1
RecordFilteredPerTask.count > 5
})
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33232: [SPARK-36027][SQL] Add the code change to pushdown filter in case of typedFilter
HyukjinKwon commented on a change in pull request #33232:
URL: https://github.com/apache/spark/pull/33232#discussion_r665029120
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##
@@ -1442,6 +1442,12 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
pushDownPredicate(filter, u.child) { predicate =>
u.withNewChildren(Seq(Filter(predicate, u.child)))
}
+
+// Push down filter predicates in case filter having child as TypedFilter.
+// In this scenario inorder to push the filter predicates there is need to
+// to push Filter beneath the TypedFilter.
+case Filter(condition, typeFilter @ TypedFilter(_, _, _, _, _)) =>
+ typeFilter.copy(child = Filter(condition, typeFilter.child))
Review comment:
The only thing makes me worried is `TypedFilter` takes an arbitrary
function that can depend on, e.g, number of records being filtered or any sort
of global variable for the task. Once the order is switched, the logic can be
broken. e.g.)
```scala
object RecordFilteredPerTask {
var count = 0
}
df.filter({ v =>
RecordFilteredPerTask.count += 1
RecordFilteredPerTask.count > 5
})
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33242: [SPARK-35735][SQL][FOLLOWUP] Fix case minute to second regex can cover by hour to minute
cloud-fan commented on a change in pull request #33242:
URL: https://github.com/apache/spark/pull/33242#discussion_r665028561
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##
@@ -327,8 +327,8 @@ object IntervalUtils {
toDTInterval(day, hour, minute, secondAndMicro(second, micro),
finalSign(firstSign, secondSign))
- case hourMinuteRegex(sign, hour, minute) =>
-checkDTIntervalStringDataType(DT(DT.HOUR, DT.MINUTE))
+ case hourMinuteRegex(sign, hour, minute)
+if startField == DT.HOUR && endField == DT.MINUTE =>
Review comment:
SGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33238: [SPARK-36020][SQL][3.1] Check logical link in remove redundant projects
cloud-fan commented on a change in pull request #33238:
URL: https://github.com/apache/spark/pull/33238#discussion_r665028096
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##
@@ -1481,4 +1481,26 @@ class AdaptiveQueryExecSuite
}
}
}
+
+ test("SPARK-36020: Check logical link in remove redundant projects") {
+withTempView("t") {
+ spark.range(10).selectExpr("id % 10 as key", "cast(id * 2 as int) as a",
+"cast(id * 3 as int) as b", "array(id, id + 1, id + 3) as
c").createOrReplaceTempView("t")
+ withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true") {
+val query =
+ """
+|WITH tt AS (
+| SELECT key, a, b, explode(c) AS c FROM t
+|)
+|SELECT t1.key, t1.c, t2.key, t2.c
+|FROM (SELECT a, b, c, key FROM tt WHERE a > 1) t1
+|JOIN (SELECT a, b, c, key FROM tt) t2
+| ON t1.key = t2.key
+|""".stripMargin
+val (origin, adaptive) = runAdaptiveAndVerifyResult(query)
Review comment:
then we can remove the 2 lines below?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33243: [SPARK-36016][SQL] Support TimestampNTZType in expression ApproxCountDistinctForIntervals
SparkQA commented on pull request #33243: URL: https://github.com/apache/spark/pull/33243#issuecomment-875262161 **[Test build #140739 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140739/testReport)** for PR 33243 at commit [`2922d6e`](https://github.com/apache/spark/commit/2922d6ed23869f78021f0ecb34ffdbf7331e993f). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #33238: [SPARK-36020][SQL][3.1] Check logical link in remove redundant projects
ulysses-you commented on a change in pull request #33238:
URL: https://github.com/apache/spark/pull/33238#discussion_r665026154
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##
@@ -1481,4 +1481,26 @@ class AdaptiveQueryExecSuite
}
}
}
+
+ test("SPARK-36020: Check logical link in remove redundant projects") {
+withTempView("t") {
+ spark.range(10).selectExpr("id % 10 as key", "cast(id * 2 as int) as a",
+"cast(id * 3 as int) as b", "array(id, id + 1, id + 3) as
c").createOrReplaceTempView("t")
+ withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true") {
+val query =
+ """
+|WITH tt AS (
+| SELECT key, a, b, explode(c) AS c FROM t
+|)
+|SELECT t1.key, t1.c, t2.key, t2.c
+|FROM (SELECT a, b, c, key FROM tt WHERE a > 1) t1
+|JOIN (SELECT a, b, c, key FROM tt) t2
+| ON t1.key = t2.key
+|""".stripMargin
+val (origin, adaptive) = runAdaptiveAndVerifyResult(query)
Review comment:
yea, I think so
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer opened a new pull request #33243: [SPARK-36016][SQL] Support TimestampNTZType in expression ApproxCountDistinctForIntervals
beliefer opened a new pull request #33243: URL: https://github.com/apache/spark/pull/33243 ### What changes were proposed in this pull request? The current `ApproxCountDistinctForInterval`s supports `TimestampType`, but not supports timestamp without time zone yet. This PR will add the function. ### Why are the changes needed? `ApproxCountDistinctForInterval` need supports `TimestampNTZType`. ### Does this PR introduce _any_ user-facing change? 'Yes'. `ApproxCountDistinctForInterval` accepts `TimestampNTZType`. ### How was this patch tested? New tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33231: [SPARK-36021][SQL] Parse interval literals should support more than 2 digits
SparkQA removed a comment on pull request #33231: URL: https://github.com/apache/spark/pull/33231#issuecomment-875257077 **[Test build #140735 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140735/testReport)** for PR 33231 at commit [`8db45ec`](https://github.com/apache/spark/commit/8db45eccfe68b2bf7ec11611b2b3a825582f6949). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33231: [SPARK-36021][SQL] Parse interval literals should support more than 2 digits
AmplabJenkins removed a comment on pull request #33231: URL: https://github.com/apache/spark/pull/33231#issuecomment-875259534 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140735/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33231: [SPARK-36021][SQL] Parse interval literals should support more than 2 digits
AmplabJenkins commented on pull request #33231: URL: https://github.com/apache/spark/pull/33231#issuecomment-875259534 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140735/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33231: [SPARK-36021][SQL] Parse interval literals should support more than 2 digits
SparkQA commented on pull request #33231: URL: https://github.com/apache/spark/pull/33231#issuecomment-875259506 **[Test build #140735 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140735/testReport)** for PR 33231 at commit [`8db45ec`](https://github.com/apache/spark/commit/8db45eccfe68b2bf7ec11611b2b3a825582f6949). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SaurabhChawla100 commented on pull request #33232: [SPARK-36027][SQL] Add the code change to pushdown filter in case of typedFilter
SaurabhChawla100 commented on pull request #33232: URL: https://github.com/apache/spark/pull/33232#issuecomment-875259211 cc @cloud-fan @HyukjinKwon @viirya -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33238: [SPARK-36020][SQL][3.1] Check logical link in remove redundant projects
cloud-fan commented on a change in pull request #33238:
URL: https://github.com/apache/spark/pull/33238#discussion_r665024445
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##
@@ -1481,4 +1481,26 @@ class AdaptiveQueryExecSuite
}
}
}
+
+ test("SPARK-36020: Check logical link in remove redundant projects") {
+withTempView("t") {
+ spark.range(10).selectExpr("id % 10 as key", "cast(id * 2 as int) as a",
+"cast(id * 3 as int) as b", "array(id, id + 1, id + 3) as
c").createOrReplaceTempView("t")
+ withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true") {
+val query =
+ """
+|WITH tt AS (
+| SELECT key, a, b, explode(c) AS c FROM t
+|)
+|SELECT t1.key, t1.c, t2.key, t2.c
+|FROM (SELECT a, b, c, key FROM tt WHERE a > 1) t1
+|JOIN (SELECT a, b, c, key FROM tt) t2
+| ON t1.key = t2.key
+|""".stripMargin
+val (origin, adaptive) = runAdaptiveAndVerifyResult(query)
Review comment:
so in 3.1, we just need to make sure this test can run?
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##
@@ -1481,4 +1481,26 @@ class AdaptiveQueryExecSuite
}
}
}
+
+ test("SPARK-36020: Check logical link in remove redundant projects") {
+withTempView("t") {
+ spark.range(10).selectExpr("id % 10 as key", "cast(id * 2 as int) as a",
+"cast(id * 3 as int) as b", "array(id, id + 1, id + 3) as
c").createOrReplaceTempView("t")
+ withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true") {
+val query =
+ """
+|WITH tt AS (
+| SELECT key, a, b, explode(c) AS c FROM t
+|)
+|SELECT t1.key, t1.c, t2.key, t2.c
+|FROM (SELECT a, b, c, key FROM tt WHERE a > 1) t1
+|JOIN (SELECT a, b, c, key FROM tt) t2
+| ON t1.key = t2.key
+|""".stripMargin
+val (origin, adaptive) = runAdaptiveAndVerifyResult(query)
Review comment:
so in 3.1, we just need to make sure this query can run?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] akshatb1 edited a comment on pull request #33135: [SPARK-35931][CORE][YARN] Ability to override Yarn Cluster Submit Class with Configuration
akshatb1 edited a comment on pull request #33135: URL: https://github.com/apache/spark/pull/33135#issuecomment-875250761 Thanks @tgravescs for looking into this. Please find my responses below. > Can you give more details about exactly what requires changing here? We have a custom implementation of setting up tokens for Yarn applications. This involves invoking some native libraries to generate different kinds of tokens and these needs to be set in AM Container launch context. Similarly, there are some changes required in Yarn configuration for running on multi-tenant Federated Yarn cluster. For this purpose, we are adding a new client similar to yarn/Client.scala along with a bunch of helper utility classes related to token handling. > If its a different yarn Client, if its way different it should just be a separate resource manager perhaps. If its something smaller (which it doesn't sound like from the description) perhaps something else needs to be more flexible. We are adding a new folder under /resource-managers/yarn such as /resource-managers/yarn/customyarn. This way it can leverage all existing internal classes and we can add custom implementation for Client in this new folder along with other utilities. > Another question - the ApplicationMaster doesn't need modified as well? ApplicationMaster does not change since only changes required are in how application submission is done. > These are all internal Spark classes as well, so making a config for it without some public or developer interface doesn't make a lot of sense. Will it be a good idea to make the private methods/fields in Client.scala to make protected to allow developers to extend the Client? Currently, it can not be extended since many methods are private. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33238: [SPARK-36020][SQL][3.1] Check logical link in remove redundant projects
SparkQA commented on pull request #33238: URL: https://github.com/apache/spark/pull/33238#issuecomment-875258760 **[Test build #140738 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140738/testReport)** for PR 33238 at commit [`d6faccb`](https://github.com/apache/spark/commit/d6faccbf9ba892d45d537df86f8a199d7e4bbc8d). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range
SparkQA commented on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-875257284 **[Test build #140737 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140737/testReport)** for PR 32959 at commit [`9a07c14`](https://github.com/apache/spark/commit/9a07c14b692054655349c2463a24af199e93fdc3). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions
SparkQA commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-875257101 **[Test build #140736 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140736/testReport)** for PR 33223 at commit [`df5e08b`](https://github.com/apache/spark/commit/df5e08bde6ee967235605de9f4c5971dcda04f27). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33231: [SPARK-36021][SQL] Parse interval literals should support more than 2 digits
SparkQA commented on pull request #33231: URL: https://github.com/apache/spark/pull/33231#issuecomment-875257077 **[Test build #140735 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140735/testReport)** for PR 33231 at commit [`8db45ec`](https://github.com/apache/spark/commit/8db45eccfe68b2bf7ec11611b2b3a825582f6949). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33241: [SPARK-36017][SQL] Support TimestampNTZType in expression ApproximatePercentile
SparkQA commented on pull request #33241: URL: https://github.com/apache/spark/pull/33241#issuecomment-875257043 **[Test build #140734 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140734/testReport)** for PR 33241 at commit [`809b412`](https://github.com/apache/spark/commit/809b41283e765b0aa02e5ee00930ca3d8c3994e4). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range
SparkQA commented on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-875256952 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45241/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33242: [SPARK-35735][SQL][FOLLOWUP] Fix case minute to second regex can cover by hour to minute
SparkQA commented on pull request #33242: URL: https://github.com/apache/spark/pull/33242#issuecomment-875257016 **[Test build #140733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140733/testReport)** for PR 33242 at commit [`5982d35`](https://github.com/apache/spark/commit/5982d35d362b5e87f4afaf9c873f47a0639f22c9). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33233: [SPARK-36026][BUILD][K8S] Upgrade kubernetes-client to 5.5.0
HyukjinKwon closed pull request #33233: URL: https://github.com/apache/spark/pull/33233 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33233: [SPARK-36026][BUILD][K8S] Upgrade kubernetes-client to 5.5.0
HyukjinKwon commented on pull request #33233: URL: https://github.com/apache/spark/pull/33233#issuecomment-875256559 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations
AmplabJenkins removed a comment on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875255715 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33231: [SPARK-36021][SQL] Parse interval literals should support more than 2 digits
AmplabJenkins removed a comment on pull request #33231: URL: https://github.com/apache/spark/pull/33231#issuecomment-875255717 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45242/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33238: [SPARK-36020][SQL][3.1] Check logical link in remove redundant projects
AmplabJenkins removed a comment on pull request #33238: URL: https://github.com/apache/spark/pull/33238#issuecomment-875255719 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45236/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33239: [WIP][SPARK-36030][SQL] Support DS v2 metrics at writing path
AmplabJenkins removed a comment on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-875255716 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140727/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33242: [SPARK-35735][SQL][FOLLOWUP] Fix case minute to second regex can cover by hour to minute
AngersZh commented on a change in pull request #33242:
URL: https://github.com/apache/spark/pull/33242#discussion_r665022549
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##
@@ -327,8 +327,8 @@ object IntervalUtils {
toDTInterval(day, hour, minute, secondAndMicro(second, micro),
finalSign(firstSign, secondSign))
- case hourMinuteRegex(sign, hour, minute) =>
-checkDTIntervalStringDataType(DT(DT.HOUR, DT.MINUTE))
+ case hourMinuteRegex(sign, hour, minute)
+if startField == DT.HOUR && endField == DT.MINUTE =>
Review comment:
How about to make all place like this? FYI @cloud-fan @MaxGekk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33238: [SPARK-36020][SQL][3.1] Check logical link in remove redundant projects
AmplabJenkins commented on pull request #33238: URL: https://github.com/apache/spark/pull/33238#issuecomment-875255719 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45236/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33231: [SPARK-36021][SQL] Parse interval literals should support more than 2 digits
AmplabJenkins commented on pull request #33231: URL: https://github.com/apache/spark/pull/33231#issuecomment-875255717 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45242/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33239: [WIP][SPARK-36030][SQL] Support DS v2 metrics at writing path
AmplabJenkins commented on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-875255716 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140727/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33237: [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations
AmplabJenkins commented on pull request #33237: URL: https://github.com/apache/spark/pull/33237#issuecomment-875255718 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu opened a new pull request #33242: [SPARK-35735][SQL][FOLLOWUP] Fix case minute to second regex can cover by hour to minute
AngersZh opened a new pull request #33242: URL: https://github.com/apache/spark/pull/33242 ### What changes were proposed in this pull request? When cast `10:10` to interval minute to second, it can be catch by hour to minute regex, here to fix this. ### Why are the changes needed? Fix bug ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Added UT -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33164: [SPARK-35958][CORE] Refactor SparkError.scala to SparkThrowable.java
HyukjinKwon commented on a change in pull request #33164:
URL: https://github.com/apache/spark/pull/33164#discussion_r665021030
##
File path: core/src/main/java/org/apache/spark/SparkThrowable.java
##
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark;
+
+/**
+ * Interface mixed into Throwables thrown from Spark.
+ *
+ * - For backwards compatibility, existing throwable types can be thrown with
an arbitrary error
+ * message with no error class. See [[SparkException]].
+ * - To promote standardization, throwables should be thrown with an error
class and message
+ * parameters to construct an error message with
SparkThrowableHelper.getMessage(). New throwable
+ * types should not accept arbitrary error messages. See
[[SparkArithmeticException]].
+ */
+public interface SparkThrowable {
+// Succinct, human-readable, unique, and consistent representation of the
error category
Review comment:
Yeah, I think you should wrap or manually convert that to `Option` ..
but I guess that's fine though? are there many changes required for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33240: [SPARK-35906][SQL][FOLLOWUP] Recursive remove sort if the maximum number of rows less than or equal to 1
SparkQA commented on pull request #33240: URL: https://github.com/apache/spark/pull/33240#issuecomment-875253865 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45240/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33231: [SPARK-36021][SQL] Parse interval literals should support more than 2 digits
SparkQA commented on pull request #33231: URL: https://github.com/apache/spark/pull/33231#issuecomment-875253570 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45242/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33164: [SPARK-35958][CORE] Refactor SparkError.scala to SparkThrowable.java
SparkQA commented on pull request #33164: URL: https://github.com/apache/spark/pull/33164#issuecomment-875252182 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45239/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer opened a new pull request #33241: [SPARK-36017][SQL] Support TimestampNTZType in expression ApproximatePercentile
beliefer opened a new pull request #33241: URL: https://github.com/apache/spark/pull/33241 ### What changes were proposed in this pull request? The current `ApproximatePercentile` supports `TimestampType`, but not supports timestamp without time zone yet. This PR will add the function. ### Why are the changes needed? `ApproximatePercentile` need supports `TimestampType`. ### Does this PR introduce _any_ user-facing change? 'Yes'. `ApproximatePercentile` accepts `TimestampType`. ### How was this patch tested? New tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33239: [WIP][SPARK-36030][SQL] Support DS v2 metrics at writing path
SparkQA commented on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-875251767 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45238/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33164: [SPARK-35958][CORE] Refactor SparkError.scala to SparkThrowable.java
HyukjinKwon commented on a change in pull request #33164:
URL: https://github.com/apache/spark/pull/33164#discussion_r665018229
##
File path: core/src/main/java/org/apache/spark/SparkThrowable.java
##
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark;
+
+import org.apache.spark.annotation.Experimental;
+
+/**
+ * :: Experimental ::
+ *
+ * Interface mixed into Throwables thrown from Spark.
+ *
+ * - For backwards compatibility, existing Throwable types can be thrown with
an arbitrary error
+ * message with a null error class. See [[SparkException]].
+ * - To promote standardization, Throwables should be thrown with an error
class and message
+ * parameters to construct an error message with
SparkThrowableHelper.getMessage(). New Throwable
+ * types should not accept arbitrary error messages. See
[[SparkArithmeticException]].
+ */
+@Experimental
+public interface SparkThrowable {
+// Succinct, human-readable, unique, and consistent representation of the
error category
Review comment:
I know many of other java files are not but strictly it should be
2-spaced
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33164: [SPARK-35958][CORE] Refactor SparkError.scala to SparkThrowable.java
HyukjinKwon commented on a change in pull request #33164:
URL: https://github.com/apache/spark/pull/33164#discussion_r665018109
##
File path: core/src/main/java/org/apache/spark/SparkThrowable.java
##
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark;
+
+import org.apache.spark.annotation.Experimental;
+
+/**
+ * :: Experimental ::
+ *
+ * Interface mixed into Throwables thrown from Spark.
+ *
+ * - For backwards compatibility, existing Throwable types can be thrown with
an arbitrary error
+ * message with a null error class. See [[SparkException]].
+ * - To promote standardization, Throwables should be thrown with an error
class and message
+ * parameters to construct an error message with
SparkThrowableHelper.getMessage(). New Throwable
+ * types should not accept arbitrary error messages. See
[[SparkArithmeticException]].
+ */
+@Experimental
+public interface SparkThrowable {
+// Succinct, human-readable, unique, and consistent representation of the
error category
Review comment:
Let's keep them 2-spaced
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] akshatb1 commented on pull request #33135: [SPARK-35931][CORE][YARN] Ability to override Yarn Cluster Submit Class with Configuration
akshatb1 commented on pull request #33135: URL: https://github.com/apache/spark/pull/33135#issuecomment-875250761 Thanks @tgravescs for looking into this. Please find my responses below. > Can you give more details about exactly what requires changing here? We have a custom implementation of setting up tokens for Yarn applications. This involves invoking some native libraries to generate different kinds of tokens and these needs to be set in AM Container launch context. Similarly, there are some changes required in Yarn configuration for running on multi-tenant Federated Yarn cluster. For this purpose, we are adding a new client similar to yarn/Client.scala along with a bunch of helper utility classes related to token handling. > If its a different yarn Client, if its way different it should just be a separate resource manager perhaps. If its something smaller (which it doesn't sound like from the description) perhaps something else needs to be more flexible. We are adding a new folder under /resource-managers/yarn such as /resource-managers/yarn/customyarn. This way it can leverage all existing internal classes and we can add custom implementation for Client in this new folder along with other utilities. > Another question - the ApplicationMaster doesn't need modified as well? ApplicationMaster does not change since only changes required are in how application submission is done. > These are all internal Spark classes as well, so making a config for it without some public or developer interface doesn't make a lot of sense. Will it be a good idea to make the fields in Client.scala to make protected to allow developers to extend the Client? Currently, it can not be extended since many methods are private. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33164: [SPARK-35958][CORE] Refactor SparkError.scala to SparkThrowable.java
HyukjinKwon commented on a change in pull request #33164: URL: https://github.com/apache/spark/pull/33164#discussion_r665018061 ## File path: core/src/main/java/org/apache/spark/SparkThrowable.java ## @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark; + +import org.apache.spark.annotation.Experimental; + +/** + * :: Experimental :: + * + * Interface mixed into Throwables thrown from Spark. + * + * - For backwards compatibility, existing Throwable types can be thrown with an arbitrary error + * message with a null error class. See [[SparkException]]. + * - To promote standardization, Throwables should be thrown with an error class and message + * parameters to construct an error message with SparkThrowableHelper.getMessage(). New Throwable + * types should not accept arbitrary error messages. See [[SparkArithmeticException]]. Review comment: Shall we add `@since`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] allisonwang-db commented on pull request #33235: [SPARK-36028][SQL] Allow Project to host outer references in scalar subqueries
allisonwang-db commented on pull request #33235: URL: https://github.com/apache/spark/pull/33235#issuecomment-875250639 cc @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ekoifman edited a comment on pull request #32872: [SPARK-35639][SQL] Make hasCoalescedPartition return true if something was actually coalesced
ekoifman edited a comment on pull request #32872: URL: https://github.com/apache/spark/pull/32872#issuecomment-875248394 @ulysses-you I provided a link https://github.com/apache/spark/pull/32872#discussion_r649665188 to an explanation. I haven't seen any response to it - is the explanation adequate? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33239: [WIP][SPARK-36030][SQL] Support DS v2 metrics at writing path
SparkQA commented on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-875248525 **[Test build #140727 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140727/testReport)** for PR 33239 at commit [`245db51`](https://github.com/apache/spark/commit/245db5148f93852c41311c5aad6b1a216b874c33). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class ContinuousWriteRDD(var prev: RDD[InternalRow], writerFactory: StreamingDataWriterFactory,` * `case class WriteToContinuousDataSource(write: StreamingWrite, query: LogicalPlan,` * `case class WriteToContinuousDataSourceExec(write: StreamingWrite, query: SparkPlan,` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33239: [WIP][SPARK-36030][SQL] Support DS v2 metrics at writing path
SparkQA removed a comment on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-875234084 **[Test build #140727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140727/testReport)** for PR 33239 at commit [`245db51`](https://github.com/apache/spark/commit/245db5148f93852c41311c5aad6b1a216b874c33). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
