[GitHub] [spark] ueshin opened a new pull request #34685: [SPARK-37443][PYTHON] Provide a profiler for Python/Pandas UDFs

ueshin opened a new pull request #34685: URL: https://github.com/apache/spark/pull/34685 ### What changes were proposed in this pull request? ### Why are the changes needed? Currently a profiler is provided for only `RDD` operations, but providing a profiler for

[GitHub] [spark] SparkQA removed a comment on pull request #34611: [SPARK-35867][SQL] Enable vectorized read for VectorizedPlainValuesReader.readBooleans

SparkQA removed a comment on pull request #34611: URL: https://github.com/apache/spark/pull/34611#issuecomment-975825876 **[Test build #145517 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145517/testReport)** for PR 34611 at commit

[GitHub] [spark] SparkQA commented on pull request #34611: [SPARK-35867][SQL] Enable vectorized read for VectorizedPlainValuesReader.readBooleans

SparkQA commented on pull request #34611: URL: https://github.com/apache/spark/pull/34611#issuecomment-976006138 **[Test build #145517 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145517/testReport)** for PR 34611 at commit

[GitHub] [spark] SparkQA commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

SparkQA commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-975988587 **[Test build #145521 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145521/testReport)** for PR 34596 at commit

[GitHub] [spark] sathiyapk commented on a change in pull request #34593: [SPARK-37324][SQL] Adds support for decimal rounding mode up, down, half_down

sathiyapk commented on a change in pull request #34593: URL: https://github.com/apache/spark/pull/34593#discussion_r754677290 ## File path: sql/core/src/test/scala/org/apache/spark/sql/MathFunctionsSuite.scala ## @@ -319,6 +370,18 @@ class MathFunctionsSuite extends QueryTest

[GitHub] [spark] sathiyapk commented on a change in pull request #34593: [SPARK-37324][SQL] Adds support for decimal rounding mode up, down, half_down

sathiyapk commented on a change in pull request #34593: URL: https://github.com/apache/spark/pull/34593#discussion_r754675110 ## File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ## @@ -2197,13 +2197,23 @@ object functions { def round(e: Column):

[GitHub] [spark] sadikovi commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

sadikovi commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-975965380 jenkins retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] sathiyapk commented on a change in pull request #34593: [SPARK-37324][SQL] Adds support for decimal rounding mode up, down, half_down

sathiyapk commented on a change in pull request #34593: URL: https://github.com/apache/spark/pull/34593#discussion_r754671246 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala ## @@ -1500,26 +1514,37 @@ abstract class

[GitHub] [spark] sathiyapk commented on a change in pull request #34593: [SPARK-37324][SQL] Adds support for decimal rounding mode up, down, half_down

sathiyapk commented on a change in pull request #34593: URL: https://github.com/apache/spark/pull/34593#discussion_r754670983 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala ## @@ -413,6 +413,18 @@ final class Decimal extends Ordered[Decimal]

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

AmplabJenkins removed a comment on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975950959 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49991/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34636: [WIP][SPARK-37359][K8S] Cleanup the Spark Kubernetes Integration tests

AmplabJenkins removed a comment on pull request #34636: URL: https://github.com/apache/spark/pull/34636#issuecomment-975950961 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49992/

[GitHub] [spark] AmplabJenkins commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

AmplabJenkins commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975950959 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49991/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34636: [WIP][SPARK-37359][K8S] Cleanup the Spark Kubernetes Integration tests

AmplabJenkins commented on pull request #34636: URL: https://github.com/apache/spark/pull/34636#issuecomment-975950961 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49992/ --

[GitHub] [spark] zero323 commented on a change in pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

zero323 commented on a change in pull request #34671: URL: https://github.com/apache/spark/pull/34671#discussion_r754647909 ## File path: python/pyspark/ml/common.py ## @@ -15,11 +15,15 @@ # limitations under the License. # +from typing import Any, Callable +from

[GitHub] [spark] SparkQA commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

SparkQA commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975932293 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49991/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34636: [WIP][SPARK-37359][K8S] Cleanup the Spark Kubernetes Integration tests

SparkQA commented on pull request #34636: URL: https://github.com/apache/spark/pull/34636#issuecomment-975931222 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49992/ -- This is an automated message from the

[GitHub] [spark] nchammas commented on a change in pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

nchammas commented on a change in pull request #34671: URL: https://github.com/apache/spark/pull/34671#discussion_r754642606 ## File path: python/pyspark/ml/common.py ## @@ -15,11 +15,15 @@ # limitations under the License. # +from typing import Any, Callable +from

[GitHub] [spark] zero323 commented on a change in pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

zero323 commented on a change in pull request #34671: URL: https://github.com/apache/spark/pull/34671#discussion_r754637469 ## File path: python/pyspark/ml/common.py ## @@ -15,11 +15,15 @@ # limitations under the License. # +from typing import Any, Callable +from

[GitHub] [spark] zero323 commented on a change in pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

zero323 commented on a change in pull request #34671: URL: https://github.com/apache/spark/pull/34671#discussion_r754637469 ## File path: python/pyspark/ml/common.py ## @@ -15,11 +15,15 @@ # limitations under the License. # +from typing import Any, Callable +from

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34611: [SPARK-35867][SQL] Enable vectorized read for VectorizedPlainValuesReader.readBooleans

AmplabJenkins removed a comment on pull request #34611: URL: https://github.com/apache/spark/pull/34611#issuecomment-975908416 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49989/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments

AmplabJenkins removed a comment on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-975908414 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145515/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

AmplabJenkins removed a comment on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975908413 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins commented on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments

AmplabJenkins commented on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-975908414 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145515/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34611: [SPARK-35867][SQL] Enable vectorized read for VectorizedPlainValuesReader.readBooleans

AmplabJenkins commented on pull request #34611: URL: https://github.com/apache/spark/pull/34611#issuecomment-975908416 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49989/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

AmplabJenkins commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975908413 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] SparkQA commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

SparkQA commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975902438 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49991/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34636: [WIP][SPARK-37359][K8S] Cleanup the Spark Kubernetes Integration tests

SparkQA commented on pull request #34636: URL: https://github.com/apache/spark/pull/34636#issuecomment-975900558 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49992/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA removed a comment on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments

SparkQA removed a comment on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-975667590 **[Test build #145515 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145515/testReport)** for PR 34668 at commit

[GitHub] [spark] SparkQA commented on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments

SparkQA commented on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-975894629 **[Test build #145515 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145515/testReport)** for PR 34668 at commit

[GitHub] [spark] zero323 commented on pull request #34655: [SPARK-37380][PYTHON] Miscellaneous Python lint infra cleanup

zero323 commented on pull request #34655: URL: https://github.com/apache/spark/pull/34655#issuecomment-975894324 > @zero323 - Is the cleanup of pycodestyle configs along the lines of what you were expecting? Looks sensible, but I wouldn't mind more eyes on this. -- This is an

[GitHub] [spark] SparkQA commented on pull request #34611: [SPARK-35867][SQL] Enable vectorized read for VectorizedPlainValuesReader.readBooleans

SparkQA commented on pull request #34611: URL: https://github.com/apache/spark/pull/34611#issuecomment-975893441 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49989/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

SparkQA commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975893327 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49990/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

SparkQA commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975886499 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49988/ -- This is an automated message from the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34636: [WIP][SPARK-37359][K8S] Cleanup the Spark Kubernetes Integration tests

AmplabJenkins removed a comment on pull request #34636: URL: https://github.com/apache/spark/pull/34636#issuecomment-975877377 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145520/

[GitHub] [spark] SparkQA removed a comment on pull request #34636: [WIP][SPARK-37359][K8S] Cleanup the Spark Kubernetes Integration tests

SparkQA removed a comment on pull request #34636: URL: https://github.com/apache/spark/pull/34636#issuecomment-975866493 **[Test build #145520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145520/testReport)** for PR 34636 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #34636: [WIP][SPARK-37359][K8S] Cleanup the Spark Kubernetes Integration tests

AmplabJenkins commented on pull request #34636: URL: https://github.com/apache/spark/pull/34636#issuecomment-975877377 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145520/ -- This

[GitHub] [spark] SparkQA commented on pull request #34636: [WIP][SPARK-37359][K8S] Cleanup the Spark Kubernetes Integration tests

SparkQA commented on pull request #34636: URL: https://github.com/apache/spark/pull/34636#issuecomment-975877062 **[Test build #145520 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145520/testReport)** for PR 34636 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

AmplabJenkins removed a comment on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975875789 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145519/

[GitHub] [spark] SparkQA removed a comment on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

SparkQA removed a comment on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975866358 **[Test build #145519 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145519/testReport)** for PR 34671 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

AmplabJenkins commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975875789 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145519/ -- This

[GitHub] [spark] SparkQA commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

SparkQA commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975875557 **[Test build #145519 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145519/testReport)** for PR 34671 at commit

[GitHub] [spark] SparkQA commented on pull request #34636: [WIP][SPARK-37359][K8S] Cleanup the Spark Kubernetes Integration tests

SparkQA commented on pull request #34636: URL: https://github.com/apache/spark/pull/34636#issuecomment-975866493 **[Test build #145520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145520/testReport)** for PR 34636 at commit

[GitHub] [spark] SparkQA commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

SparkQA commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975866358 **[Test build #145519 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145519/testReport)** for PR 34671 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

AmplabJenkins removed a comment on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975865837 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145518/

[GitHub] [spark] AmplabJenkins commented on pull request #34684: [SPARK-37442] - InMemoryRelation statistics bug causing broadcast join failures with AQE enabled

AmplabJenkins commented on pull request #34684: URL: https://github.com/apache/spark/pull/34684#issuecomment-975866093 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] AmplabJenkins commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

AmplabJenkins commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975865837 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145518/ -- This

[GitHub] [spark] SparkQA commented on pull request #34611: [SPARK-35867][SQL] Enable vectorized read for VectorizedPlainValuesReader.readBooleans

SparkQA commented on pull request #34611: URL: https://github.com/apache/spark/pull/34611#issuecomment-975862121 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49989/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

SparkQA commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975861755 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49990/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

SparkQA commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975857224 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49988/ -- This is an automated message from the Apache

[GitHub] [spark] ChenMichael commented on pull request #34684: [SPARK-37442] - InMemoryRelation statistics bug causing broadcast join failures with AQE enabled

ChenMichael commented on pull request #34684: URL: https://github.com/apache/spark/pull/34684#issuecomment-975851968 @cloud-fan can you take a look when you have some time? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] nchammas commented on pull request #34655: [SPARK-37380][PYTHON] Miscellaneous Python lint infra cleanup

nchammas commented on pull request #34655: URL: https://github.com/apache/spark/pull/34655#issuecomment-975848602 @zero323 - Is the cleanup of pycodestyle configs along the lines of what you were expecting? -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] ChenMichael edited a comment on pull request #34684: [SPARK-37442] - Bug when AQE is enabled where replanning tries to rea…

ChenMichael edited a comment on pull request #34684: URL: https://github.com/apache/spark/pull/34684#issuecomment-975845896 I'm not sure if this is the best way to solve this bug so I will detail the other solution I could come up with and then detail the possible problems I could see.

[GitHub] [spark] ChenMichael commented on pull request #34684: [SPARK-37442] - Bug when AQE is enabled where replanning tries to rea…

ChenMichael commented on pull request #34684: URL: https://github.com/apache/spark/pull/34684#issuecomment-975845896 I'm not sure if this is the best way to solve this bug so I will detail the other solution I could come up with and then compare the possible problems with them. 1.

[GitHub] [spark] nchammas commented on a change in pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

nchammas commented on a change in pull request #34671: URL: https://github.com/apache/spark/pull/34671#discussion_r754570168 ## File path: python/pyspark/mllib/common.py ## @@ -113,16 +117,16 @@ def _java2py(sc, r, encoding="bytes"): return r -def callJavaFunc(sc,

[GitHub] [spark] nchammas commented on a change in pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

nchammas commented on a change in pull request #34671: URL: https://github.com/apache/spark/pull/34671#discussion_r754568896 ## File path: python/pyspark/mllib/common.py ## @@ -113,16 +117,16 @@ def _java2py(sc, r, encoding="bytes"): return r -def callJavaFunc(sc,

[GitHub] [spark] SparkQA removed a comment on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

SparkQA removed a comment on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975828766 **[Test build #145518 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145518/testReport)** for PR 34671 at commit

[GitHub] [spark] SparkQA commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

SparkQA commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975839386 **[Test build #145518 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145518/testReport)** for PR 34671 at commit

[GitHub] [spark] zero323 commented on a change in pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

zero323 commented on a change in pull request #34671: URL: https://github.com/apache/spark/pull/34671#discussion_r754565478 ## File path: python/pyspark/ml/_typing.pyi ## @@ -64,7 +65,7 @@ MultilabelClassificationEvaluatorMetricType = Union[ Literal["microRecall"],

[GitHub] [spark] ChenMichael opened a new pull request #34684: [SPARK-37442] - Bug when AQE is enabled where replanning tries to rea…

ChenMichael opened a new pull request #34684: URL: https://github.com/apache/spark/pull/34684 ### What changes were proposed in this pull request? Immediately materialize underlying rdd cache (using .count) for an InMemoryRelation when `buildBuffers` is called.

[GitHub] [spark] zero323 commented on a change in pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

zero323 commented on a change in pull request #34671: URL: https://github.com/apache/spark/pull/34671#discussion_r754564756 ## File path: python/pyspark/mllib/_typing.pyi ## @@ -16,8 +16,11 @@ # specific language governing permissions and limitations # under the License.

[GitHub] [spark] zero323 commented on a change in pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

zero323 commented on a change in pull request #34671: URL: https://github.com/apache/spark/pull/34671#discussion_r754563130 ## File path: python/pyspark/mllib/common.py ## @@ -113,16 +117,16 @@ def _java2py(sc, r, encoding="bytes"): return r -def callJavaFunc(sc,

[GitHub] [spark] zero323 commented on a change in pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

zero323 commented on a change in pull request #34671: URL: https://github.com/apache/spark/pull/34671#discussion_r754563130 ## File path: python/pyspark/mllib/common.py ## @@ -113,16 +117,16 @@ def _java2py(sc, r, encoding="bytes"): return r -def callJavaFunc(sc,

[GitHub] [spark] nchammas commented on a change in pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

nchammas commented on a change in pull request #34671: URL: https://github.com/apache/spark/pull/34671#discussion_r754560349 ## File path: python/pyspark/mllib/_typing.pyi ## @@ -16,8 +16,11 @@ # specific language governing permissions and limitations # under the License.

[GitHub] [spark] zero323 commented on a change in pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

zero323 commented on a change in pull request #34671: URL: https://github.com/apache/spark/pull/34671#discussion_r754559725 ## File path: python/pyspark/mllib/_typing.pyi ## @@ -16,8 +16,11 @@ # specific language governing permissions and limitations # under the License.

[GitHub] [spark] nchammas commented on a change in pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

nchammas commented on a change in pull request #34671: URL: https://github.com/apache/spark/pull/34671#discussion_r754558752 ## File path: python/pyspark/ml/_typing.pyi ## @@ -64,7 +65,7 @@ MultilabelClassificationEvaluatorMetricType = Union[ Literal["microRecall"],

[GitHub] [spark] SparkQA commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

SparkQA commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975828766 **[Test build #145518 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145518/testReport)** for PR 34671 at commit

[GitHub] [spark] xinrong-databricks commented on a change in pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

xinrong-databricks commented on a change in pull request #34671: URL: https://github.com/apache/spark/pull/34671#discussion_r754556211 ## File path: python/pyspark/ml/common.py ## @@ -53,24 +57,24 @@ def _new_smart_decode(obj): # this will call the ML version of

[GitHub] [spark] SparkQA commented on pull request #34611: [SPARK-35867][SQL] Enable vectorized read for VectorizedPlainValuesReader.readBooleans

SparkQA commented on pull request #34611: URL: https://github.com/apache/spark/pull/34611#issuecomment-975825876 **[Test build #145517 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145517/testReport)** for PR 34611 at commit

[GitHub] [spark] zero323 commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

zero323 commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975825755 > The number of `ignore[attr-defined]` hints required seems a little wrong. But I suppose addressing that would require changes to `SparkContext`, which is out of scope for

[GitHub] [spark] xinrong-databricks commented on a change in pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

xinrong-databricks commented on a change in pull request #34671: URL: https://github.com/apache/spark/pull/34671#discussion_r754555187 ## File path: python/pyspark/ml/common.py ## @@ -53,24 +57,24 @@ def _new_smart_decode(obj): # this will call the ML version of

[GitHub] [spark] xinrong-databricks commented on a change in pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

xinrong-databricks commented on a change in pull request #34671: URL: https://github.com/apache/spark/pull/34671#discussion_r754554096 ## File path: python/pyspark/ml/common.py ## @@ -53,24 +57,24 @@ def _new_smart_decode(obj): # this will call the ML version of

[GitHub] [spark] SparkQA removed a comment on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

SparkQA removed a comment on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975823020 **[Test build #145516 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145516/testReport)** for PR 34671 at commit

[GitHub] [spark] SparkQA commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

SparkQA commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975823967 **[Test build #145516 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145516/testReport)** for PR 34671 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

AmplabJenkins commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975823990 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145516/ -- This

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

AmplabJenkins removed a comment on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975823990 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145516/

[GitHub] [spark] kazuyukitanimura commented on a change in pull request #34611: [SPARK-35867][SQL] Enable vectorized read for VectorizedPlainValuesReader.readBooleans

kazuyukitanimura commented on a change in pull request #34611: URL: https://github.com/apache/spark/pull/34611#discussion_r754552871 ## File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedPlainValuesReader.java ## @@ -53,19 +53,47 @@

[GitHub] [spark] SparkQA commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

SparkQA commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975823020 **[Test build #145516 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145516/testReport)** for PR 34671 at commit

[GitHub] [spark] kazuyukitanimura commented on a change in pull request #34611: [SPARK-35867][SQL] Enable vectorized read for VectorizedPlainValuesReader.readBooleans

kazuyukitanimura commented on a change in pull request #34611: URL: https://github.com/apache/spark/pull/34611#discussion_r754552760 ## File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedPlainValuesReader.java ## @@ -53,19 +53,47 @@

[GitHub] [spark] kazuyukitanimura commented on a change in pull request #34611: [SPARK-35867][SQL] Enable vectorized read for VectorizedPlainValuesReader.readBooleans

kazuyukitanimura commented on a change in pull request #34611: URL: https://github.com/apache/spark/pull/34611#discussion_r754552682 ## File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedPlainValuesReader.java ## @@ -53,19 +53,47 @@

[GitHub] [spark] zero323 commented on a change in pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

zero323 commented on a change in pull request #34671: URL: https://github.com/apache/spark/pull/34671#discussion_r754552177 ## File path: python/pyspark/ml/_typing.pyi ## @@ -64,7 +65,7 @@ MultilabelClassificationEvaluatorMetricType = Union[ Literal["microRecall"],

[GitHub] [spark] zero323 commented on a change in pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

zero323 commented on a change in pull request #34671: URL: https://github.com/apache/spark/pull/34671#discussion_r754551374 ## File path: python/pyspark/ml/common.py ## @@ -53,24 +57,24 @@ def _new_smart_decode(obj): # this will call the ML version of pythonToJava() -def

[GitHub] [spark] zero323 edited a comment on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

zero323 edited a comment on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-974525917 > Just to be clear, are you saying I should split this PR into ml/common.py vs. mllib/common.py? > > And then have an umbrella ticket for adding type annotations to

[GitHub] [spark] nchammas commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

nchammas commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975814940 The number of `ignore[attr-defined]` hints required seems a little wrong. But I suppose addressing that would require changes to `SparkContext`, which is out of scope for this

[GitHub] [spark] yliou commented on pull request #34622: SPARK-37340 Display StageIds in Operators for SQL UI

yliou commented on pull request #34622: URL: https://github.com/apache/spark/pull/34622#issuecomment-975808167 cc @tgravescs would this feature be of interest? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] yliou commented on pull request #34637: Spark-37349 add SQL Rest API parsing logic

yliou commented on pull request #34637: URL: https://github.com/apache/spark/pull/34637#issuecomment-975806584 cc @gengliangwang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] nchammas commented on a change in pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

nchammas commented on a change in pull request #34671: URL: https://github.com/apache/spark/pull/34671#discussion_r754533879 ## File path: python/pyspark/ml/common.py ## @@ -53,24 +57,24 @@ def _new_smart_decode(obj): # this will call the ML version of pythonToJava() -def

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34681: [SPARK-37438][SQL] ANSI mode: Use store assignment rules for resolving function invocation

AmplabJenkins removed a comment on pull request #34681: URL: https://github.com/apache/spark/pull/34681#issuecomment-975786325 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145513/

[GitHub] [spark] SparkQA removed a comment on pull request #34681: [SPARK-37438][SQL] ANSI mode: Use store assignment rules for resolving function invocation

SparkQA removed a comment on pull request #34681: URL: https://github.com/apache/spark/pull/34681#issuecomment-975667360 **[Test build #145513 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145513/testReport)** for PR 34681 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #34681: [SPARK-37438][SQL] ANSI mode: Use store assignment rules for resolving function invocation

AmplabJenkins commented on pull request #34681: URL: https://github.com/apache/spark/pull/34681#issuecomment-975786325 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145513/ -- This

[GitHub] [spark] SparkQA commented on pull request #34681: [SPARK-37438][SQL] ANSI mode: Use store assignment rules for resolving function invocation

SparkQA commented on pull request #34681: URL: https://github.com/apache/spark/pull/34681#issuecomment-975786004 **[Test build #145513 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145513/testReport)** for PR 34681 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

AmplabJenkins removed a comment on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975775073 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49985/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments

AmplabJenkins removed a comment on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-975775068 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49986/

[GitHub] [spark] AmplabJenkins commented on pull request #34681: [SPARK-37438][SQL] ANSI mode: Use store assignment rules for resolving function invocation

AmplabJenkins commented on pull request #34681: URL: https://github.com/apache/spark/pull/34681#issuecomment-975775070 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49987/ --

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34681: [SPARK-37438][SQL] ANSI mode: Use store assignment rules for resolving function invocation

AmplabJenkins removed a comment on pull request #34681: URL: https://github.com/apache/spark/pull/34681#issuecomment-975775070 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49987/

[GitHub] [spark] AmplabJenkins commented on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments

AmplabJenkins commented on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-975775068 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49986/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

AmplabJenkins commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975775073 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49985/ --

[GitHub] [spark] SparkQA commented on pull request #34681: [SPARK-37438][SQL] ANSI mode: Use store assignment rules for resolving function invocation

SparkQA commented on pull request #34681: URL: https://github.com/apache/spark/pull/34681#issuecomment-975766915 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49987/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34671: [SPARK-37399][SPARK-37403][PySpark][ML] Merge {ml, mllib}/common.pyi into common.py

SparkQA commented on pull request #34671: URL: https://github.com/apache/spark/pull/34671#issuecomment-975760231 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49985/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34668: [SPARK-37389][SQL] Check unclosed bracketed comments

SparkQA commented on pull request #34668: URL: https://github.com/apache/spark/pull/34668#issuecomment-975753614 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49986/ -- This is an automated message from the

[GitHub] [spark] gaborgsomogyi commented on a change in pull request #29024: [SPARK-32001][SQL]Create JDBC authentication provider developer API

gaborgsomogyi commented on a change in pull request #29024: URL: https://github.com/apache/spark/pull/29024#discussion_r754487136 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/connection/ConnectionProvider.scala ## @@ -18,60 +18,45 @@

<    1   2   3   4   5   6   >