zhengruifeng commented on PR #42088:
URL: https://github.com/apache/spark/pull/42088#issuecomment-1643364098
late LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To un
HyukjinKwon commented on PR #42086:
URL: https://github.com/apache/spark/pull/42086#issuecomment-1643363137
I am fine with this as a workaround for now but such implementation
depending on tags is sort of flaky. The tags are easily lost when you, e.g.,
copy the expressions IIRC.
--
This
beliefer commented on PR #42084:
URL: https://github.com/apache/spark/pull/42084#issuecomment-1643357973
ping @cloud-fan cc @MaxGekk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
yihua commented on PR #40728:
URL: https://github.com/apache/spark/pull/40728#issuecomment-1643350395
Hi @vkorukanti, this is an important performance improvement for using the
row index from Parquet. Is the PR targeted for Spark 3.5?
--
This is an automated message from the Apache Git Se
HyukjinKwon closed pull request #42088: [SPARK-44491][INFRA] Add `branch-3.5`
to `publish_snapshot` GitHub Action job
URL: https://github.com/apache/spark/pull/42088
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
HyukjinKwon commented on PR #42088:
URL: https://github.com/apache/spark/pull/42088#issuecomment-1643340583
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
itholic commented on PR #42086:
URL: https://github.com/apache/spark/pull/42086#issuecomment-1643329346
I got it. Just created single ticket here: SPARK-44492 for addressing
undefined remaining tests so we don't miss it.
--
This is an automated message from the Apache Git Service.
To resp
liangyu-1 commented on code in PR #42058:
URL: https://github.com/apache/spark/pull/42058#discussion_r1267703981
##
streaming/src/main/scala/org/apache/spark/streaming/scheduler/ExecutorAllocationManager.scala:
##
@@ -102,6 +102,11 @@ private[streaming] class ExecutorAllocationM
zhengruifeng commented on PR #42086:
URL: https://github.com/apache/spark/pull/42086#issuecomment-1643320025
> > The good news is that 90% UTs can be resolved by this single one, and I
think we only need to touch 3~4 more rules.
>
> Great! Could you help creating tickets for remaining
surnaik commented on PR #41856:
URL: https://github.com/apache/spark/pull/41856#issuecomment-1643310115
Back from a break. I will use the official dbgen from TPCH website and
update the PR. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, ple
itholic closed pull request #42041: [DO-NOT-MERGE][PS][TESTS] Enable pandas API
on Spark tests related to SPARK-43611
URL: https://github.com/apache/spark/pull/42041
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
itholic commented on PR #42041:
URL: https://github.com/apache/spark/pull/42041#issuecomment-1643309098
Closing since now we have a fix for tests:
https://github.com/apache/spark/pull/42086
--
This is an automated message from the Apache Git Service.
To respond to the message, please log
ulysses-you commented on PR #40524:
URL: https://github.com/apache/spark/pull/40524#issuecomment-1643307262
After offline discussion, @pan3793 will try to take over this pr. Thank you
all!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log o
itholic commented on PR #42086:
URL: https://github.com/apache/spark/pull/42086#issuecomment-1643306997
> The good news is that 90% UTs can be resolved by this single one, and I
think we only need to touch 3~4 more rules.
Great! Could you help creating tickets for remaining 3~4 more r
yaooqinn commented on PR #40524:
URL: https://github.com/apache/spark/pull/40524#issuecomment-1643292939
cc @ulysses-you who is the last one to touch this part
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL ab
yaooqinn commented on PR #41951:
URL: https://github.com/apache/spark/pull/41951#issuecomment-1643284132
Thanks, merged to master and 3.5.
PS, K8s IT failures are irrelevant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to Git
yaooqinn closed pull request #41951: [SPARK-44367][SQL][UI] Show error message
on UI for each failed query
URL: https://github.com/apache/spark/pull/41951
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to g
LuciferYang commented on PR #42088:
URL: https://github.com/apache/spark/pull/42088#issuecomment-1643272217
cc @HyukjinKwon @zhengruifeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the spec
panbingkun opened a new pull request, #42088:
URL: https://github.com/apache/spark/pull/42088
### What changes were proposed in this pull request?
This PR aims to add `branch-3.5` to `publish_snapshot` GitHub Action job.
### Why are the changes needed?
Since GitHub Action Cron jo
mathewjacob1002 opened a new pull request, #42087:
URL: https://github.com/apache/spark/pull/42087
### What changes were proposed in this pull request?
Added examples to the docstring of using DeepspeedTorchDistributor
### Why are the changes needed?
More concrete examples, a
zhengruifeng closed pull request #42040: [WIP][SPARK-43611][SQL][PS][CONNCECT]
Fix unexpected `AnalysisException` from Spark Connect client
URL: https://github.com/apache/spark/pull/42040
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
zhengruifeng commented on PR #42040:
URL: https://github.com/apache/spark/pull/42040#issuecomment-1643204561
close this one in favor of https://github.com/apache/spark/pull/42086
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub
beliefer commented on PR #41932:
URL: https://github.com/apache/spark/pull/41932#issuecomment-1643189573
The CI failure is unrelated to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to th
zhengruifeng commented on PR #42086:
URL: https://github.com/apache/spark/pull/42086#issuecomment-1643187065
I have checked with @cloud-fan that we might have to modify the rules one by
one.
The good news is that 90% UTs can be resolved by this single one, and I
think we only need to tou
zhengruifeng opened a new pull request, #42086:
URL: https://github.com/apache/spark/pull/42086
### What changes were proposed in this pull request?
Make `ExtractWindowExpressions` retain the `PLAN_ID_TAG `
### Why are the changes needed?
In https://github.com/apache/spark/pu
cxzl25 opened a new pull request, #42085:
URL: https://github.com/apache/spark/pull/42085
### What changes were proposed in this pull request?
Remove `TaskPagedTable`
### Why are the changes needed?
In [SPARK-21809](https://issues.apache.org/jira/browse/SPARK-21809), we
introdu
HyukjinKwon closed pull request #41948: [SPARK-44380][SQL][PYTHON] Support for
Python UDTF to analyze in Python
URL: https://github.com/apache/spark/pull/41948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above
HyukjinKwon commented on PR #41948:
URL: https://github.com/apache/spark/pull/41948#issuecomment-1643063875
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
beliefer opened a new pull request, #42084:
URL: https://github.com/apache/spark/pull/42084
### What changes were proposed in this pull request?
https://github.com/apache/spark/pull/41850 uses
`TYPE_CHECK_FAILURE_WITH_HINT`, it should be
`DATATYPE_MISMATCH.TYPE_CHECK_FAILURE_WITH_HINT`.
cloud-fan closed pull request #42007: [SPARK-44431][SQL] Fix behavior of null
IN (empty list) in optimization rules
URL: https://github.com/apache/spark/pull/42007
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL a
cloud-fan commented on PR #42007:
URL: https://github.com/apache/spark/pull/42007#issuecomment-1643037992
thanks, merging to master/3.5!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specif
richardc-db opened a new pull request, #42083:
URL: https://github.com/apache/spark/pull/42083
### What changes were proposed in this pull request?
Adds support to deserialize long types when creating `Metadata` objects from
`JObject`s.
### Why are the changes needed?
HyukjinKwon commented on code in PR #42037:
URL: https://github.com/apache/spark/pull/42037#discussion_r1268879200
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala:
##
@@ -152,15 +153,25 @@ class OrcFileFormat
assert(supportBat
panbingkun commented on code in PR #41349:
URL: https://github.com/apache/spark/pull/41349#discussion_r1268872661
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2SessionCatalog.scala:
##
@@ -89,10 +89,12 @@ class V2SessionCatalog(catalog: SessionCatalo
panbingkun commented on PR #42082:
URL: https://github.com/apache/spark/pull/42082#issuecomment-1643026104
cc @cloud-fan @MaxGekk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comm
panbingkun commented on code in PR #42082:
URL: https://github.com/apache/spark/pull/42082#discussion_r1268872155
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2SessionCatalog.scala:
##
@@ -88,19 +88,11 @@ class V2SessionCatalog(catalog: SessionCatalo
panbingkun opened a new pull request, #42082:
URL: https://github.com/apache/spark/pull/42082
### What changes were proposed in this pull request?
- The pr is following up https://github.com/apache/spark/pull/41349.
- The pr aims to simplify code logic after merge `_LEGACY_ERROR_TEMP_13
Hisoka-X commented on PR #42081:
URL: https://github.com/apache/spark/pull/42081#issuecomment-1643018942
> Is this related to the GA failure of the master?
No, just bug I fonud when debug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please
LuciferYang commented on PR #42081:
URL: https://github.com/apache/spark/pull/42081#issuecomment-1643016092
Is this related to the GA failure of the master?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
U
beliefer commented on code in PR #41850:
URL: https://github.com/apache/spark/pull/41850#discussion_r1268862800
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala:
##
@@ -277,13 +277,13 @@ trait CheckAnalysis extends PredicateHelper with
L
Hisoka-X commented on PR #41347:
URL: https://github.com/apache/spark/pull/41347#issuecomment-1643006858
Thanks @cloud-fan for your help and @allisonwang-db
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL abov
cloud-fan closed pull request #41347: [SPARK-43838][SQL] Fix subquery on single
table with having clause can't be optimized
URL: https://github.com/apache/spark/pull/41347
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use t
cloud-fan commented on PR #41347:
URL: https://github.com/apache/spark/pull/41347#issuecomment-1643004942
The k8s failure is unrelated, I'm merging it to master, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
panbingkun commented on code in PR #41349:
URL: https://github.com/apache/spark/pull/41349#discussion_r1268852208
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2SessionCatalog.scala:
##
@@ -89,10 +89,12 @@ class V2SessionCatalog(catalog: SessionCatalo
Hisoka-X commented on PR #42081:
URL: https://github.com/apache/spark/pull/42081#issuecomment-1642997959
cc @Yikun @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comme
Hisoka-X opened a new pull request, #42081:
URL: https://github.com/apache/spark/pull/42081
### What changes were proposed in this pull request?
Fix KubernetesSuite report NPE when not set
`spark.kubernetes.test.unpackSparkDir`
```java
Exception encountered when invoking
panbingkun commented on code in PR #41349:
URL: https://github.com/apache/spark/pull/41349#discussion_r1268851681
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2SessionCatalog.scala:
##
@@ -89,10 +89,12 @@ class V2SessionCatalog(catalog: SessionCatalo
itholic commented on code in PR #41711:
URL: https://github.com/apache/spark/pull/41711#discussion_r1268843077
##
dev/error_message_refiner.py:
##
@@ -0,0 +1,265 @@
+#!/usr/bin/env python3
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor
panbingkun commented on code in PR #41349:
URL: https://github.com/apache/spark/pull/41349#discussion_r1268843065
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2SessionCatalog.scala:
##
@@ -89,10 +89,12 @@ class V2SessionCatalog(catalog: SessionCatalo
zhenlineo commented on code in PR #42009:
URL: https://github.com/apache/spark/pull/42009#discussion_r1268824443
##
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/SparkSessionE2ESuite.scala:
##
@@ -96,5 +103,151 @@ class SparkSessionE2ESuite extends RemoteSpark
ericm-db opened a new pull request, #42080:
URL: https://github.com/apache/spark/pull/42080
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How wa
allisonwang-db commented on PR #42075:
URL: https://github.com/apache/spark/pull/42075#issuecomment-1642927436
cc @HyukjinKwon @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the spec
github-actions[bot] commented on PR #40608:
URL: https://github.com/apache/spark/pull/40608#issuecomment-1642927305
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
github-actions[bot] commented on PR #40728:
URL: https://github.com/apache/spark/pull/40728#issuecomment-1642927268
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
xinrong-meng opened a new pull request, #42079:
URL: https://github.com/apache/spark/pull/42079
### What changes were proposed in this pull request?
Implement Arrow `self_destruct` of `toPandas` for memory savings.
Now the Spark configuration
`spark.sql.execution.arrow.pyspark.self
HyukjinKwon closed pull request #42072: [SPARK-44481][CONNECT][PYTHON] Make
pyspark.sql.is_remote an API
URL: https://github.com/apache/spark/pull/42072
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
HyukjinKwon commented on PR #42072:
URL: https://github.com/apache/spark/pull/42072#issuecomment-1642913899
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon closed pull request #41831: [SPARK-44278][CONNECT] Implement a GRPC
server interceptor that cleans up thread local properties
URL: https://github.com/apache/spark/pull/41831
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to Gi
HyukjinKwon commented on PR #41831:
URL: https://github.com/apache/spark/pull/41831#issuecomment-1642910764
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon commented on code in PR #42072:
URL: https://github.com/apache/spark/pull/42072#discussion_r1268791343
##
python/pyspark/sql/__init__.py:
##
@@ -72,4 +73,5 @@
"DataFrameWriter",
"DataFrameWriterV2",
"PandasCogroupedOps",
+"is_remote",
Review Comme
juliuszsompolski commented on code in PR #42009:
URL: https://github.com/apache/spark/pull/42009#discussion_r1268751980
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:
##
@@ -40,6 +40,7 @@ private[sql] class SparkResult[T](
asl3 commented on code in PR #41711:
URL: https://github.com/apache/spark/pull/41711#discussion_r1268722533
##
dev/error_message_refiner.py:
##
@@ -0,0 +1,265 @@
+#!/usr/bin/env python3
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor lic
asl3 commented on code in PR #41711:
URL: https://github.com/apache/spark/pull/41711#discussion_r1268722533
##
dev/error_message_refiner.py:
##
@@ -0,0 +1,265 @@
+#!/usr/bin/env python3
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor lic
asl3 commented on code in PR #41711:
URL: https://github.com/apache/spark/pull/41711#discussion_r1268719990
##
dev/error_message_refiner.py:
##
@@ -0,0 +1,265 @@
+#!/usr/bin/env python3
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor lic
grundprinzip commented on code in PR #42009:
URL: https://github.com/apache/spark/pull/42009#discussion_r1268715542
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:
##
@@ -40,6 +40,7 @@ private[sql] class SparkResult[T](
mathewjacob1002 opened a new pull request, #42078:
URL: https://github.com/apache/spark/pull/42078
First commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscri
juliuszsompolski commented on PR #42009:
URL: https://github.com/apache/spark/pull/42009#issuecomment-1642777980
cc @zhenlineo @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the sp
WweiL opened a new pull request, #42077:
URL: https://github.com/apache/spark/pull/42077
### What changes were proposed in this pull request?
Add the missing field batchDuration to StreamingQueryProgress json method.
Also modify tests accordingly
### Why are the changes
ueshin commented on code in PR #41948:
URL: https://github.com/apache/spark/pull/41948#discussion_r1268593999
##
python/pyspark/sql/udtf.py:
##
@@ -153,6 +175,19 @@ def _validate_udtf_handler(cls: Any) -> None:
error_class="INVALID_UDTF_NO_EVAL", message_parameters=
hvanhovell opened a new pull request, #42076:
URL: https://github.com/apache/spark/pull/42076
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How
dtenedor commented on code in PR #41948:
URL: https://github.com/apache/spark/pull/41948#discussion_r1268552267
##
python/pyspark/sql/udtf.py:
##
@@ -153,6 +175,19 @@ def _validate_udtf_handler(cls: Any) -> None:
error_class="INVALID_UDTF_NO_EVAL", message_parameter
juliuszsompolski commented on code in PR #42009:
URL: https://github.com/apache/spark/pull/42009#discussion_r1268542613
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -613,16 +613,30 @@ class SparkSession private[sql] (
/**
*
ueshin commented on PR #41948:
URL: https://github.com/apache/spark/pull/41948#issuecomment-1642586143
The failing test is [Spark on Kubernetes Integration
test](https://github.com/ueshin/apache-spark/actions/runs/5596078914/jobs/10247102939#logs)
that seems to be broken in `master` branch.
allisonwang-db opened a new pull request, #42075:
URL: https://github.com/apache/spark/pull/42075
### What changes were proposed in this pull request?
This PR supports non-deterministic table-valued functions. More
specifically, it supports running non-deterministic Python UDT
siying opened a new pull request, #42074:
URL: https://github.com/apache/spark/pull/42074
Change the serialization format for group-by-with-state outputs: include an
explicit hidden column indicating how many data and state records there are.
The current implementation of ApplyInPanda
siying commented on PR #42046:
URL: https://github.com/apache/spark/pull/42046#issuecomment-1642557386
> @siying There was a conflict. Could you please create a PR against
branch-3.4? Thanks in advance!
>
> (Btw, I didn't indicate that title is not accurate. Could you please fix
the
siying commented on PR #42046:
URL: https://github.com/apache/spark/pull/42046#issuecomment-1642555659
Sure. Will do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
T
ericm-db commented on PR #42066:
URL: https://github.com/apache/spark/pull/42066#issuecomment-1642470993
> CI looks failing. Could you please look into it?
https://github.com/ericm-db/spark/actions/runs/5595467996/jobs/10231376237
@HeartSaVioR looks like I was setting the runnable mai
amaliujia commented on code in PR #41928:
URL: https://github.com/apache/spark/pull/41928#discussion_r1268235385
##
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkAnalysisUtils.scala:
##
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) un
amaliujia commented on code in PR #41928:
URL: https://github.com/apache/spark/pull/41928#discussion_r1268233819
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParserUtils.scala:
##
@@ -16,34 +16,20 @@
*/
package org.apache.spark.sql.catalyst.parser
-im
cloud-fan commented on code in PR #41349:
URL: https://github.com/apache/spark/pull/41349#discussion_r1268198286
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2SessionCatalog.scala:
##
@@ -89,10 +89,12 @@ class V2SessionCatalog(catalog: SessionCatalog
cloud-fan commented on code in PR #41349:
URL: https://github.com/apache/spark/pull/41349#discussion_r1268192638
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2SessionCatalog.scala:
##
@@ -89,10 +89,12 @@ class V2SessionCatalog(catalog: SessionCatalog
cloud-fan commented on code in PR #41850:
URL: https://github.com/apache/spark/pull/41850#discussion_r1268138809
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala:
##
@@ -277,13 +277,13 @@ trait CheckAnalysis extends PredicateHelper with
hvanhovell closed pull request #42011: [SPARK-44396][Connect] Direct Arrow
Deserialization
URL: https://github.com/apache/spark/pull/42011
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specifi
hvanhovell commented on PR #42011:
URL: https://github.com/apache/spark/pull/42011#issuecomment-1642080475
Merging this. Test failure is unrelated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
jchen5 commented on PR #42007:
URL: https://github.com/apache/spark/pull/42007#issuecomment-1642057004
@cloud-fan PR is updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific commen
heyihong commented on PR #41831:
URL: https://github.com/apache/spark/pull/41831#issuecomment-1642046358
> Should this interceptor be included by default (as the outermost
interceptor)?
I am not sure... But maybe we can do this in a separate pr if needed
--
This is an automated mes
harupy commented on code in PR #42072:
URL: https://github.com/apache/spark/pull/42072#discussion_r1267980107
##
python/pyspark/sql/__init__.py:
##
@@ -72,4 +73,5 @@
"DataFrameWriter",
"DataFrameWriterV2",
"PandasCogroupedOps",
+"is_remote",
Review Comment:
beliefer commented on code in PR #41932:
URL: https://github.com/apache/spark/pull/41932#discussion_r1267967007
##
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/util/RemoteSparkSession.scala:
##
@@ -139,7 +139,18 @@ object SparkConnectServerUtil
LuciferYang commented on code in PR #41932:
URL: https://github.com/apache/spark/pull/41932#discussion_r1267940632
##
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/util/RemoteSparkSession.scala:
##
@@ -139,7 +139,18 @@ object SparkConnectServerU
panbingkun commented on PR #42073:
URL: https://github.com/apache/spark/pull/42073#issuecomment-1641909732
The manual testing process is as follows:
1.My local env as follows:
172.xxx.xxx.xxx
2.Set the following configuration in `spark-defauls.conf`
- spark.connect.grpc.bindin
panbingkun opened a new pull request, #42073:
URL: https://github.com/apache/spark/pull/42073
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch t
beliefer commented on code in PR #41932:
URL: https://github.com/apache/spark/pull/41932#discussion_r1267892365
##
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/ClientE2ETestSuite.scala:
##
@@ -1161,6 +1161,27 @@ class ClientE2ETestSuite extends RemoteSparkSes
HyukjinKwon opened a new pull request, #42072:
URL: https://github.com/apache/spark/pull/42072
### What changes were proposed in this pull request?
### Why are the changes needed?
For the end users to be able to do if-else, e.g., for dispatching the code
path to the legacy mode
yaooqinn closed pull request #42054: [SPARK-44470][BUILD] Setting version to
4.0.0-SNAPSHOT
URL: https://github.com/apache/spark/pull/42054
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specif
yaooqinn commented on PR #42054:
URL: https://github.com/apache/spark/pull/42054#issuecomment-1641794593
CLOSE as duplicated and fixed by SPARK-44467
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
Deependra-Patel opened a new pull request, #42071:
URL: https://github.com/apache/spark/pull/42071
This will be available as external shuffle service metric
### What changes were proposed in this pull request?
Adding three more metrics to ShuffleMetrics (exposed by External Shuffle
HyukjinKwon closed pull request #42070: [MINOR][INFRA] Update the labeler for
CORE and CONNECT
URL: https://github.com/apache/spark/pull/42070
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the spe
HyukjinKwon commented on PR #42070:
URL: https://github.com/apache/spark/pull/42070#issuecomment-1641707945
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon commented on PR #42068:
URL: https://github.com/apache/spark/pull/42068#issuecomment-1641706328
Merged to master and branch-3.5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the sp
1 - 100 of 114 matches
Mail list logo
