MaxGekk commented on code in PR #38615:
URL: https://github.com/apache/spark/pull/38615#discussion_r1020704358
##
core/src/main/resources/error/error-classes.json:
##
@@ -630,6 +630,11 @@
"Input schema can only contain STRING as a key type for a
MAP."
]
},
+
AmplabJenkins commented on PR #38601:
URL: https://github.com/apache/spark/pull/38601#issuecomment-1312357558
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
AmplabJenkins commented on PR #38607:
URL: https://github.com/apache/spark/pull/38607#issuecomment-1312357530
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
AmplabJenkins commented on PR #38603:
URL: https://github.com/apache/spark/pull/38603#issuecomment-1312357548
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
Dam1029 commented on PR #38518:
URL: https://github.com/apache/spark/pull/38518#issuecomment-1312350979
@dongjoon-hyun @Ngone51 Could you help take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above
WeichenXu123 commented on code in PR #37734:
URL: https://github.com/apache/spark/pull/37734#discussion_r1020654939
##
python/pyspark/ml/functions.py:
##
@@ -106,6 +138,602 @@ def array_to_vector(col: Column) -> Column:
return
viirya commented on code in PR #38626:
URL: https://github.com/apache/spark/pull/38626#discussion_r1020639613
##
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala:
##
@@ -51,8 +51,10 @@ class SparkOptimizer(
Batch("Optimize Metadata Only Query",
viirya commented on code in PR #38626:
URL: https://github.com/apache/spark/pull/38626#discussion_r1020639496
##
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala:
##
@@ -51,8 +51,10 @@ class SparkOptimizer(
Batch("Optimize Metadata Only Query",
viirya commented on code in PR #38626:
URL: https://github.com/apache/spark/pull/38626#discussion_r1020639233
##
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala:
##
@@ -51,8 +51,10 @@ class SparkOptimizer(
Batch("Optimize Metadata Only Query",
github-actions[bot] closed pull request #37365: [SPARK-39938][PYTHON][PS]
Accept all inputs of prefix/suffix which implement __str__ in
add_predix/add_suffix
URL: https://github.com/apache/spark/pull/37365
--
This is an automated message from the Apache Git Service.
To respond to the
github-actions[bot] closed pull request #37355: [SPARK-39930][SQL] Introduce
Cache Hints
URL: https://github.com/apache/spark/pull/37355
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
github-actions[bot] commented on PR #37346:
URL: https://github.com/apache/spark/pull/37346#issuecomment-1312282414
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
amaliujia commented on PR #38632:
URL: https://github.com/apache/spark/pull/38632#issuecomment-1312268172
R: @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To
amaliujia opened a new pull request, #38632:
URL: https://github.com/apache/spark/pull/38632
### What changes were proposed in this pull request?
I was writing test cases to test expressions and realized that we can allow
`Project` without input plan. For example, `SELECT 1`
xinrong-meng commented on code in PR #38611:
URL: https://github.com/apache/spark/pull/38611#discussion_r1020600278
##
dev/infra/Dockerfile:
##
@@ -32,7 +32,7 @@ RUN $APT_INSTALL software-properties-common git libxml2-dev
pkg-config curl wget
RUN update-alternatives --set
AmplabJenkins commented on PR #38611:
URL: https://github.com/apache/spark/pull/38611#issuecomment-1312240061
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
AmplabJenkins commented on PR #38615:
URL: https://github.com/apache/spark/pull/38615#issuecomment-1312240034
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
grundprinzip commented on code in PR #38631:
URL: https://github.com/apache/spark/pull/38631#discussion_r1020586736
##
python/pyspark/sql/connect/column.py:
##
@@ -82,6 +82,74 @@ def to_plan(self, session: "RemoteSparkSession") ->
"proto.Expression":
def __str__(self) ->
felipepessoto commented on PR #37616:
URL: https://github.com/apache/spark/pull/37616#issuecomment-1312224727
For Scala is expected to need to call `.expr`, or we need to fix it as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on
mridulm commented on PR #38091:
URL: https://github.com/apache/spark/pull/38091#issuecomment-131768
Let us see if the recent fix addresses the issue - else we can take that
route @LuciferYang
--
This is an automated message from the Apache Git Service.
To respond to the message,
grundprinzip commented on code in PR #38631:
URL: https://github.com/apache/spark/pull/38631#discussion_r1020577947
##
connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##
@@ -334,7 +334,11 @@ class SparkConnectPlanner(session:
ueshin commented on code in PR #38611:
URL: https://github.com/apache/spark/pull/38611#discussion_r1020481025
##
dev/infra/Dockerfile:
##
@@ -32,7 +32,7 @@ RUN $APT_INSTALL software-properties-common git libxml2-dev
pkg-config curl wget
RUN update-alternatives --set java
gengliangwang commented on PR #38567:
URL: https://github.com/apache/spark/pull/38567#issuecomment-1312210749
@mridulm FYI I have sent the SPIP to the dev list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
SandishKumarHN commented on code in PR #38603:
URL: https://github.com/apache/spark/pull/38603#discussion_r1020572339
##
python/pyspark/sql/protobuf/functions.py:
##
@@ -48,8 +48,11 @@ def from_protobuf(
--
data : :class:`~pyspark.sql.Column` or str
amaliujia commented on code in PR #38621:
URL: https://github.com/apache/spark/pull/38621#discussion_r1020483723
##
connector/connect/src/main/protobuf/spark/connect/relations.proto:
##
@@ -253,6 +254,23 @@ message Repartition {
bool shuffle = 3;
}
+// Compose the string
grundprinzip commented on code in PR #38631:
URL: https://github.com/apache/spark/pull/38631#discussion_r1020565080
##
connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##
@@ -334,7 +334,11 @@ class SparkConnectPlanner(session:
grundprinzip commented on code in PR #38631:
URL: https://github.com/apache/spark/pull/38631#discussion_r1020564580
##
python/pyspark/sql/connect/column.py:
##
@@ -82,6 +82,74 @@ def to_plan(self, session: "RemoteSparkSession") ->
"proto.Expression":
def __str__(self) ->
amaliujia commented on code in PR #38607:
URL: https://github.com/apache/spark/pull/38607#discussion_r1020564169
##
python/pyspark/sql/connect/plan.py:
##
@@ -712,6 +712,8 @@ def __init__(self, child: Optional["LogicalPlan"], alias:
str) -> None:
def plan(self, session:
amaliujia commented on PR #38630:
URL: https://github.com/apache/spark/pull/38630#issuecomment-1312198788
Yeah let me take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
amaliujia commented on code in PR #38631:
URL: https://github.com/apache/spark/pull/38631#discussion_r1020563219
##
connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##
@@ -334,7 +334,11 @@ class SparkConnectPlanner(session:
amaliujia commented on code in PR #38631:
URL: https://github.com/apache/spark/pull/38631#discussion_r1020563219
##
connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##
@@ -334,7 +334,11 @@ class SparkConnectPlanner(session:
grundprinzip commented on code in PR #38607:
URL: https://github.com/apache/spark/pull/38607#discussion_r1020562864
##
python/pyspark/sql/connect/plan.py:
##
@@ -712,6 +712,8 @@ def __init__(self, child: Optional["LogicalPlan"], alias:
str) -> None:
def plan(self,
amaliujia commented on code in PR #38631:
URL: https://github.com/apache/spark/pull/38631#discussion_r1020561328
##
python/pyspark/sql/connect/column.py:
##
@@ -82,6 +82,74 @@ def to_plan(self, session: "RemoteSparkSession") ->
"proto.Expression":
def __str__(self) ->
amaliujia commented on code in PR #38631:
URL: https://github.com/apache/spark/pull/38631#discussion_r1020561852
##
connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##
@@ -334,7 +334,11 @@ class SparkConnectPlanner(session:
grundprinzip commented on PR #38630:
URL: https://github.com/apache/spark/pull/38630#issuecomment-1312195716
All this in mind it still makes more sense to keep this as an optional
string field instead of an enum.
--
This is an automated message from the Apache Git Service.
To respond to
grundprinzip commented on PR #38631:
URL: https://github.com/apache/spark/pull/38631#issuecomment-1312192934
@amaliujia @cloud-fan @HyukjinKwon @hvanhovell
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
amaliujia commented on PR #38630:
URL: https://github.com/apache/spark/pull/38630#issuecomment-1312190938
@grundprinzip I actually want to log to different entries for the usage from
different clients.
The perfect logging to me is
x jobs submitted through proto (which
grundprinzip opened a new pull request, #38631:
URL: https://github.com/apache/spark/pull/38631
### What changes were proposed in this pull request?
This extends the implementation of column aliases in Spark Connect with
supporting lists of column names and providing the appropriate
grundprinzip commented on code in PR #38630:
URL: https://github.com/apache/spark/pull/38630#discussion_r1020553812
##
connector/connect/src/main/protobuf/spark/connect/base.proto:
##
@@ -48,6 +54,9 @@ message Request {
// The logical plan to be executed / analyzed.
Plan
grundprinzip commented on code in PR #38630:
URL: https://github.com/apache/spark/pull/38630#discussion_r1020553414
##
connector/connect/src/main/protobuf/spark/connect/base.proto:
##
@@ -48,6 +54,9 @@ message Request {
// The logical plan to be executed / analyzed.
Plan
amaliujia commented on PR #38630:
URL: https://github.com/apache/spark/pull/38630#issuecomment-1312181336
@grundprinzip @HyukjinKwon
cc @hvanhovell
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
amaliujia opened a new pull request, #38630:
URL: https://github.com/apache/spark/pull/38630
### What changes were proposed in this pull request?
This PRs introduces a ENUM into Connect proto that can be included into
Request to indicate the client type.
### Why are
MaxGekk commented on code in PR #38623:
URL: https://github.com/apache/spark/pull/38623#discussion_r1020544244
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala:
##
@@ -321,7 +321,10 @@ abstract class StreamExecution(
// to `new
MaxGekk opened a new pull request, #38629:
URL: https://github.com/apache/spark/pull/38629
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How
amaliujia commented on code in PR #38621:
URL: https://github.com/apache/spark/pull/38621#discussion_r1020488570
##
python/pyspark/sql/tests/connect/test_connect_basic.py:
##
@@ -217,6 +217,12 @@ def test_empty_dataset(self):
def test_session(self):
amaliujia commented on code in PR #38595:
URL: https://github.com/apache/spark/pull/38595#discussion_r1020520053
##
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -3804,6 +3804,13 @@ class Dataset[T] private[sql](
} catch {
case _: ParseException
amaliujia commented on code in PR #38595:
URL: https://github.com/apache/spark/pull/38595#discussion_r1020516505
##
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -3804,6 +3804,13 @@ class Dataset[T] private[sql](
} catch {
case _: ParseException
zsxwing commented on code in PR #38623:
URL: https://github.com/apache/spark/pull/38623#discussion_r1020491006
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala:
##
@@ -321,7 +321,10 @@ abstract class StreamExecution(
// to `new
amaliujia commented on code in PR #38621:
URL: https://github.com/apache/spark/pull/38621#discussion_r1020488570
##
python/pyspark/sql/tests/connect/test_connect_basic.py:
##
@@ -217,6 +217,12 @@ def test_empty_dataset(self):
def test_session(self):
amaliujia commented on code in PR #38621:
URL: https://github.com/apache/spark/pull/38621#discussion_r1020484857
##
connector/connect/src/main/protobuf/spark/connect/relations.proto:
##
@@ -253,6 +254,23 @@ message Repartition {
bool shuffle = 3;
}
+// Compose the string
amaliujia commented on code in PR #38621:
URL: https://github.com/apache/spark/pull/38621#discussion_r1020483723
##
connector/connect/src/main/protobuf/spark/connect/relations.proto:
##
@@ -253,6 +254,23 @@ message Repartition {
bool shuffle = 3;
}
+// Compose the string
ueshin commented on code in PR #38611:
URL: https://github.com/apache/spark/pull/38611#discussion_r1020481025
##
dev/infra/Dockerfile:
##
@@ -32,7 +32,7 @@ RUN $APT_INSTALL software-properties-common git libxml2-dev
pkg-config curl wget
RUN update-alternatives --set java
kazuyukitanimura commented on PR #38628:
URL: https://github.com/apache/spark/pull/38628#issuecomment-1312069028
cc @huaxingao @sunchao @viirya
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
kazuyukitanimura opened a new pull request, #38628:
URL: https://github.com/apache/spark/pull/38628
### What changes were proposed in this pull request?
Parquet supports FIXED_LEN_BYTE_ARRAY (FLBA) data type. However, Spark
Parquet reader currently cannot handle FLBA.
This PR proposes
carlfu-db commented on code in PR #38404:
URL: https://github.com/apache/spark/pull/38404#discussion_r1013470376
##
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##
@@ -319,7 +319,7 @@ query
insertInto
: INSERT OVERWRITE TABLE?
grundprinzip commented on PR #38627:
URL: https://github.com/apache/spark/pull/38627#issuecomment-1311969221
R: @cloud-fan @amaliujia @zhengruifeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
grundprinzip opened a new pull request, #38627:
URL: https://github.com/apache/spark/pull/38627
### What changes were proposed in this pull request?
This is a follow-up improving the behavior and compatibility for aggregate
relations using Spark Connect. Previously, Spark Connect
19Serhii99 commented on PR #38574:
URL: https://github.com/apache/spark/pull/38574#issuecomment-1311956108
Need help with fixing the integration tests. I did not expect they would
fail as I had replaced the constants with the methods and that's it.
--
This is an automated message from
srielau commented on code in PR #38615:
URL: https://github.com/apache/spark/pull/38615#discussion_r1020406807
##
core/src/main/resources/error/error-classes.json:
##
@@ -630,6 +630,11 @@
"Input schema can only contain STRING as a key type for a
MAP."
]
},
+
AmplabJenkins commented on PR #38622:
URL: https://github.com/apache/spark/pull/38622#issuecomment-1311933990
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
cloud-fan commented on PR #38626:
URL: https://github.com/apache/spark/pull/38626#issuecomment-1311906245
cc @aokolnychyi @viirya
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
cloud-fan opened a new pull request, #38626:
URL: https://github.com/apache/spark/pull/38626
### What changes were proposed in this pull request?
This is a followup of https://github.com/apache/spark/pull/38557 . We found
that some optimizer rules can't be applied twice
WeichenXu123 commented on PR #37734:
URL: https://github.com/apache/spark/pull/37734#issuecomment-1311877149
@mengxr Could you make a final pass ? The PR is LGTM once all my comments
addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please
WeichenXu123 commented on code in PR #37734:
URL: https://github.com/apache/spark/pull/37734#discussion_r1020345415
##
python/pyspark/ml/functions.py:
##
@@ -106,6 +117,601 @@ def array_to_vector(col: Column) -> Column:
return
WeichenXu123 commented on code in PR #37734:
URL: https://github.com/apache/spark/pull/37734#discussion_r1020344325
##
python/pyspark/ml/functions.py:
##
@@ -106,6 +117,601 @@ def array_to_vector(col: Column) -> Column:
return
WeichenXu123 commented on code in PR #37734:
URL: https://github.com/apache/spark/pull/37734#discussion_r1020344325
##
python/pyspark/ml/functions.py:
##
@@ -106,6 +117,601 @@ def array_to_vector(col: Column) -> Column:
return
deepyaman commented on PR #38625:
URL: https://github.com/apache/spark/pull/38625#issuecomment-1311866247
Not sure why the Build job is failing; Actions are enabled on my fork, and I
did run the following, as specified in the failed job, just in case:
```bash
git fetch upstream
WeichenXu123 commented on code in PR #37734:
URL: https://github.com/apache/spark/pull/37734#discussion_r1020333239
##
python/pyspark/ml/functions.py:
##
@@ -106,6 +117,601 @@ def array_to_vector(col: Column) -> Column:
return
WeichenXu123 commented on code in PR #37734:
URL: https://github.com/apache/spark/pull/37734#discussion_r1020332942
##
python/pyspark/ml/functions.py:
##
@@ -106,6 +117,601 @@ def array_to_vector(col: Column) -> Column:
return
WeichenXu123 commented on code in PR #37734:
URL: https://github.com/apache/spark/pull/37734#discussion_r1020331415
##
python/pyspark/ml/functions.py:
##
@@ -106,6 +117,601 @@ def array_to_vector(col: Column) -> Column:
return
srielau commented on code in PR #38531:
URL: https://github.com/apache/spark/pull/38531#discussion_r1020324789
##
core/src/main/resources/error/error-classes.json:
##
@@ -290,6 +290,46 @@
"Null typed values cannot be used as arguments of ."
]
},
+
deepyaman opened a new pull request, #38625:
URL: https://github.com/apache/spark/pull/38625
### What changes were proposed in this pull request?
Update the docstring for the `.groupby()` method.
### Why are the changes needed?
The `.groupby()` method accept
WeichenXu123 commented on code in PR #37734:
URL: https://github.com/apache/spark/pull/37734#discussion_r1020309114
##
python/pyspark/ml/model_cache.py:
##
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license
AmplabJenkins commented on PR #38624:
URL: https://github.com/apache/spark/pull/38624#issuecomment-1311784451
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
srowen closed pull request #38596: [SPARK-41093][BUILD] Remove
netty-tcnative-classes from Spark dependencyList
URL: https://github.com/apache/spark/pull/38596
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
srowen commented on PR #38596:
URL: https://github.com/apache/spark/pull/38596#issuecomment-1311748449
Merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To
mridulm commented on PR #38617:
URL: https://github.com/apache/spark/pull/38617#issuecomment-1311717433
Thanks @HyukjinKwon , @LuciferYang !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
wankunde commented on code in PR #38495:
URL: https://github.com/apache/spark/pull/38495#discussion_r1020228031
##
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/command/AlterTableDropPartitionSuite.scala:
##
@@ -39,11 +39,11 @@ class AlterTableDropPartitionSuite
MaxGekk commented on PR #38531:
URL: https://github.com/apache/spark/pull/38531#issuecomment-1311696603
also cc @dtenedor @cloud-fan @srielau @itholic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
MaxGekk commented on code in PR #38531:
URL: https://github.com/apache/spark/pull/38531#discussion_r1020213977
##
core/src/main/resources/error/error-classes.json:
##
@@ -290,6 +290,46 @@
"Null typed values cannot be used as arguments of ."
]
},
+
HyukjinKwon commented on code in PR #38621:
URL: https://github.com/apache/spark/pull/38621#discussion_r1020209283
##
connector/connect/src/main/protobuf/spark/connect/relations.proto:
##
@@ -253,6 +254,23 @@ message Repartition {
bool shuffle = 3;
}
+// Compose the
HyukjinKwon commented on code in PR #38621:
URL: https://github.com/apache/spark/pull/38621#discussion_r1020208301
##
python/pyspark/sql/connect/dataframe.py:
##
@@ -388,8 +388,55 @@ def sample(
session=self._session,
)
-def show(self, n: int,
HyukjinKwon commented on code in PR #38621:
URL: https://github.com/apache/spark/pull/38621#discussion_r1020207990
##
python/pyspark/sql/connect/dataframe.py:
##
@@ -388,8 +388,55 @@ def sample(
session=self._session,
)
-def show(self, n: int,
MaxGekk closed pull request #38507: [SPARK-40372][SQL] Migrate failures of
array type checks onto error classes
URL: https://github.com/apache/spark/pull/38507
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
wangyum commented on code in PR #38495:
URL: https://github.com/apache/spark/pull/38495#discussion_r1020196122
##
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/command/AlterTableDropPartitionSuite.scala:
##
@@ -39,11 +39,11 @@ class AlterTableDropPartitionSuite
MaxGekk commented on PR #38507:
URL: https://github.com/apache/spark/pull/38507#issuecomment-1311659255
+1, LGTM. Merging to master. All GAs passed on the previous commit, and the
last one is just a rebase.
Thank you, @LuciferYang.
--
This is an automated message from the Apache Git
EnricoMi opened a new pull request, #38624:
URL: https://github.com/apache/spark/pull/38624
### What changes were proposed in this pull request?
Add `applyInArrow` method to PySpark `groupBy` and `groupBy.cogroup` to
allow for user functions that work on Arrow. Similar to existing
HyukjinKwon closed pull request #38614: [SPARK-41005][CONNECT][FOLLOWUP]
Collect should use `submitJob` instead of `runJob`
URL: https://github.com/apache/spark/pull/38614
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
HyukjinKwon commented on PR #38614:
URL: https://github.com/apache/spark/pull/38614#issuecomment-1311628662
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
HyukjinKwon commented on PR #38613:
URL: https://github.com/apache/spark/pull/38613#issuecomment-1311621739
Actually let's just go with https://github.com/apache/spark/pull/38614
approach which is simpler. This approach can't easily dedup the codes anyway
because of ordering anyway.
--
HyukjinKwon closed pull request #38613:
[SPARK-41005][CONNECT][PYTHON][FOLLOW-UP] Fetch/send partitions in parallel for
Arrow based collect
URL: https://github.com/apache/spark/pull/38613
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on
hvanhovell commented on code in PR #38613:
URL: https://github.com/apache/spark/pull/38613#discussion_r1020162907
##
connector/connect/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectStreamHandler.scala:
##
@@ -144,36 +144,10 @@ class
HyukjinKwon commented on code in PR #38613:
URL: https://github.com/apache/spark/pull/38613#discussion_r1020130165
##
connector/connect/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectStreamHandler.scala:
##
@@ -144,36 +144,10 @@ class
hvanhovell commented on code in PR #38613:
URL: https://github.com/apache/spark/pull/38613#discussion_r1020133986
##
connector/connect/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectStreamHandler.scala:
##
@@ -144,36 +144,10 @@ class
MaxGekk commented on code in PR #38615:
URL: https://github.com/apache/spark/pull/38615#discussion_r1020153905
##
core/src/main/resources/error/error-classes.json:
##
@@ -630,6 +630,11 @@
"Input schema can only contain STRING as a key type for a
MAP."
]
},
+
hvanhovell commented on code in PR #38613:
URL: https://github.com/apache/spark/pull/38613#discussion_r1020133986
##
connector/connect/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectStreamHandler.scala:
##
@@ -144,36 +144,10 @@ class
HyukjinKwon commented on code in PR #38613:
URL: https://github.com/apache/spark/pull/38613#discussion_r1020130165
##
connector/connect/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectStreamHandler.scala:
##
@@ -144,36 +144,10 @@ class
HyukjinKwon commented on code in PR #38613:
URL: https://github.com/apache/spark/pull/38613#discussion_r1020127951
##
connector/connect/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectStreamHandler.scala:
##
@@ -56,7 +56,7 @@ class
HyukjinKwon commented on code in PR #38613:
URL: https://github.com/apache/spark/pull/38613#discussion_r1020127788
##
connector/connect/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectStreamHandler.scala:
##
@@ -56,7 +56,7 @@ class
hvanhovell commented on code in PR #38613:
URL: https://github.com/apache/spark/pull/38613#discussion_r1020126889
##
connector/connect/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectStreamHandler.scala:
##
@@ -184,9 +158,30 @@ class
1 - 100 of 120 matches
Mail list logo