MaxGekk commented on code in PR #45064:
URL: https://github.com/apache/spark/pull/45064#discussion_r1489031135
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collationExpressions.scala:
##
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software
HeartSaVioR commented on PR #45023:
URL: https://github.com/apache/spark/pull/45023#issuecomment-1943113395
Could you please check the GA build result and fix accordingly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and
HeartSaVioR commented on code in PR #44927:
URL: https://github.com/apache/spark/pull/44927#discussion_r1488847231
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##
@@ -82,6 +84,39 @@ class IncrementalExecution(
HeartSaVioR commented on code in PR #45092:
URL: https://github.com/apache/spark/pull/45092#discussion_r1488831105
##
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##
@@ -1863,6 +1864,91 @@ class RocksDBSuite extends
HeartSaVioR commented on PR #43210:
URL: https://github.com/apache/spark/pull/43210#issuecomment-1942967890
The error message I've seen was following:
```
[autosummary] failed to import
'pyspark.ml.connect.classification.LogisticRegression': no module named
HeartSaVioR commented on PR #43210:
URL: https://github.com/apache/spark/pull/43210#issuecomment-1942961137
It seems like pyspark docs build is failing due to this - during running
release script against branch-3.5. I can see the docs build pass after
reverting this commit.
It's really
itholic commented on code in PR #44859:
URL: https://github.com/apache/spark/pull/44859#discussion_r1488780632
##
python/pyspark/sql/types.py:
##
@@ -2214,12 +2211,9 @@ def verify_acceptable_types(obj: Any) -> None:
# subclass of them can not be fromInternal in JVM
itholic commented on PR #44881:
URL: https://github.com/apache/spark/pull/44881#issuecomment-1942942082
Yeah, Pandas fixes many bugs from Pandas 2.2.0 that brings couple of
behavior changes
Let me fix them. Thanks for the confirm!
--
This is an automated message from the Apache
dongjoon-hyun commented on PR #45079:
URL: https://github.com/apache/spark/pull/45079#issuecomment-1942919284
You're welcome. Feel free to ping me again on this PR. I'll be here Today
for support.
--
This is an automated message from the Apache Git Service.
To respond to the message,
jingz-db commented on PR #45079:
URL: https://github.com/apache/spark/pull/45079#issuecomment-1942918594
>BTW, in the community, we trust CIs as the ground truth.
This makes sense, I am double checking. Thanks for the quick response!
--
This is an automated message from the Apache
dongjoon-hyun commented on PR #45079:
URL: https://github.com/apache/spark/pull/45079#issuecomment-1942911238
BTW, in the community, we trust CIs as the ground truth. Does your GitHub
Action also fail like you mentioned?
--
This is an automated message from the Apache Git Service.
To
dongjoon-hyun commented on PR #45079:
URL: https://github.com/apache/spark/pull/45079#issuecomment-1942910281
Could you clear up your Maven or Ivy cache?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above
jingz-db commented on PR #45079:
URL: https://github.com/apache/spark/pull/45079#issuecomment-1942909715
I just tried `build/sbt clean package` and then `build/sbt "sql/testOnly
org.apache.spark.sql.execution.python.PythonDataSourceSuite"`, it still gives
the same error as above.
And I
dongjoon-hyun commented on PR #45079:
URL: https://github.com/apache/spark/pull/45079#issuecomment-1942909458
Also, to @cloud-fan and @HyukjinKwon , could you double-check with @jingz-db
and @chaoqin-li1123 ? I can help you if there is a reproducible example in
Apache Spark master branch.
dongjoon-hyun commented on PR #45079:
URL: https://github.com/apache/spark/pull/45079#issuecomment-1942908662
To @jingz-db and @chaoqin-li1123 , are you sure that you are using Apache
Spark `master` instead of `Databricks` master?
--
This is an automated message from the Apache Git
dongjoon-hyun commented on PR #45079:
URL: https://github.com/apache/spark/pull/45079#issuecomment-1942907749
I also tried the following. It succeeded like the following too.
```
$ build/sbt
...
sbt:spark-parent> testOnly
dongjoon-hyun commented on PR #45079:
URL: https://github.com/apache/spark/pull/45079#issuecomment-1942903356
For the record, the following is the result from Apache Spark master branch.
```
$ git log --oneline -n1
63b97c6ad82 (HEAD -> master, apache/master, apache/HEAD)
dongjoon-hyun commented on PR #45079:
URL: https://github.com/apache/spark/pull/45079#issuecomment-1942900495
So, something like this?
```
$ build/sbt "sql/testOnly
org.apache.spark.sql.execution.python.PythonDataSourceSuite"
```
--
This is an automated message from the Apache
chaoqin-li1123 commented on PR #45079:
URL: https://github.com/apache/spark/pull/45079#issuecomment-1942899902
`built/sbt` to enter the scala shell, and `testOnly
org.apache.spark.sql.execution.python.PythonDataSourceSuite` to run the test
within the scala shell. @dongjoon-hyun
--
This
dongjoon-hyun commented on PR #45079:
URL: https://github.com/apache/spark/pull/45079#issuecomment-1942897094
Ur, a full command please, @chaoqin-li1123 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above
chaoqin-li1123 commented on PR #45079:
URL: https://github.com/apache/spark/pull/45079#issuecomment-1942896374
Thanks @dongjoon-hyun
My command is
> build/sbt
>> testOnly org.apache.spark.sql.execution.python.PythonDataSourceSuite
--
This is an automated message from the
dongjoon-hyun commented on PR #45079:
URL: https://github.com/apache/spark/pull/45079#issuecomment-1942895237
I can help you when you provide a reproducible procedure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
dongjoon-hyun commented on PR #45079:
URL: https://github.com/apache/spark/pull/45079#issuecomment-1942894624
Please give me a reproducible command line, @jingz-db . :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
jingz-db commented on PR #45079:
URL: https://github.com/apache/spark/pull/45079#issuecomment-1942893943
Hi @dongjoon-hyun , similar error also happens on my local env with errors
below:
```scala
[error]
ueshin commented on code in PR #45073:
URL: https://github.com/apache/spark/pull/45073#discussion_r1488746220
##
python/pyspark/sql/profiler.py:
##
@@ -158,6 +159,70 @@ def _profile_results(self) -> "ProfileResults":
"""
...
+def dump_perf_profiles(self,
dongjoon-hyun commented on PR #45079:
URL: https://github.com/apache/spark/pull/45079#issuecomment-1942893118
What is your command, @chaoqin-li1123 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
chaoqin-li1123 commented on PR #45079:
URL: https://github.com/apache/spark/pull/45079#issuecomment-1942892329
It seems that this commit break my sbt build in latest master branch
The error message is
github-actions[bot] commented on PR #43079:
URL: https://github.com/apache/spark/pull/43079#issuecomment-1942891730
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
github-actions[bot] closed pull request #43651: [SPARK-45782][CORE][PYTHON] Add
Dataframe API df.explainString()
URL: https://github.com/apache/spark/pull/43651
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
jingz-db opened a new pull request, #45094:
URL: https://github.com/apache/spark/pull/45094
### What changes were proposed in this pull request?
This PR adds changes for MapState implementation in State Api v2. This
implementation adds a new encoder/decoder to encode grouping
xinrong-meng commented on code in PR #45073:
URL: https://github.com/apache/spark/pull/45073#discussion_r1488693924
##
python/pyspark/sql/profiler.py:
##
@@ -158,6 +159,70 @@ def _profile_results(self) -> "ProfileResults":
"""
...
+def
xinrong-meng commented on code in PR #45073:
URL: https://github.com/apache/spark/pull/45073#discussion_r1488687912
##
python/pyspark/sql/profiler.py:
##
@@ -158,6 +159,70 @@ def _profile_results(self) -> "ProfileResults":
"""
...
+def
xinrong-meng commented on code in PR #45073:
URL: https://github.com/apache/spark/pull/45073#discussion_r1488687912
##
python/pyspark/sql/profiler.py:
##
@@ -158,6 +159,70 @@ def _profile_results(self) -> "ProfileResults":
"""
...
+def
liorregev opened a new pull request, #45093:
URL: https://github.com/apache/spark/pull/45093
AliasAwareOutputExpression does not detect that
`select(F.struct($"my_field"))` retains partitioning in case the dataset was
partitioning by `$"my_field"` before the select.
This causes an
HeartSaVioR closed pull request #45038: [SPARK-46979][SS] Add support for
specifying key and value encoder separately and also for each col family in
RocksDB state store provider
URL: https://github.com/apache/spark/pull/45038
--
This is an automated message from the Apache Git Service.
To
HeartSaVioR commented on PR #45038:
URL: https://github.com/apache/spark/pull/45038#issuecomment-1942570003
Thanks! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
sahnib commented on PR #45092:
URL: https://github.com/apache/spark/pull/45092#issuecomment-1942560146
cc: @HeartSaVioR PTAL, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
MaxGekk commented on code in PR #45064:
URL: https://github.com/apache/spark/pull/45064#discussion_r1488560783
##
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:
##
@@ -1410,6 +1422,13 @@ public boolean equals(final Object other) {
}
}
+
dbatomic commented on code in PR #45064:
URL: https://github.com/apache/spark/pull/45064#discussion_r1488536286
##
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:
##
@@ -1410,6 +1422,13 @@ public boolean equals(final Object other) {
}
}
+
sahnib opened a new pull request, #45092:
URL: https://github.com/apache/spark/pull/45092
…
### What changes were proposed in this pull request?
This change cleans up any dangling files tracked as being previously
uploaded if they were cleaned up from the
WweiL commented on PR #45091:
URL: https://github.com/apache/spark/pull/45091#issuecomment-1942363588
@grundprinzip
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To
WweiL opened a new pull request, #45091:
URL: https://github.com/apache/spark/pull/45091
### What changes were proposed in this pull request?
Currently, the StreamingQueryListener for Connect runs on the server side.
From a customer point of view, the purpose of a
dbatomic commented on code in PR #45064:
URL: https://github.com/apache/spark/pull/45064#discussion_r1488515575
##
sql/core/src/test/scala/org/apache/spark/sql/CollationSuite.scala:
##
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
dbatomic commented on code in PR #45064:
URL: https://github.com/apache/spark/pull/45064#discussion_r1488515363
##
sql/core/src/test/scala/org/apache/spark/sql/CollationSuite.scala:
##
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
sahnib commented on code in PR #44961:
URL: https://github.com/apache/spark/pull/44961#discussion_r1488459138
##
sql/api/src/main/scala/org/apache/spark/sql/streaming/ValueState.scala:
##
@@ -46,5 +46,5 @@ private[sql] trait ValueState[S] extends Serializable {
def
dongjoon-hyun commented on PR #45084:
URL: https://github.com/apache/spark/pull/45084#issuecomment-1942113291
Since the RC1 vote fails, I backported this to branch-3.5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
jingz-db commented on code in PR #44927:
URL: https://github.com/apache/spark/pull/44927#discussion_r1488336417
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##
@@ -184,6 +185,41 @@ class IncrementalExecution(
}
}
+
MaxGekk closed pull request #45082: [SPARK-47028][SQL][TESTS] Check
`SparkUnsupportedOperationException` instead of `UnsupportedOperationException`
URL: https://github.com/apache/spark/pull/45082
--
This is an automated message from the Apache Git Service.
To respond to the message, please
MaxGekk commented on PR #45082:
URL: https://github.com/apache/spark/pull/45082#issuecomment-1941167022
Merging to master. Thank you, @LuciferYang for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
itholic commented on code in PR #44881:
URL: https://github.com/apache/spark/pull/44881#discussion_r1487597019
##
python/pyspark/pandas/frame.py:
##
@@ -10607,7 +10607,9 @@ def melt(
name_like_string(name) if name is not None else
"variable_{}".format(i)
MaxGekk commented on code in PR #45064:
URL: https://github.com/apache/spark/pull/45064#discussion_r1487518940
##
sql/core/src/test/scala/org/apache/spark/sql/CollationSuite.scala:
##
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+
MaxGekk commented on code in PR #45064:
URL: https://github.com/apache/spark/pull/45064#discussion_r1487489877
##
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:
##
@@ -1410,6 +1422,13 @@ public boolean equals(final Object other) {
}
}
+
HeartSaVioR commented on code in PR #44961:
URL: https://github.com/apache/spark/pull/44961#discussion_r1487446994
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala:
##
@@ -67,6 +67,16 @@ trait ReadStateStore {
def get(key: UnsafeRow,
HeartSaVioR commented on code in PR #44961:
URL: https://github.com/apache/spark/pull/44961#discussion_r1487255434
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/StatePartitionReader.scala:
##
@@ -78,7 +78,7 @@ class StatePartitionReader(
MaxGekk commented on PR #45082:
URL: https://github.com/apache/spark/pull/45082#issuecomment-1940685375
@panbingkun @srielau @LuciferYang @beliefer @cloud-fan Could you review this
PR, please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please
55 matches
Mail list logo
