LuciferYang opened a new pull request, #45365:
URL: https://github.com/apache/spark/pull/45365
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How
github-actions[bot] closed pull request #43987: [MINOR][SS] Relax access
constraints on fields of FileStreamSink so the class is extendable, split long
method
URL: https://github.com/apache/spark/pull/43987
--
This is an automated message from the Apache Git Service.
To respond to the messag
github-actions[bot] closed pull request #43981: [MINOR][DOCS] Add official
documentation on the natural order of structs. Second proposal.
URL: https://github.com/apache/spark/pull/43981
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to G
github-actions[bot] closed pull request #43938: [SPARK-46037][SQL] When Left
Join build Left and codegen is turned off, ShuffledHashJoinExec may result in
incorrect results
URL: https://github.com/apache/spark/pull/43938
--
This is an automated message from the Apache Git Service.
To respond
github-actions[bot] closed pull request #43871: [MINOR][DOCS] Clarify sort
behaviour for structs
URL: https://github.com/apache/spark/pull/43871
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the s
HyukjinKwon commented on PR #45366:
URL: https://github.com/apache/spark/pull/45366#issuecomment-1975451930
https://github.com/HyukjinKwon/spark/actions/runs/8134128130
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use t
HyukjinKwon commented on code in PR #45363:
URL: https://github.com/apache/spark/pull/45363#discussion_r1510453526
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala:
##
@@ -259,18 +259,19 @@ class RelationalGroupedDataset private[
SteNicholas commented on PR #45351:
URL: https://github.com/apache/spark/pull/45351#issuecomment-1975460640
@parthchandra, thank you try it out. Have you tried anything wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub an
HyukjinKwon commented on PR #45358:
URL: https://github.com/apache/spark/pull/45358#issuecomment-1975462556
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon closed pull request #45358: [SPARK-47249][CONNECT] Fix bug where
all connect executions are considered abandoned regardless of their actual
status
URL: https://github.com/apache/spark/pull/45358
--
This is an automated message from the Apache Git Service.
To respond to the mess
HyukjinKwon commented on PR #45353:
URL: https://github.com/apache/spark/pull/45353#issuecomment-1975466398
@wForget mind filling:
> Does this PR introduce any user-facing change?
> How was this patch tested?
> Was this patch authored or co-authored using generative AI tooling?
HyukjinKwon commented on PR #45334:
URL: https://github.com/apache/spark/pull/45334#issuecomment-1975469964
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon closed pull request #45334: [MINOR] Update outdated comments for
class `o.a.s.s.functions`
URL: https://github.com/apache/spark/pull/45334
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
panbingkun commented on PR #45235:
URL: https://github.com/apache/spark/pull/45235#issuecomment-1975486898
Thank you for helping to fix this issue. Thank you very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use t
panbingkun commented on PR #45033:
URL: https://github.com/apache/spark/pull/45033#issuecomment-1975489055
+1, LGTM. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific commen
HyukjinKwon commented on PR #45351:
URL: https://github.com/apache/spark/pull/45351#issuecomment-1975495691
The original PR does not have its test together
(https://github.com/apache/spark/pull/44021). @SteNicholas Mind describing how
you tested this?
--
This is an automated message from
TakawaAkirayo closed pull request #45364: [SPARK-47253][CORE][3.5] Allow
LiveEventBus to stop without the completely draining of event queue
URL: https://github.com/apache/spark/pull/45364
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
TakawaAkirayo opened a new pull request, #45367:
URL: https://github.com/apache/spark/pull/45367
### What changes were proposed in this pull request?
Add config
spark.scheduler.listenerbus.eventqueue.waitForEventDispatchExitOnStop(default
true).
Before this PR: The event queue
LuciferYang closed pull request #45356: [SPARK-47246][SQL] Replace
`InternalRow.fromSeq` with `new GenericInternalRow` to save a collection
conversion
URL: https://github.com/apache/spark/pull/45356
--
This is an automated message from the Apache Git Service.
To respond to the message, pleas
LuciferYang commented on PR #45356:
URL: https://github.com/apache/spark/pull/45356#issuecomment-1975533193
Merged into master for Spark 4.0. Thanks @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
ulysses-you commented on code in PR #45357:
URL: https://github.com/apache/spark/pull/45357#discussion_r1510518917
##
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CoalesceShufflePartitions.scala:
##
@@ -126,9 +126,12 @@ case class CoalesceShufflePartitions(ses
waywtdcc commented on PR #40744:
URL: https://github.com/apache/spark/pull/40744#issuecomment-1975541287
Can this PR be merged? I also encountered this scenario
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL a
cloud-fan commented on PR #45290:
URL: https://github.com/apache/spark/pull/45290#issuecomment-1975551130
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific c
cloud-fan closed pull request #45290: [SPARK-46834][SQL][Collations] Support
for aggregates
URL: https://github.com/apache/spark/pull/45290
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specif
wForget commented on PR #45353:
URL: https://github.com/apache/spark/pull/45353#issuecomment-1975557375
> @wForget mind filling:
>
> > Does this PR introduce any user-facing change?
> > How was this patch tested?
> > Was this patch authored or co-authored using generative AI tool
wForget commented on PR #45353:
URL: https://github.com/apache/spark/pull/45353#issuecomment-1975565457
> if it is called after finishWriting()
This seems difficult to reproduce, so I have trouble adding a test case to
cover this fix. But I guess the `if (!writing) {` judgment is to a
cloud-fan commented on PR #45176:
URL: https://github.com/apache/spark/pull/45176#issuecomment-1975565892
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific c
cloud-fan closed pull request #45176: [SPARK-46973][SQL] Skip V2 table lookup
when a table is in V1 table cache
URL: https://github.com/apache/spark/pull/45176
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above
LuciferYang commented on PR #45344:
URL: https://github.com/apache/spark/pull/45344#issuecomment-1975570775
late LGTM.
Before this, I always increase `file descriptors` as a workaround ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please
panbingkun opened a new pull request, #45368:
URL: https://github.com/apache/spark/pull/45368
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
###
LuciferYang commented on code in PR #45368:
URL: https://github.com/apache/spark/pull/45368#discussion_r1510546756
##
sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala:
##
@@ -500,8 +500,7 @@ class FileBasedDataSourceSuite extends QueryTest
test("SPA
cloud-fan commented on code in PR #45357:
URL: https://github.com/apache/spark/pull/45357#discussion_r1510551047
##
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CoalesceShufflePartitions.scala:
##
@@ -126,9 +126,12 @@ case class CoalesceShufflePartitions(sessi
dongjoon-hyun commented on PR #45366:
URL: https://github.com/apache/spark/pull/45366#issuecomment-1975628705
Thank you for trying this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specif
dtarima commented on PR #45181:
URL: https://github.com/apache/spark/pull/45181#issuecomment-1975636851
@doki23 Do you have time to continue working on the pull request? I seems to
me that it's close to completion.
--
This is an automated message from the Apache Git Service.
To respond to
panbingkun closed pull request #45368: [WIP][SPARK-47265][SQL][TESTS] Enable
test of v2 data sources in `FileBasedDataSourceSuite`
URL: https://github.com/apache/spark/pull/45368
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub an
nchammas commented on code in PR #45363:
URL: https://github.com/apache/spark/pull/45363#discussion_r151059
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala:
##
@@ -259,18 +259,19 @@ class RelationalGroupedDataset private[sql
dengziming commented on code in PR #45288:
URL: https://github.com/apache/spark/pull/45288#discussion_r1510559579
##
common/utils/src/main/resources/error/error-classes.json:
##
@@ -2884,6 +2890,12 @@
],
"sqlState" : "42809"
},
+ "NOT_A_UNRESOLVED_ENCODER" : {
Rev
dengziming commented on code in PR #45288:
URL: https://github.com/apache/spark/pull/45288#discussion_r1510560284
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala:
##
@@ -565,7 +565,7 @@ abstract class UnaryExpression extends Expression w
ulysses-you commented on code in PR #45357:
URL: https://github.com/apache/spark/pull/45357#discussion_r1510571235
##
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CoalesceShufflePartitions.scala:
##
@@ -126,9 +126,12 @@ case class CoalesceShufflePartitions(ses
LuciferYang commented on code in PR #45366:
URL: https://github.com/apache/spark/pull/45366#discussion_r1510589219
##
core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala:
##
@@ -731,6 +731,7 @@ class MasterSuite extends SparkFunSuite
master.registerApplic
LuciferYang commented on PR #45367:
URL: https://github.com/apache/spark/pull/45367#issuecomment-1975736636
@TakawaAkirayo Doesn't master have this issue? The pr should be submitted to
the `master` first, then backported to other branches if need.
--
This is an automated message from the
cloud-fan commented on code in PR #45357:
URL: https://github.com/apache/spark/pull/45357#discussion_r1510594245
##
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CoalesceShufflePartitions.scala:
##
@@ -126,9 +126,12 @@ case class CoalesceShufflePartitions(sessi
cloud-fan commented on PR #45133:
URL: https://github.com/apache/spark/pull/45133#issuecomment-1975747191
> Semantically, we should wrap in any_value(), but any_value() throws
RuntimeReplaceableAggregate.aggBufferAttributes should not be called and is not
fully supported
It's because
HyukjinKwon commented on code in PR #45363:
URL: https://github.com/apache/spark/pull/45363#discussion_r1510598001
##
python/pyspark/sql/group.py:
##
@@ -450,6 +446,10 @@ def pivot(self, pivot_col: str, values:
Optional[List["LiteralType"]] = None) ->
values : list, op
HyukjinKwon closed pull request #45353: [SPARK-46094][CORE][FOLLOW-UP] Mark
`writing` as volatile at ExecutorJVMProfiler
URL: https://github.com/apache/spark/pull/45353
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
HyukjinKwon commented on PR #45353:
URL: https://github.com/apache/spark/pull/45353#issuecomment-1975750226
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
doki23 commented on PR #45181:
URL: https://github.com/apache/spark/pull/45181#issuecomment-1975750721
> @doki23 Do you have time to continue working on the pull request? I seems
to me that it's close to completion.
@dtarima Of course, I'm glad to move it forward.
--
This is an aut
cloud-fan commented on code in PR #45282:
URL: https://github.com/apache/spark/pull/45282#discussion_r1510609021
##
sql/core/src/main/scala/org/apache/spark/sql/execution/ExplainUtils.scala:
##
@@ -76,7 +76,7 @@ object ExplainUtils extends AdaptiveSparkPlanHelper {
* 1. Ge
cloud-fan commented on PR #45282:
URL: https://github.com/apache/spark/pull/45282#issuecomment-1975767660
cc @robreeves @liuzqt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comme
cloud-fan commented on PR #45282:
URL: https://github.com/apache/spark/pull/45282#issuecomment-1975770696
I think we can't remove the mutable states (TreeNodeTag) any time soon, we
must live with it and the call sites should be careful when setting it. For
EXPLAIN, my preference is to have
cloud-fan commented on code in PR #45282:
URL: https://github.com/apache/spark/pull/45282#discussion_r1510612237
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala:
##
@@ -1009,18 +1009,19 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]]
ulysses-you commented on code in PR #45282:
URL: https://github.com/apache/spark/pull/45282#discussion_r1510625804
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala:
##
@@ -1009,18 +1009,19 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]
anton5798 commented on PR #45133:
URL: https://github.com/apache/spark/pull/45133#issuecomment-1975792360
@cloud-fan updated the code & pr desc!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to th
ulysses-you commented on code in PR #45282:
URL: https://github.com/apache/spark/pull/45282#discussion_r1510631629
##
sql/core/src/main/scala/org/apache/spark/sql/execution/ExplainUtils.scala:
##
@@ -76,7 +76,7 @@ object ExplainUtils extends AdaptiveSparkPlanHelper {
* 1.
cloud-fan commented on code in PR #45282:
URL: https://github.com/apache/spark/pull/45282#discussion_r1510653381
##
sql/core/src/main/scala/org/apache/spark/sql/execution/ExplainUtils.scala:
##
@@ -76,7 +76,7 @@ object ExplainUtils extends AdaptiveSparkPlanHelper {
* 1. Ge
cloud-fan commented on code in PR #45282:
URL: https://github.com/apache/spark/pull/45282#discussion_r1510653381
##
sql/core/src/main/scala/org/apache/spark/sql/execution/ExplainUtils.scala:
##
@@ -76,7 +76,7 @@ object ExplainUtils extends AdaptiveSparkPlanHelper {
* 1. Ge
cloud-fan commented on code in PR #45282:
URL: https://github.com/apache/spark/pull/45282#discussion_r1510653731
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala:
##
@@ -1009,18 +1009,19 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]]
TakawaAkirayo commented on PR #45367:
URL: https://github.com/apache/spark/pull/45367#issuecomment-1975874852
@LuciferYang The master branch have this issue too, this pr is already
submitted in master branch. I add [3.5] in the title is because I though I must
have a target version for PR.
LuciferYang commented on PR #45367:
URL: https://github.com/apache/spark/pull/45367#issuecomment-1975880459
> this pr is already submitted in master branch
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
ericm-db commented on code in PR #45359:
URL: https://github.com/apache/spark/pull/45359#discussion_r1510692025
##
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateSuite.scala:
##
@@ -195,6 +179,20 @@ class TransformWithStateSuite extends StateStoreMetric
SteNicholas commented on PR #45351:
URL: https://github.com/apache/spark/pull/45351#issuecomment-1975895901
> The original PR does not have its test together (#44021). @SteNicholas
Mind describing how you tested this?
@HyukjinKwon, I only tested this bump in production environment of
HyukjinKwon commented on PR #45366:
URL: https://github.com/apache/spark/pull/45366#issuecomment-1975906508
https://github.com/HyukjinKwon/spark/actions/runs/8136924661/job/22234172189
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
HyukjinKwon commented on PR #45351:
URL: https://github.com/apache/spark/pull/45351#issuecomment-1975909802
That's fine. My point is that please describe it in the PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHu
63 matches
Mail list logo
