dongjoon-hyun commented on PR #36858:
URL: https://github.com/apache/spark/pull/36858#issuecomment-1154340620
Thank you, @huaxingao !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
huaxingao commented on PR #36858:
URL: https://github.com/apache/spark/pull/36858#issuecomment-1154339963
@dongjoon-hyun Thanks for pinging me. The change looks good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and u
dongjoon-hyun commented on PR #36832:
URL: https://github.com/apache/spark/pull/36832#issuecomment-1154339726
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific commen
dongjoon-hyun closed pull request #36832: [SPARK-39439][CORE] Check final file
if in-progress event log file does not exist
URL: https://github.com/apache/spark/pull/36832
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use t
dongjoon-hyun commented on PR #36858:
URL: https://github.com/apache/spark/pull/36858#issuecomment-1154275690
Could you review this test PR, @huaxingao ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
dongjoon-hyun opened a new pull request, #36858:
URL: https://github.com/apache/spark/pull/36858
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### H
cloud-fan closed pull request #36854: [SPARK-39437][SQL][TEST][3.3] Normalize
plan id separately in PlanStabilitySuite
URL: https://github.com/apache/spark/pull/36854
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
UR
dongjoon-hyun commented on PR #36854:
URL: https://github.com/apache/spark/pull/36854#issuecomment-1154193491
Merged to branch-3.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific co
MaxGekk opened a new pull request, #36857:
URL: https://github.com/apache/spark/pull/36857
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was
abhishekd0907 commented on PR #35683:
URL: https://github.com/apache/spark/pull/35683#issuecomment-1154149964
> Can you update the description to reflect the changes made to the PR
@abhishekd0907 ? Specifically - this is only related to decomissioning and we
do not handle shuffle ?
>
maryannxue commented on PR #36845:
URL: https://github.com/apache/spark/pull/36845#issuecomment-1154125105
@wangyum I don't think this fix will solve all problems. Empty relation
propagation can cause invalid plans in other ways, e.g., a
`BroadcastExchangeExec` ending up as the child of a r
cxzl25 commented on code in PR #36740:
URL: https://github.com/apache/spark/pull/36740#discussion_r895854783
##
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##
@@ -2176,4 +2176,32 @@ class SubquerySuite extends QueryTest with
SharedSparkSession with Adaptiv
cxzl25 commented on code in PR #36740:
URL: https://github.com/apache/spark/pull/36740#discussion_r895854783
##
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##
@@ -2176,4 +2176,32 @@ class SubquerySuite extends QueryTest with
SharedSparkSession with Adaptiv
cloud-fan commented on code in PR #36641:
URL: https://github.com/apache/spark/pull/36641#discussion_r895850050
##
sql/core/src/main/scala/org/apache/spark/sql/internal/CatalogImpl.scala:
##
@@ -250,8 +251,14 @@ class CatalogImpl(sparkSession: SparkSession) extends
Catalog {
cloud-fan commented on code in PR #36641:
URL: https://github.com/apache/spark/pull/36641#discussion_r895849585
##
sql/core/src/main/scala/org/apache/spark/sql/internal/CatalogImpl.scala:
##
@@ -287,16 +294,44 @@ class CatalogImpl(sparkSession: SparkSession) extends
Catalog {
cloud-fan commented on code in PR #36641:
URL: https://github.com/apache/spark/pull/36641#discussion_r895848763
##
sql/core/src/test/scala/org/apache/spark/sql/internal/CatalogSuite.scala:
##
@@ -681,4 +681,60 @@ class CatalogSuite extends SharedSparkSession with
AnalysisTest w
beobest2 commented on PR #36729:
URL: https://github.com/apache/spark/pull/36729#issuecomment-1154055619
@HyukjinKwon I've been missing out because there have been some other
workloads lately. I will be able to register for PR this week. I have created
an issue here. (https://issues.apache.
cloud-fan commented on code in PR #36641:
URL: https://github.com/apache/spark/pull/36641#discussion_r895846403
##
sql/core/src/main/scala/org/apache/spark/sql/internal/CatalogImpl.scala:
##
@@ -250,8 +251,14 @@ class CatalogImpl(sparkSession: SparkSession) extends
Catalog {
MaxGekk commented on PR #36852:
URL: https://github.com/apache/spark/pull/36852#issuecomment-1154053041
@panbingkun Could you fix the test failure, please:
```
QueryExecutionErrorsSuite.UNSUPPORTED_SAVE_MODE: unsupported null saveMode
whether the path exists or not
org.scalatest.exc
cloud-fan commented on code in PR #36564:
URL: https://github.com/apache/spark/pull/36564#discussion_r895826751
##
core/src/test/scala/org/apache/spark/scheduler/OutputCommitCoordinatorSuite.scala:
##
@@ -270,6 +263,16 @@ class OutputCommitCoordinatorSuite extends SparkFunSuite
srowen commented on PR #36499:
URL: https://github.com/apache/spark/pull/36499#issuecomment-1154024801
I think you have to retrigger on your end - can you try re-running the jobs?
or push a dummy empty commit?
--
This is an automated message from the Apache Git Service.
To respond to the
cloud-fan commented on code in PR #36785:
URL: https://github.com/apache/spark/pull/36785#discussion_r895811053
##
sql/core/src/main/scala/org/apache/spark/sql/execution/AliasAwareOutputExpression.scala:
##
@@ -25,15 +25,15 @@ import
org.apache.spark.sql.catalyst.plans.physical
srowen commented on code in PR #36856:
URL: https://github.com/apache/spark/pull/36856#discussion_r895807892
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:
##
@@ -53,6 +53,17 @@ case class UnaryMinus(
override def toString: String =
srowen commented on code in PR #35017:
URL: https://github.com/apache/spark/pull/35017#discussion_r895805700
##
sql/core/src/main/java/org/apache/spark/sql/api/java/UDF11.java:
##
@@ -26,5 +26,6 @@
*/
@Stable
public interface UDF11
extends Serializable {
- R call(T1 t1, T2
tgravescs commented on PR #36716:
URL: https://github.com/apache/spark/pull/36716#issuecomment-1154005738
assuming we would allow this with dynamic allocation doc updates, it needs
doc updates for stage level scheduling sections and be sure to talk about None
cores option and perhaps how co
cloud-fan commented on code in PR #36740:
URL: https://github.com/apache/spark/pull/36740#discussion_r895797089
##
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##
@@ -2176,4 +2176,32 @@ class SubquerySuite extends QueryTest with
SharedSparkSession with Adap
cloud-fan commented on PR #36698:
URL: https://github.com/apache/spark/pull/36698#issuecomment-1153996878
For backport, I think your workaround to switch rule order is safer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and
dtenedor commented on code in PR #36771:
URL: https://github.com/apache/spark/pull/36771#discussion_r895767393
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2SessionCatalog.scala:
##
@@ -43,6 +43,8 @@ class V2SessionCatalog(catalog: SessionCatalog)
tgravescs commented on PR #36716:
URL: https://github.com/apache/spark/pull/36716#issuecomment-1153964695
I kind of disagree because it doesn't work as expected compared to other
resource manager. This to me is very confusing. I kind of hate to add more
features on what I would consider
Ngone51 commented on code in PR #36162:
URL: https://github.com/apache/spark/pull/36162#discussion_r895722949
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -1218,6 +1249,71 @@ private[spark] class TaskSetManager(
def executorAdded(): Unit = {
Ngone51 commented on code in PR #36162:
URL: https://github.com/apache/spark/pull/36162#discussion_r895718247
##
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##
@@ -1218,6 +1249,71 @@ private[spark] class TaskSetManager(
def executorAdded(): Unit = {
Ngone51 commented on code in PR #36162:
URL: https://github.com/apache/spark/pull/36162#discussion_r895715985
##
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##
@@ -2073,6 +2073,37 @@ package object config {
.timeConf(TimeUnit.MILLISECONDS)
.
manuzhang commented on PR #36698:
URL: https://github.com/apache/spark/pull/36698#issuecomment-1153901847
Could you help review if I submitted a PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go t
pan3793 commented on code in PR #36832:
URL: https://github.com/apache/spark/pull/36832#discussion_r895699684
##
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##
@@ -715,7 +715,7 @@ private[history] class FsHistoryProvider(conf: SparkConf,
clock:
srowen commented on code in PR #34406:
URL: https://github.com/apache/spark/pull/34406#discussion_r895698008
##
python/pyspark/pandas/namespace.py:
##
@@ -272,7 +272,7 @@ def read_csv(
The character used to denote the start and end of a quoted item.
Quoted items can in
ulysses-you opened a new pull request, #36856:
URL: https://github.com/apache/spark/pull/36856
### What changes were proposed in this pull request?
Pull out data type matching from `eval`. So we do not need to match data
type for every row. Here is the change list:
-
LuciferYang commented on PR #36837:
URL: https://github.com/apache/spark/pull/36837#issuecomment-1153836826
```
// Before this PR
org.apache.spark.sql.catalyst.analysis.DeduplicateRelations 269181786 /
1028666896 124 / 1302
// After this PR
org.apache.spark.sql.catalyst.analys
wangyum commented on code in PR #36847:
URL: https://github.com/apache/spark/pull/36847#discussion_r895634771
##
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##
@@ -4456,6 +4456,20 @@ class SQLQuerySuite extends QueryTest with
SharedSparkSession with Adapti
wangyum commented on code in PR #36850:
URL: https://github.com/apache/spark/pull/36850#discussion_r895609365
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##
@@ -1501,19 +1501,42 @@ object EliminateSorts extends Rule[LogicalPlan] {
*
panbingkun commented on PR #36852:
URL: https://github.com/apache/spark/pull/36852#issuecomment-1153793630
> @panbingkun Could you update PR's description and add "This is a backport
of #36350".
sure！
--
This is an automated message from the Apache Git Service.
To respond to the me
HyukjinKwon closed pull request #36683: [SPARK-39301][SQL][PYTHON] Leverage
LocalRelation and respect Arrow batch size in createDataFrame with Arrow
optimization
URL: https://github.com/apache/spark/pull/36683
--
This is an automated message from the Apache Git Service.
To respond to the mes
HyukjinKwon commented on PR #36683:
URL: https://github.com/apache/spark/pull/36683#issuecomment-1153784036
Let me get this in. It's the early stage of Spark 3.4 so should be good
timing to merge such stuff.
Merged to master.
--
This is an automated message from the Apache Git Serv
MaxGekk commented on code in PR #36852:
URL: https://github.com/apache/spark/pull/36852#discussion_r895568330
##
core/src/main/resources/error/error-classes.json:
##
@@ -184,6 +184,17 @@
"UNSUPPORTED_OPERATION" : {
"message" : [ "The operation is not supported: " ]
},
cloud-fan closed pull request #36853: [SPARK-39316][FOLLOWUP][SQL][TESTS]
Update q83.ansi golden files result
URL: https://github.com/apache/spark/pull/36853
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above t
cloud-fan commented on PR #36853:
URL: https://github.com/apache/spark/pull/36853#issuecomment-1153753040
merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
cloud-fan commented on PR #36698:
URL: https://github.com/apache/spark/pull/36698#issuecomment-1153752402
@manuzhang this is a big refactor and we can't backport...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
U
cloud-fan commented on PR #36811:
URL: https://github.com/apache/spark/pull/36811#issuecomment-1153751639
late LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsub
cloud-fan commented on PR #36854:
URL: https://github.com/apache/spark/pull/36854#issuecomment-1153750232
cc @dongjoon-hyun , I've regen-ed plan golden files under ANSI mode as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to Gi
cloud-fan opened a new pull request, #36854:
URL: https://github.com/apache/spark/pull/36854
### What changes were proposed in this pull request?
In `PlanStabilitySuite`, we normalize expression IDs by matching `#\d+` in
the explain string. However, this regex can match plan i
MaxGekk commented on PR #36852:
URL: https://github.com/apache/spark/pull/36852#issuecomment-1153746752
@panbingkun Could you update PR's description and add "This is a backport of
https://github.com/apache/spark/pull/36350";.
--
This is an automated message from the Apache Git Service.
T
ulysses-you commented on PR #36845:
URL: https://github.com/apache/spark/pull/36845#issuecomment-1153742980
I can re-produce it by:
```sql
CREATE TABLE t1(c1 int) USING PARQUET PARTITIONED BY (p1 string);
CREATE TABLE t2(c2 int) USING PARQUET PARTITIONED BY (p2 string);
SELEC
MaxGekk closed pull request #36811: [SPARK-39451][SQL] Support casting
intervals to integrals in ANSI mode
URL: https://github.com/apache/spark/pull/36811
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to g
MaxGekk commented on PR #36811:
URL: https://github.com/apache/spark/pull/36811#issuecomment-1153723269
Merging to master. Thank you, @gengliangwang and @srielau for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and
panbingkun commented on PR #36852:
URL: https://github.com/apache/spark/pull/36852#issuecomment-1153686788
> I guess we need @MaxGekk 's final sign-off.
OK，Thank you for review @dongjoon-hyun
ping @MaxGekk
--
This is an automated message from the Apache Git Service.
To respond
HyukjinKwon commented on PR #36729:
URL: https://github.com/apache/spark/pull/36729#issuecomment-1153679133
@beobest2, is there any progress on
https://github.com/apache/spark/pull/36729#issuecomment-1141632078?
--
This is an automated message from the Apache Git Service.
To respond to th
HyukjinKwon commented on PR #36749:
URL: https://github.com/apache/spark/pull/36749#issuecomment-1153678058
@beobest2 is there any progress on
https://github.com/apache/spark/pull/36749#issuecomment-1144639942?
--
This is an automated message from the Apache Git Service.
To respond to the
dongjoon-hyun commented on PR #36853:
URL: https://github.com/apache/spark/pull/36853#issuecomment-1153666935
cc @ulysses-you and @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the s
dongjoon-hyun commented on PR #36698:
URL: https://github.com/apache/spark/pull/36698#issuecomment-115366
Here is @wangyum 's follow-up PR
- https://github.com/apache/spark/pull/36853
--
This is an automated message from the Apache Git Service.
To respond to the message, please log
dongjoon-hyun commented on PR #36852:
URL: https://github.com/apache/spark/pull/36852#issuecomment-1153663961
I guess we need @MaxGekk 's final sign-off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
wangyum opened a new pull request, #36853:
URL: https://github.com/apache/spark/pull/36853
### What changes were proposed in this pull request?
Re-generate golden files:
```
SPARK_GENERATE_GOLDEN_FILES=1 SPARK_ANSI_SQL_MODE=true build/sbt
"sql/testOnly *PlanStability*Suite"
`
beliefer commented on PR #36830:
URL: https://github.com/apache/spark/pull/36830#issuecomment-1153565155
ping @huaxingao cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specifi
dongjoon-hyun commented on code in PR #36832:
URL: https://github.com/apache/spark/pull/36832#discussion_r895397461
##
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##
@@ -715,7 +715,7 @@ private[history] class FsHistoryProvider(conf: SparkConf,
c
dongjoon-hyun commented on code in PR #36832:
URL: https://github.com/apache/spark/pull/36832#discussion_r895389373
##
core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala:
##
@@ -221,6 +222,48 @@ abstract class FsHistoryProviderSuite extends
SparkFu
dongjoon-hyun commented on code in PR #36832:
URL: https://github.com/apache/spark/pull/36832#discussion_r895388483
##
core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala:
##
@@ -221,6 +222,48 @@ abstract class FsHistoryProviderSuite extends
SparkFu
101 - 164 of 164 matches
Mail list logo
