wangyum opened a new pull request, #43812:
URL: https://github.com/apache/spark/pull/43812
### What changes were proposed in this pull request?
The common type of decimal(x, 0) and string is double. But the common type
of int/bigint and string are int/bigint.
This PR updates `P
LuciferYang commented on PR #43796:
URL: https://github.com/apache/spark/pull/43796#issuecomment-1811959949
@dongjoon-hyun I want to clarify the issue. We don't want to use `record`
here because `field` in the original class doesn't provide an Accessor, but
since `record` automatically gene
HyukjinKwon closed pull request #43799: [SPARK-45764][PYTHON][DOCS] Make code
block copyable
URL: https://github.com/apache/spark/pull/43799
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the speci
HyukjinKwon commented on PR #43799:
URL: https://github.com/apache/spark/pull/43799#issuecomment-1811958045
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
dongjoon-hyun commented on PR #43808:
URL: https://github.com/apache/spark/pull/43808#issuecomment-1811953814
Thank you so much for the details and a upcoming prototype test case.
For that issue, you can file a new JIRA issue.
> Though I ought to point out that while running my test
panbingkun commented on PR #43799:
URL: https://github.com/apache/spark/pull/43799#issuecomment-1811951239
> @panbingkun shall we update the already released docs?
I think it should be possible, but I need to investigate how to add it more
easily. 😄
--
This is an automated message
LuciferYang commented on PR #43796:
URL: https://github.com/apache/spark/pull/43796#issuecomment-1811951525
> This is a nice syntax in general, @LuciferYang .
>
> However, we cannot use this when the class provides information hiding.
>
> ```java
> jshell> private static clas
ahshahid commented on PR #43808:
URL: https://github.com/apache/spark/pull/43808#issuecomment-1811946724
@dongjoon-hyun I think the reason for not catching the issue of reuse of
exchange is a mix of multiple things
1) Spark is not testing with any concrete DataSourceV2 implementation. (
LuciferYang commented on code in PR #43796:
URL: https://github.com/apache/spark/pull/43796#discussion_r1393752725
##
common/network-common/src/main/java/org/apache/spark/network/protocol/StreamChunkId.java:
##
@@ -26,14 +26,7 @@
/**
* Encapsulates a request for a particular c
zhengruifeng commented on PR #42125:
URL: https://github.com/apache/spark/pull/42125#issuecomment-1811928579
> > the python linter fails with
> > ```
> > Python compilation failed with the following errors:
> > *** Error compiling 'dev/aexpy/aexpy/diffing/evaluators/typing.py'...
dongjoon-hyun commented on code in PR #43796:
URL: https://github.com/apache/spark/pull/43796#discussion_r1393751280
##
common/network-common/src/main/java/org/apache/spark/network/protocol/StreamChunkId.java:
##
@@ -26,14 +26,7 @@
/**
* Encapsulates a request for a particular
dongjoon-hyun commented on PR #41072:
URL: https://github.com/apache/spark/pull/41072#issuecomment-1811916481
Thank you, @thepinetree and @cloud-fan . Given that this is a long-standing
overflow bug, do you think we can have this fix in other live release branches,
`branch-3.4` and `branch-
itholic commented on code in PR #43799:
URL: https://github.com/apache/spark/pull/43799#discussion_r1393735691
##
dev/requirements.txt:
##
@@ -37,6 +37,7 @@ numpydoc
jinja2<3.0.0
sphinx<3.1.0
sphinx-plotly-directive
+sphinx-copybutton
Review Comment:
Sounds good. Thanks!
zhengruifeng commented on PR #43799:
URL: https://github.com/apache/spark/pull/43799#issuecomment-1811910417
@panbingkun shall we update the already released docs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
UR
beliefer commented on PR #43786:
URL: https://github.com/apache/spark/pull/43786#issuecomment-1811910009
The GA failure is unrelated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
maintian commented on code in PR #43798:
URL: https://github.com/apache/spark/pull/43798#discussion_r1393718772
##
python/pyspark/sql/readwriter.py:
##
@@ -1936,7 +1936,23 @@ def parquet(
if partitionBy is not None:
self.partitionBy(partitionBy)
se
dongjoon-hyun commented on code in PR #43781:
URL: https://github.com/apache/spark/pull/43781#discussion_r1393707711
##
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##
@@ -4541,6 +4541,15 @@ object SQLConf {
.booleanConf
.createWithDefau
maintian commented on code in PR #43798:
URL: https://github.com/apache/spark/pull/43798#discussion_r1393707527
##
python/pyspark/sql/readwriter.py:
##
@@ -1936,7 +1936,23 @@ def parquet(
if partitionBy is not None:
self.partitionBy(partitionBy)
se
dongjoon-hyun commented on PR #43760:
URL: https://github.com/apache/spark/pull/43760#issuecomment-1811882080
Gentle ping~, @maryannxue .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the speci
cloud-fan commented on PR #43781:
URL: https://github.com/apache/spark/pull/43781#issuecomment-1811881698
> Is there reproducer that can be added as unit test to show the issue in
e2e example?
I think the updated tests show the problem.
--
This is an automated message from the Apa
cloud-fan commented on code in PR #43781:
URL: https://github.com/apache/spark/pull/43781#discussion_r1393704857
##
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##
@@ -4541,6 +4541,15 @@ object SQLConf {
.booleanConf
.createWithDefault(f
cloud-fan commented on code in PR #43781:
URL: https://github.com/apache/spark/pull/43781#discussion_r1393704651
##
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##
@@ -4541,6 +4541,15 @@ object SQLConf {
.booleanConf
.createWithDefault(f
dongjoon-hyun commented on code in PR #43781:
URL: https://github.com/apache/spark/pull/43781#discussion_r1393703751
##
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##
@@ -4541,6 +4541,15 @@ object SQLConf {
.booleanConf
.createWithDefau
cloud-fan closed pull request #41072: [SPARK-43393][SQL] Address sequence
expression overflow bug.
URL: https://github.com/apache/spark/pull/41072
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
cloud-fan commented on PR #41072:
URL: https://github.com/apache/spark/pull/41072#issuecomment-1811877484
thanks, merging to master/3.5!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specif
dongjoon-hyun commented on PR #43797:
URL: https://github.com/apache/spark/pull/43797#issuecomment-1811877168
BTW, the first commit failed with the following relevant failure. Could you
double-check that because the last commit seems irrelevant with that failure,
@cloud-fan ?
https:/
HyukjinKwon commented on code in PR #43798:
URL: https://github.com/apache/spark/pull/43798#discussion_r1393699124
##
python/pyspark/sql/readwriter.py:
##
@@ -1936,7 +1936,23 @@ def parquet(
if partitionBy is not None:
self.partitionBy(partitionBy)
cloud-fan commented on code in PR #43809:
URL: https://github.com/apache/spark/pull/43809#discussion_r1393654355
##
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala:
##
@@ -246,7 +246,15 @@ class DataFrameReader private[sql](sparkSession:
SparkSession) extends
ahshahid commented on PR #43806:
URL: https://github.com/apache/spark/pull/43806#issuecomment-1811817310
The other option is that we make the canonicalized form of both
SubqueryAdaptiveBroadcastExec and SubqueryBroadcastExec to be of type
SubqueryBroadcastExec. that way equals and hashCode
ahshahid commented on PR #43807:
URL: https://github.com/apache/spark/pull/43807#issuecomment-1811817204
The other option is that we make the canonicalized form of both
SubqueryAdaptiveBroadcastExec and SubqueryBroadcastExec to be of type
SubqueryBroadcastExec. that way equals and hashCode
cloud-fan commented on code in PR #43809:
URL: https://github.com/apache/spark/pull/43809#discussion_r1393642111
##
python/pyspark/sql/datasource.py:
##
@@ -45,30 +45,19 @@ class DataSource(ABC):
"""
@final
-def __init__(
-self,
-paths: List[str],
ahshahid commented on PR #43808:
URL: https://github.com/apache/spark/pull/43808#issuecomment-1811808992
@HyukjinKwon thanks for correcting the titles of the PRs. will take care
next time..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log
dead-1ine commented on PR #43794:
URL: https://github.com/apache/spark/pull/43794#issuecomment-1811789159
Thank you :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To un
allisonwang-db opened a new pull request, #43811:
URL: https://github.com/apache/spark/pull/43811
### What changes were proposed in this pull request?
This PR improves the docstring of the dataframe function `mapInPandas`.
### Why are the changes needed?
To improv
allisonwang-db opened a new pull request, #43810:
URL: https://github.com/apache/spark/pull/43810
### What changes were proposed in this pull request?
This PR supports non-deterministic UDFs in MapInPandas and MapInArrow.
### Why are the changes needed?
Currently,
maintian commented on code in PR #43798:
URL: https://github.com/apache/spark/pull/43798#discussion_r1393610399
##
python/pyspark/sql/readwriter.py:
##
@@ -1936,7 +1936,23 @@ def parquet(
if partitionBy is not None:
self.partitionBy(partitionBy)
se
patsukp-db commented on code in PR #43629:
URL: https://github.com/apache/spark/pull/43629#discussion_r1393596801
##
sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala:
##
@@ -37,6 +39,7 @@ import org.apache.spark.sql.execution.QueryExecution
impo
patsukp-db commented on code in PR #43629:
URL: https://github.com/apache/spark/pull/43629#discussion_r1393596801
##
sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala:
##
@@ -37,6 +39,7 @@ import org.apache.spark.sql.execution.QueryExecution
impo
ahshahid commented on code in PR #43806:
URL: https://github.com/apache/spark/pull/43806#discussion_r1393590377
##
sql/core/src/main/scala/org/apache/spark/sql/execution/SubqueryAdaptiveBroadcastExec.scala:
##
@@ -44,9 +46,21 @@ case class SubqueryAdaptiveBroadcastExec(
thr
ahshahid commented on code in PR #43806:
URL: https://github.com/apache/spark/pull/43806#discussion_r1393590377
##
sql/core/src/main/scala/org/apache/spark/sql/execution/SubqueryAdaptiveBroadcastExec.scala:
##
@@ -44,9 +46,21 @@ case class SubqueryAdaptiveBroadcastExec(
thr
ahshahid commented on code in PR #43806:
URL: https://github.com/apache/spark/pull/43806#discussion_r1393590377
##
sql/core/src/main/scala/org/apache/spark/sql/execution/SubqueryAdaptiveBroadcastExec.scala:
##
@@ -44,9 +46,21 @@ case class SubqueryAdaptiveBroadcastExec(
thr
ahshahid commented on code in PR #43806:
URL: https://github.com/apache/spark/pull/43806#discussion_r1393590377
##
sql/core/src/main/scala/org/apache/spark/sql/execution/SubqueryAdaptiveBroadcastExec.scala:
##
@@ -44,9 +46,21 @@ case class SubqueryAdaptiveBroadcastExec(
thr
ahshahid commented on code in PR #43806:
URL: https://github.com/apache/spark/pull/43806#discussion_r1393590377
##
sql/core/src/main/scala/org/apache/spark/sql/execution/SubqueryAdaptiveBroadcastExec.scala:
##
@@ -44,9 +46,21 @@ case class SubqueryAdaptiveBroadcastExec(
thr
ueshin commented on code in PR #43682:
URL: https://github.com/apache/spark/pull/43682#discussion_r1393589351
##
python/pyspark/sql/tests/test_udtf.py:
##
@@ -2482,6 +2533,7 @@ def tearDownClass(cls):
super(UDTFTests, cls).tearDownClass()
+'''
Review Comment:
allisonwang-db commented on code in PR #43784:
URL: https://github.com/apache/spark/pull/43784#discussion_r1393582670
##
python/pyspark/sql/tests/test_python_datasource.py:
##
@@ -118,25 +118,26 @@ def reader(self, schema) -> "DataSourceReader":
self.spark.dataSource.
cloud-fan commented on PR #43797:
URL: https://github.com/apache/spark/pull/43797#issuecomment-1811735675
> Given the following warning, this sounds like this could cause a
correctness issue. Did I understand correctly?
Yes, but only when you manipulate logical plans directly. SQL/Dat
ulysses-you commented on code in PR #43806:
URL: https://github.com/apache/spark/pull/43806#discussion_r1393580889
##
sql/core/src/main/scala/org/apache/spark/sql/execution/SubqueryAdaptiveBroadcastExec.scala:
##
@@ -44,9 +46,21 @@ case class SubqueryAdaptiveBroadcastExec(
yaooqinn commented on PR #43805:
URL: https://github.com/apache/spark/pull/43805#issuecomment-1811732522
FYI,
add orderable support.
https://issues.apache.org/jira/browse/SPARK-29679
https://issues.apache.org/jira/browse/SPARK-29385
drop orderable support.
https://i
cloud-fan commented on code in PR #43781:
URL: https://github.com/apache/spark/pull/43781#discussion_r1393576370
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecision.scala:
##
@@ -64,7 +65,11 @@ object DecimalPrecision extends TypeCoercionRule {
HyukjinKwon commented on code in PR #43784:
URL: https://github.com/apache/spark/pull/43784#discussion_r1393562595
##
python/pyspark/sql/tests/test_python_datasource.py:
##
@@ -118,25 +118,26 @@ def reader(self, schema) -> "DataSourceReader":
self.spark.dataSource.reg
HyukjinKwon commented on code in PR #43789:
URL: https://github.com/apache/spark/pull/43789#discussion_r1393558732
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XSDToSchema.scala:
##
@@ -35,34 +38,32 @@ import org.apache.spark.sql.types._
object XSDT
HyukjinKwon commented on code in PR #43789:
URL: https://github.com/apache/spark/pull/43789#discussion_r1393556662
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XSDToSchema.scala:
##
@@ -35,34 +38,32 @@ import org.apache.spark.sql.types._
object XSDT
HyukjinKwon commented on code in PR #43789:
URL: https://github.com/apache/spark/pull/43789#discussion_r1393556057
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XSDToSchema.scala:
##
@@ -35,34 +38,32 @@ import org.apache.spark.sql.types._
object XSDT
HyukjinKwon commented on code in PR #43789:
URL: https://github.com/apache/spark/pull/43789#discussion_r1393554560
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XSDToSchema.scala:
##
@@ -35,34 +38,32 @@ import org.apache.spark.sql.types._
object XSDT
HyukjinKwon commented on code in PR #43789:
URL: https://github.com/apache/spark/pull/43789#discussion_r1393554560
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XSDToSchema.scala:
##
@@ -35,34 +38,32 @@ import org.apache.spark.sql.types._
object XSDT
HyukjinKwon closed pull request #43794: [MINOR][DOCS] Correct additional Conda
documentation URL to fix 404 errors
URL: https://github.com/apache/spark/pull/43794
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL ab
beliefer commented on PR #43802:
URL: https://github.com/apache/spark/pull/43802#issuecomment-1811694498
@HyukjinKwon Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comme
HyukjinKwon commented on PR #43794:
URL: https://github.com/apache/spark/pull/43794#issuecomment-1811694428
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon commented on PR #43794:
URL: https://github.com/apache/spark/pull/43794#issuecomment-1811694361
I locally verified this change.
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
U
HyukjinKwon closed pull request #43795: [SPARK-45918][PS] Optimize
`MultiIndex.symmetric_difference`
URL: https://github.com/apache/spark/pull/43795
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to t
HyukjinKwon commented on PR #43795:
URL: https://github.com/apache/spark/pull/43795#issuecomment-1811693758
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon commented on code in PR #43797:
URL: https://github.com/apache/spark/pull/43797#discussion_r1393547242
##
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/SubstituteUnresolvedOrdinalsSuite.scala:
##
@@ -67,4 +68,22 @@ class SubstituteUnresolvedOrdin
HyukjinKwon commented on code in PR #43798:
URL: https://github.com/apache/spark/pull/43798#discussion_r1393546849
##
python/pyspark/sql/readwriter.py:
##
@@ -1936,7 +1936,23 @@ def parquet(
if partitionBy is not None:
self.partitionBy(partitionBy)
HyukjinKwon closed pull request #43802: [SPARK-45844][SQL][FOLLOWUP] Improve
the caseSensitivityOrdering for XmlInferSchema
URL: https://github.com/apache/spark/pull/43802
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use t
HyukjinKwon commented on PR #43802:
URL: https://github.com/apache/spark/pull/43802#issuecomment-1811690078
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon closed pull request #43804: [SPARK-45562][SQL][FOLLOW-UP] XML: Fix
SQLSTATE for missing rowTag error
URL: https://github.com/apache/spark/pull/43804
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL abov
HyukjinKwon commented on PR #43804:
URL: https://github.com/apache/spark/pull/43804#issuecomment-1811689374
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon commented on code in PR #43805:
URL: https://github.com/apache/spark/pull/43805#discussion_r1393545050
##
sql/api/src/main/scala/org/apache/spark/sql/catalyst/expressions/OrderUtils.scala:
##
@@ -16,15 +16,16 @@
*/
package org.apache.spark.sql.catalyst.expressions
HyukjinKwon commented on PR #43806:
URL: https://github.com/apache/spark/pull/43806#issuecomment-1811687115
cc @peter-toth and @ulysses-you FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
HyukjinKwon commented on PR #43807:
URL: https://github.com/apache/spark/pull/43807#issuecomment-1811686321
cc @ulysses-you and @peter-toth FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
HyukjinKwon commented on PR #43808:
URL: https://github.com/apache/spark/pull/43808#issuecomment-1811684533
cc @ulysses-you and @peter-toth FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
yaooqinn commented on code in PR #43746:
URL: https://github.com/apache/spark/pull/43746#discussion_r1393541374
##
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##
@@ -2087,6 +2087,17 @@ package object config {
.doubleConf
.createOptional
+
yaooqinn commented on PR #43746:
URL: https://github.com/apache/spark/pull/43746#issuecomment-1811680290
> Preemption on yarn shouldn't be going against the number of failed
executors. If it is then something has changed and we should fix that.
Yes, you are right
> This is a co
HyukjinKwon closed pull request #43790: [SPARK-45913][PYTHON] Make the internal
attributes private from PySpark errors.
URL: https://github.com/apache/spark/pull/43790
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
U
HyukjinKwon commented on PR #43790:
URL: https://github.com/apache/spark/pull/43790#issuecomment-1811678634
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon commented on PR #38624:
URL: https://github.com/apache/spark/pull/38624#issuecomment-1811676307
cc @ueshin and @xinrong-meng for review if you find some time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
panbingkun commented on PR #43799:
URL: https://github.com/apache/spark/pull/43799#issuecomment-1811673011
> You might need to revert
[8375103](https://github.com/apache/spark/commit/83751035685c84c681e88ac6e55fbcc9d6d37ef5)
Done.
--
This is an automated message from the Apache Git
panbingkun commented on code in PR #43799:
URL: https://github.com/apache/spark/pull/43799#discussion_r1393532221
##
dev/requirements.txt:
##
@@ -37,6 +37,7 @@ numpydoc
jinja2<3.0.0
sphinx<3.1.0
sphinx-plotly-directive
+sphinx-copybutton
Review Comment:
Currently `sphinx-
HyukjinKwon closed pull request #43690: [SPARK-45813][CONNECT][PYTHON] Return
the observed metrics from commands
URL: https://github.com/apache/spark/pull/43690
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL abov
HyukjinKwon commented on PR #43690:
URL: https://github.com/apache/spark/pull/43690#issuecomment-1811671653
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HeartSaVioR closed pull request #43425: [SPARK-45511][SS] State Data Source -
Reader
URL: https://github.com/apache/spark/pull/43425
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comm
HeartSaVioR commented on PR #43425:
URL: https://github.com/apache/spark/pull/43425#issuecomment-1811667303
Thanks all for reviewing! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
beliefer commented on PR #43786:
URL: https://github.com/apache/spark/pull/43786#issuecomment-1811662016
ping @MaxGekk cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
itholic commented on PR #43790:
URL: https://github.com/apache/spark/pull/43790#issuecomment-1811642917
CI passed. cc @ueshin @HyukjinKwon could you review an additional fix when
you find some time?
--
This is an automated message from the Apache Git Service.
To respond to the message, pl
dtenedor commented on code in PR #43682:
URL: https://github.com/apache/spark/pull/43682#discussion_r1393418903
##
python/pyspark/sql/tests/test_udtf.py:
##
@@ -2467,6 +2468,53 @@ def terminate(self):
[Row(count=20, buffer="abc")],
)
+def test_udtf_wi
dtenedor commented on code in PR #43682:
URL: https://github.com/apache/spark/pull/43682#discussion_r1393418317
##
python/pyspark/sql/tests/test_udtf.py:
##
@@ -2467,6 +2468,53 @@ def terminate(self):
[Row(count=20, buffer="abc")],
)
+def test_udtf_wi
allisonwang-db commented on code in PR #43784:
URL: https://github.com/apache/spark/pull/43784#discussion_r1393417932
##
python/pyspark/sql/tests/test_python_datasource.py:
##
@@ -118,25 +118,26 @@ def reader(self, schema) -> "DataSourceReader":
self.spark.dataSource.
allisonwang-db commented on PR #43809:
URL: https://github.com/apache/spark/pull/43809#issuecomment-1811529611
cc @HyukjinKwon @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the spec
allisonwang-db opened a new pull request, #43809:
URL: https://github.com/apache/spark/pull/43809
### What changes were proposed in this pull request?
This PR updates how to handle `path` values from the `load()` method.
It changes the DataSource class constructor and add `p
ahshahid opened a new pull request, #43808:
URL: https://github.com/apache/spark/pull/43808
### What changes were proposed in this pull request?
Implementing equals and hashCode in the InMemoryBatchScan and
InMemoryV2FilterBatchScan so that the pushed runtime filters are taken into
acco
viirya commented on code in PR #43781:
URL: https://github.com/apache/spark/pull/43781#discussion_r1393385818
##
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##
@@ -4541,6 +4541,15 @@ object SQLConf {
.booleanConf
.createWithDefault(fals
viirya commented on code in PR #43781:
URL: https://github.com/apache/spark/pull/43781#discussion_r1393383832
##
docs/sql-ref-ansi-compliance.md:
##
@@ -240,6 +240,25 @@ The least common type resolution is used to:
- Derive the result type for expressions such as the case expre
allisonwang-db commented on code in PR #43784:
URL: https://github.com/apache/spark/pull/43784#discussion_r1393381350
##
python/pyspark/sql/tests/test_python_datasource.py:
##
@@ -118,25 +118,26 @@ def reader(self, schema) -> "DataSourceReader":
self.spark.dataSource.
viirya commented on code in PR #43781:
URL: https://github.com/apache/spark/pull/43781#discussion_r1393380697
##
docs/sql-ref-ansi-compliance.md:
##
@@ -240,6 +240,25 @@ The least common type resolution is used to:
- Derive the result type for expressions such as the case expre
viirya commented on code in PR #43781:
URL: https://github.com/apache/spark/pull/43781#discussion_r1393380697
##
docs/sql-ref-ansi-compliance.md:
##
@@ -240,6 +240,25 @@ The least common type resolution is used to:
- Derive the result type for expressions such as the case expre
viirya commented on code in PR #43781:
URL: https://github.com/apache/spark/pull/43781#discussion_r1393379712
##
docs/sql-ref-ansi-compliance.md:
##
@@ -240,6 +240,25 @@ The least common type resolution is used to:
- Derive the result type for expressions such as the case expre
ahshahid opened a new pull request, #43807:
URL: https://github.com/apache/spark/pull/43807
### What changes were proposed in this pull request?
Implementing equals and hashCode in SubqueryBroadcastExec so that it is made
equivalent to SubqueryAdaptiveBroadcastExec . During the bug testin
mridulm commented on PR #43743:
URL: https://github.com/apache/spark/pull/43743#issuecomment-1811435846
Sounds good to me, thanks @dongjoon-hyun !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
HeartSaVioR commented on PR #43425:
URL: https://github.com/apache/spark/pull/43425#issuecomment-1811428536
I'll rebase to retrigger CI and merge if everything is good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use t
ahshahid opened a new pull request, #43806:
URL: https://github.com/apache/spark/pull/43806
### What changes were proposed in this pull request?
The canonicalization of SubqueryAdaptiveBroadcastExec is now canonicalizing
the buildPlan : LogicalPlan
SubqueryAdaptiveBroadcastExec is now
1 - 100 of 190 matches
Mail list logo
