Ngone51 commented on code in PR #47197:
URL: https://github.com/apache/spark/pull/47197#discussion_r1663616953
##
core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala:
##
@@ -264,12 +264,37 @@ class TaskMetrics private[spark] () extends Serializable {
/**
* Ext
Ngone51 commented on code in PR #47197:
URL: https://github.com/apache/spark/pull/47197#discussion_r1663611672
##
core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala:
##
@@ -340,7 +365,7 @@ private[spark] object TaskMetrics extends Logging {
externalAccums.a
Ngone51 commented on code in PR #47197:
URL: https://github.com/apache/spark/pull/47197#discussion_r1663609924
##
core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala:
##
@@ -264,12 +264,37 @@ class TaskMetrics private[spark] () extends Serializable {
/**
* Ext
cloud-fan commented on code in PR #47197:
URL: https://github.com/apache/spark/pull/47197#discussion_r1663603187
##
core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala:
##
@@ -340,7 +365,7 @@ private[spark] object TaskMetrics extends Logging {
externalAccums
mridulm commented on code in PR #47197:
URL: https://github.com/apache/spark/pull/47197#discussion_r1663603110
##
core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala:
##
@@ -264,12 +264,37 @@ class TaskMetrics private[spark] () extends Serializable {
/**
* Ext
Ngone51 opened a new pull request, #47197:
URL: https://github.com/apache/spark/pull/47197
### What changes were proposed in this pull request?
This PR proposes to use the `ArrayBuffer` together with the read/write lock
rather than `CopyOnWriteArrayList` for `TaskMetrics._
aokolnychyi commented on PR #47189:
URL: https://github.com/apache/spark/pull/47189#issuecomment-2205197284
Hm, seems like a failure in streaming. I'll check tomorrow.
```
[info] - SPARK-41224: collect data using arrow *** FAILED *** (39
milliseconds)
[info] VerifyEvents.this.
HyukjinKwon commented on PR #47191:
URL: https://github.com/apache/spark/pull/47191#issuecomment-2205127779
cc @HeartSaVioR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon commented on PR #47195:
URL: https://github.com/apache/spark/pull/47195#issuecomment-2205127608
Mind filing a JiRA? See also https://spark.apache.org/contributing.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHu
jackylee-ch commented on code in PR #47193:
URL: https://github.com/apache/spark/pull/47193#discussion_r1663475103
##
sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFEvaluators.scala:
##
@@ -129,7 +129,7 @@ class HiveGenericUDFEvaluator(
override def returnType: Data
amaliujia commented on PR #47196:
URL: https://github.com/apache/spark/pull/47196#issuecomment-2205080743
@cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To uns
amaliujia opened a new pull request, #47196:
URL: https://github.com/apache/spark/pull/47196
### What changes were proposed in this pull request?
Use `checkDatasetUnorderly` in DeprecatedDatasetAggregatorSuite. The tests
do not need depending on the ordering of the result.
yaooqinn commented on code in PR #47185:
URL: https://github.com/apache/spark/pull/47185#discussion_r1663462271
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InlineCTE.scala:
##
@@ -140,26 +140,31 @@ case class InlineCTE(
cteMap: mutable.Map[Long,
yaooqinn commented on code in PR #47185:
URL: https://github.com/apache/spark/pull/47185#discussion_r1663462046
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InlineCTE.scala:
##
@@ -140,26 +140,31 @@ case class InlineCTE(
cteMap: mutable.Map[Long,
amaliujia commented on code in PR #47171:
URL: https://github.com/apache/spark/pull/47171#discussion_r1663461576
##
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##
@@ -42,6 +42,18 @@ package object config {
private[spark] val SPARK_TASK_PREFIX = "spark.
amaliujia commented on code in PR #47185:
URL: https://github.com/apache/spark/pull/47185#discussion_r1663459563
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InlineCTE.scala:
##
@@ -140,26 +140,31 @@ case class InlineCTE(
cteMap: mutable.Map[Long
ragnarok56 opened a new pull request, #47195:
URL: https://github.com/apache/spark/pull/47195
### What changes were proposed in this pull request?
Fix an issue that was identified during testing with changes from
https://github.com/apache/spark/pull/45362. When setting `maxCachedFile
viirya commented on code in PR #47086:
URL: https://github.com/apache/spark/pull/47086#discussion_r1663449734
##
python/pyspark/sql/dataframe.py:
##
@@ -5984,6 +5984,41 @@ def writeTo(self, table: str) -> DataFrameWriterV2:
"""
...
+@dispatch_df_method
+
cloud-fan commented on code in PR #47193:
URL: https://github.com/apache/spark/pull/47193#discussion_r1663441930
##
sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFEvaluators.scala:
##
@@ -129,7 +129,7 @@ class HiveGenericUDFEvaluator(
override def returnType: DataTy
yaooqinn commented on code in PR #47171:
URL: https://github.com/apache/spark/pull/47171#discussion_r1663441737
##
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##
@@ -42,6 +42,18 @@ package object config {
private[spark] val SPARK_TASK_PREFIX = "spark.t
cloud-fan closed pull request #47156: [SPARK-48760][SQL] Introduce ALTER TABLE
... CLUSTER BY SQL syntax to change clustering columns
URL: https://github.com/apache/spark/pull/47156
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub
HyukjinKwon closed pull request #47173: [SPARK-48774][SQL] Use SparkSession in
SQLImplicits
URL: https://github.com/apache/spark/pull/47173
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specif
HyukjinKwon commented on PR #47173:
URL: https://github.com/apache/spark/pull/47173#issuecomment-2205012789
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
panbingkun commented on PR #47191:
URL: https://github.com/apache/spark/pull/47191#issuecomment-2205008709
cc @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
cloud-fan commented on PR #47156:
URL: https://github.com/apache/spark/pull/47156#issuecomment-2205008789
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific c
panbingkun commented on PR #47194:
URL: https://github.com/apache/spark/pull/47194#issuecomment-2205000451
cc @HyukjinKwon @LuciferYang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specif
panbingkun commented on PR #47194:
URL: https://github.com/apache/spark/pull/47194#issuecomment-2205000216
Error detail:
```
Cannot invoke "java.net.URL.getFile()" because the return value of
"java.lang.ClassLoader.getResource(String)" is null
java.lang.NullPointerException: Cannot
panbingkun opened a new pull request, #47194:
URL: https://github.com/apache/spark/pull/47194
### What changes were proposed in this pull request?
The pr aims to replace `getResource` with `getWorkspaceFilePath` to enable
`HiveUDFSuite` to run successfully in the IDE.
### Why are t
LuciferYang commented on PR #47090:
URL: https://github.com/apache/spark/pull/47090#issuecomment-2204997073
cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
T
LuciferYang commented on code in PR #47193:
URL: https://github.com/apache/spark/pull/47193#discussion_r1663418649
##
sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFEvaluators.scala:
##
@@ -129,7 +129,7 @@ class HiveGenericUDFEvaluator(
override def returnType: Data
yaooqinn commented on code in PR #39555:
URL: https://github.com/apache/spark/pull/39555#discussion_r1663417611
##
sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala:
##
@@ -120,19 +121,18 @@ private[hive] class DeferredObjectAdapter(oi:
ObjectInspector, dataType:
yaooqinn commented on PR #47186:
URL: https://github.com/apache/spark/pull/47186#issuecomment-2204987052
It looks very uncommon and incurs a cognitive cost for users. Also cc
@dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log
panbingkun commented on code in PR #39555:
URL: https://github.com/apache/spark/pull/39555#discussion_r1663411103
##
sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala:
##
@@ -120,19 +121,18 @@ private[hive] class DeferredObjectAdapter(oi:
ObjectInspector, dataTyp
panbingkun opened a new pull request, #47193:
URL: https://github.com/apache/spark/pull/47193
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How
yaooqinn commented on code in PR #47185:
URL: https://github.com/apache/spark/pull/47185#discussion_r1663401781
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InlineCTE.scala:
##
@@ -140,26 +140,31 @@ case class InlineCTE(
cteMap: mutable.Map[Long,
cloud-fan commented on code in PR #47185:
URL: https://github.com/apache/spark/pull/47185#discussion_r1663398411
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InlineCTE.scala:
##
@@ -140,26 +140,31 @@ case class InlineCTE(
cteMap: mutable.Map[Long
AngersZh commented on PR #44767:
URL: https://github.com/apache/spark/pull/44767#issuecomment-2204898273
> what was behaviour before? Would be great to show the result before/after
For the query in `cache.sql`
```
EXPLAIN EXTENDED SELECT * FROM cache_nested_cte_table
```
panbingkun commented on code in PR #47097:
URL: https://github.com/apache/spark/pull/47097#discussion_r1663363438
##
docs/sql-ref-syntax-ddl-alter-table.md:
##
@@ -236,20 +236,30 @@ ALTER TABLE table_identifier DROP [ IF EXISTS ]
partition_spec [PURGE]
### SET AND UNSET
liuzqt opened a new pull request, #47192:
URL: https://github.com/apache/spark/pull/47192
### What changes were proposed in this pull request?
Add task on/off heap execution memory in `TaskMetrics`, tracked in
`TaskMemoryManager`.
### Why are the changes needed?
anishshri-db commented on code in PR #47104:
URL: https://github.com/apache/spark/pull/47104#discussion_r1663358230
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/SchemaHelper.scala:
##
@@ -28,6 +33,61 @@ import org.apache.spark.util.Utils
/**
* Hel
anishshri-db commented on code in PR #47104:
URL: https://github.com/apache/spark/pull/47104#discussion_r1663357917
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TransformWithStateExec.scala:
##
@@ -340,11 +370,48 @@ case class TransformWithStateExec(
anishshri-db commented on code in PR #47104:
URL: https://github.com/apache/spark/pull/47104#discussion_r1663357402
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TransformWithStateExec.scala:
##
@@ -92,6 +93,35 @@ case class TransformWithStateExec(
}
anishshri-db commented on code in PR #47104:
URL: https://github.com/apache/spark/pull/47104#discussion_r1663356046
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateSchemaV3File.scala:
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Fou
anishshri-db commented on code in PR #47104:
URL: https://github.com/apache/spark/pull/47104#discussion_r1663348445
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##
@@ -187,23 +187,33 @@ class IncrementalExecution(
}
}
HyukjinKwon closed pull request #47187: [SPARK-48785][DOCS] Add a simple Python
data source example in the user guide
URL: https://github.com/apache/spark/pull/47187
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
HyukjinKwon commented on PR #47187:
URL: https://github.com/apache/spark/pull/47187#issuecomment-2204817803
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
anishshri-db commented on code in PR #47104:
URL: https://github.com/apache/spark/pull/47104#discussion_r1663347349
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala:
##
@@ -189,7 +189,9 @@ trait FlatMapGroupsWithStateExecBase
anishshri-db commented on code in PR #47104:
URL: https://github.com/apache/spark/pull/47104#discussion_r1663344634
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ColumnFamilySchemaFactory.scala:
##
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Fo
HyukjinKwon commented on PR #44767:
URL: https://github.com/apache/spark/pull/44767#issuecomment-2204787858
what was behaviour before? Would be great to show the result before/after
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to Git
HyukjinKwon commented on code in PR #46849:
URL: https://github.com/apache/spark/pull/46849#discussion_r1663332341
##
connector/connect/common/src/main/protobuf/spark/connect/expressions.proto:
##
@@ -401,6 +402,11 @@ message JavaUDF {
bool aggregate = 3;
}
+message TypedA
HyukjinKwon commented on PR #47190:
URL: https://github.com/apache/spark/pull/47190#issuecomment-2204778320
cc @cloud-fan and @allisonwang-db is this related to something you're
working on?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log
HyukjinKwon closed pull request #47086: [SPARK-48714][PYTHON] Implement
`DataFrame.mergeInto` in PySpark
URL: https://github.com/apache/spark/pull/47086
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
HyukjinKwon commented on PR #47086:
URL: https://github.com/apache/spark/pull/47086#issuecomment-2204769072
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon closed pull request #23: [SPARK-48777][BUILD]Migrate build system
to Bazel
URL: https://github.com/apache/spark-connect-go/pull/23
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the spe
HyukjinKwon commented on PR #23:
URL: https://github.com/apache/spark-connect-go/pull/23#issuecomment-2204767584
Merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comm
HyukjinKwon closed pull request #47083: [SPARK-48710][PYTHON] Use NumPy 2.0
compatible types
URL: https://github.com/apache/spark/pull/47083
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the spec
HyukjinKwon closed pull request #47175: [SPARK-48710][PYTHON][3.5] Limit NumPy
version to supported range (>=1.15,<2)
URL: https://github.com/apache/spark/pull/47175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
HyukjinKwon commented on PR #47175:
URL: https://github.com/apache/spark/pull/47175#issuecomment-2204766403
Merged to `branch-3.5` and `branch-3.4`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go t
HyukjinKwon commented on PR #47083:
URL: https://github.com/apache/spark/pull/47083#issuecomment-2204766546
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon commented on PR #47175:
URL: https://github.com/apache/spark/pull/47175#issuecomment-2204765055
> Do we need this in branch-3.4 too?
i will backport this to `branch-3.4`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on
panbingkun opened a new pull request, #47191:
URL: https://github.com/apache/spark/pull/47191
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How
ericm-db closed pull request #47109: Op state metadata
URL: https://github.com/apache/spark/pull/47109
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: r
panbingkun commented on code in PR #47097:
URL: https://github.com/apache/spark/pull/47097#discussion_r1663267483
##
docs/sql-ref-syntax-ddl-alter-table.md:
##
@@ -236,20 +236,30 @@ ALTER TABLE table_identifier DROP [ IF EXISTS ]
partition_spec [PURGE]
### SET AND UNSET
panbingkun commented on code in PR #47097:
URL: https://github.com/apache/spark/pull/47097#discussion_r1663267483
##
docs/sql-ref-syntax-ddl-alter-table.md:
##
@@ -236,20 +236,30 @@ ALTER TABLE table_identifier DROP [ IF EXISTS ]
partition_spec [PURGE]
### SET AND UNSET
aokolnychyi commented on code in PR #47190:
URL: https://github.com/apache/spark/pull/47190#discussion_r1663250258
##
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Procedure.java:
##
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) un
aokolnychyi commented on code in PR #47190:
URL: https://github.com/apache/spark/pull/47190#discussion_r1663248731
##
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ProcedureParameterImpl.java:
##
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Founda
jiangzho opened a new pull request, #17:
URL: https://github.com/apache/spark-kubernetes-operator/pull/17
### What changes were proposed in this pull request?
This is a breakdown PR of #12 - introducing config properties for controller
module. Also, the config package incl
aokolnychyi commented on code in PR #47190:
URL: https://github.com/apache/spark/pull/47190#discussion_r1663231799
##
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ProcedureParameter.java:
##
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation
aokolnychyi commented on code in PR #47190:
URL: https://github.com/apache/spark/pull/47190#discussion_r1663231799
##
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ProcedureParameter.java:
##
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation
aokolnychyi commented on code in PR #47190:
URL: https://github.com/apache/spark/pull/47190#discussion_r1663231799
##
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ProcedureParameter.java:
##
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation
aokolnychyi commented on PR #47189:
URL: https://github.com/apache/spark/pull/47189#issuecomment-2204564355
cc @cloud-fan @viirya @dongjoon-hyun @huaxingao @sunchao @HyukjinKwon
@gengliangwang
--
This is an automated message from the Apache Git Service.
To respond to the message, please
aokolnychyi commented on PR #47190:
URL: https://github.com/apache/spark/pull/47190#issuecomment-2204563955
cc @cloud-fan @viirya @dongjoon-hyun @huaxingao @sunchao @HyukjinKwon
@gengliangwang
--
This is an automated message from the Apache Git Service.
To respond to the message, please
aokolnychyi opened a new pull request, #47190:
URL: https://github.com/apache/spark/pull/47190
### What changes were proposed in this pull request?
This PR contains new connector APIs for loading stored procedures per
[discussed and
voted](https://lists.apache.org/thread/
jakirkham commented on PR #47083:
URL: https://github.com/apache/spark/pull/47083#issuecomment-2204550456
It should be possible to write code that is compatible with NumPy 1 & 2.
That is what most projects are doing
Would look over [the migration guide](
https://numpy.org/devdocs/num
sunchao commented on PR #46447:
URL: https://github.com/apache/spark/pull/46447#issuecomment-2204544790
The benchmark results look OK to me as well - there is no big deviation from
the previous result. Thanks @Fokko for the PR!
--
This is an automated message from the Apache Git Service.
sunchao commented on code in PR #46447:
URL: https://github.com/apache/spark/pull/46447#discussion_r1663221944
##
sql/core/benchmarks/DataSourceReadBenchmark-results.txt:
##
@@ -1,431 +1,431 @@
-
aokolnychyi commented on code in PR #47189:
URL: https://github.com/apache/spark/pull/47189#discussion_r1663218593
##
common/utils/src/main/resources/error/error-conditions.json:
##
@@ -3578,7 +3578,7 @@
},
"REQUIRED_PARAMETER_NOT_FOUND" : {
"message" : [
- "Cann
aokolnychyi commented on code in PR #47189:
URL: https://github.com/apache/spark/pull/47189#discussion_r1663218593
##
common/utils/src/main/resources/error/error-conditions.json:
##
@@ -3578,7 +3578,7 @@
},
"REQUIRED_PARAMETER_NOT_FOUND" : {
"message" : [
- "Cann
aokolnychyi opened a new pull request, #47189:
URL: https://github.com/apache/spark/pull/47189
### What changes were proposed in this pull request?
This PR makes errors in `NamedParametersSupport` generic so that we can
reuse that class to handle argument rearrangement bot
ericm-db commented on code in PR #47104:
URL: https://github.com/apache/spark/pull/47104#discussion_r1663216919
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulProcessorHandleImpl.scala:
##
@@ -232,14 +209,14 @@ class StatefulProcessorHandleImpl(
anishshri-db commented on code in PR #47104:
URL: https://github.com/apache/spark/pull/47104#discussion_r1663216827
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateSchemaV3File.scala:
##
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Fou
anishshri-db commented on code in PR #47104:
URL: https://github.com/apache/spark/pull/47104#discussion_r1663215623
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/SchemaHelper.scala:
##
@@ -28,6 +33,61 @@ import org.apache.spark.util.Utils
/**
* Hel
anishshri-db commented on code in PR #47104:
URL: https://github.com/apache/spark/pull/47104#discussion_r1663213134
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TransformWithStateExec.scala:
##
@@ -340,11 +370,54 @@ case class TransformWithStateExec(
anishshri-db commented on code in PR #47104:
URL: https://github.com/apache/spark/pull/47104#discussion_r1663212263
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala:
##
@@ -246,7 +246,12 @@ case class StreamingSymmetricHash
anishshri-db commented on code in PR #47104:
URL: https://github.com/apache/spark/pull/47104#discussion_r1663211544
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulProcessorHandleImpl.scala:
##
@@ -232,14 +209,14 @@ class StatefulProcessorHandleImpl(
anishshri-db commented on code in PR #47104:
URL: https://github.com/apache/spark/pull/47104#discussion_r1663210224
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##
@@ -187,23 +187,33 @@ class IncrementalExecution(
}
}
anishshri-db commented on code in PR #47104:
URL: https://github.com/apache/spark/pull/47104#discussion_r1663208852
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala:
##
@@ -325,6 +338,23 @@ class HDFSMetadataLog[T <: AnyRef :
ClassTag](s
anishshri-db commented on code in PR #47104:
URL: https://github.com/apache/spark/pull/47104#discussion_r1663207877
##
common/utils/src/main/resources/error/error-conditions.json:
##
@@ -3803,6 +3803,12 @@
],
"sqlState" : "XXKST"
},
+ "STATE_STORE_NEW_COLUMN_FAMILY
allisonwang-db commented on code in PR #46826:
URL: https://github.com/apache/spark/pull/46826#discussion_r1663198085
##
python/pyspark/sql/datasource.py:
##
@@ -332,8 +331,13 @@ def partitions(self) -> Sequence[InputPartition]:
message_parameters={"feature": "parti
eason-yuchen-liu opened a new pull request, #47188:
URL: https://github.com/apache/spark/pull/47188
### What changes were proposed in this pull request?
This PR adds ability of showing the evolution of state as Change Data
Capture (CDC) format to state data source.
An e
jingz-db closed pull request #47169: [DRAFT] Virtual Column Family for RocksDB
- StateStore API change
URL: https://github.com/apache/spark/pull/47169
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
allisonwang-db commented on PR #47184:
URL: https://github.com/apache/spark/pull/47184#issuecomment-2204413706
cc @srielau @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
allisonwang-db opened a new pull request, #47187:
URL: https://github.com/apache/spark/pull/47187
### What changes were proposed in this pull request?
This PR adds a self-contained, simple example implementation of a Python
data source in the user guide to help users get start
HeartSaVioR closed pull request #47167: [SPARK-48770][SS] Change to read
operator metadata once on driver to check if we can find info for
numColsPrefixKey used for session window agg queries
URL: https://github.com/apache/spark/pull/47167
--
This is an automated message from the Apache Git
HeartSaVioR commented on PR #47167:
URL: https://github.com/apache/spark/pull/47167#issuecomment-2204327011
Thanks! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
amaliujia commented on code in PR #47171:
URL: https://github.com/apache/spark/pull/47171#discussion_r1663120895
##
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##
@@ -42,6 +42,17 @@ package object config {
private[spark] val SPARK_TASK_PREFIX = "spark.
amaliujia commented on code in PR #47171:
URL: https://github.com/apache/spark/pull/47171#discussion_r1663120895
##
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##
@@ -42,6 +42,17 @@ package object config {
private[spark] val SPARK_TASK_PREFIX = "spark.
HeartSaVioR closed pull request #46944: [SPARK-48589][SQL][SS] Add option
snapshotStartBatchId and snapshotPartitionId to state data source
URL: https://github.com/apache/spark/pull/46944
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
HeartSaVioR commented on PR #46944:
URL: https://github.com/apache/spark/pull/46944#issuecomment-2204313492
Thanks! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
gene-db opened a new pull request, #47186:
URL: https://github.com/apache/spark/pull/47186
### What changes were proposed in this pull request?
Add the `:::` (triple colon) as syntactic sugar for `try_cast`.
### Why are the changes needed?
This syntactic sugar makes it ea
1 - 100 of 177 matches
Mail list logo
