[jira] [Resolved] (SPARK-43165) Move canWrite to DataTypeUtils

[ https://issues.apache.org/jira/browse/SPARK-43165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-43165. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40825

[jira] [Updated] (SPARK-43179) Add option for applications to control saving of metadata in the External Shuffle Service LevelDB

[ https://issues.apache.org/jira/browse/SPARK-43179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated SPARK-43179: -- Summary: Add option for applications to control saving of metadata in the External Shuffle

[jira] [Updated] (SPARK-43179) Add option for applications to control saving of metadata in External Shuffle Service LevelDB

[ https://issues.apache.org/jira/browse/SPARK-43179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated SPARK-43179: -- Summary: Add option for applications to control saving of metadata in External Shuffle

[jira] [Created] (SPARK-43179) Allow applications to control whether their metadata gets saved by the shuffle server in the db

Chandni Singh created SPARK-43179: - Summary: Allow applications to control whether their metadata gets saved by the shuffle server in the db Key: SPARK-43179 URL: https://issues.apache.org/jira/browse/SPARK-43179

[jira] [Commented] (SPARK-35877) Spark Protobuf jar has CVE issue CVE-2015-5237

[ https://issues.apache.org/jira/browse/SPARK-35877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713891#comment-17713891 ] Abhay Dandekar commented on SPARK-35877: Dear team, any target version for this protobuf

[jira] [Commented] (SPARK-42845) Assign a name to the error class _LEGACY_ERROR_TEMP_2010

[ https://issues.apache.org/jira/browse/SPARK-42845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713886#comment-17713886 ] Snoot.io commented on SPARK-42845: -- User 'liang3zy22' has created a pull request for this issue:

[jira] [Commented] (SPARK-42845) Assign a name to the error class _LEGACY_ERROR_TEMP_2010

[ https://issues.apache.org/jira/browse/SPARK-42845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713885#comment-17713885 ] Snoot.io commented on SPARK-42845: -- User 'liang3zy22' has created a pull request for this issue:

[jira] [Resolved] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-43170. - Resolution: Not A Bug > The spark sql like statement is pushed down to parquet for execution,

[jira] [Updated] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-43170: Attachment: screenshot-1.png > The spark sql like statement is pushed down to parquet for

[jira] [Commented] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713861#comment-17713861 ] Yuming Wang commented on SPARK-43170: - Maybe your partition exists, but there is no data under the

[jira] [Commented] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713858#comment-17713858 ] Yuming Wang commented on SPARK-43170: - I can't reproduce this issue:

[jira] [Updated] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-43170: Attachment: image-2023-04-19-10-59-44-118.png > The spark sql like statement is pushed down to

[jira] [Commented] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713855#comment-17713855 ] todd commented on SPARK-43170: -- [~yumwang]  The code only executes spark.sql("xxx"), but does not perform

[jira] [Assigned] (SPARK-43098) Should not handle the COUNT bug when the GROUP BY clause of a correlated scalar subquery is non-empty

[ https://issues.apache.org/jira/browse/SPARK-43098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-43098: --- Assignee: Jack Chen > Should not handle the COUNT bug when the GROUP BY clause of a

[jira] [Resolved] (SPARK-43098) Should not handle the COUNT bug when the GROUP BY clause of a correlated scalar subquery is non-empty

[ https://issues.apache.org/jira/browse/SPARK-43098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-43098. - Fix Version/s: 3.5.0 3.4.1 Resolution: Fixed Issue resolved by pull

[jira] [Resolved] (SPARK-43146) Implement eager evaluation.

[ https://issues.apache.org/jira/browse/SPARK-43146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43146. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40800

[jira] [Assigned] (SPARK-43146) Implement eager evaluation.

[ https://issues.apache.org/jira/browse/SPARK-43146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43146: Assignee: Takuya Ueshin > Implement eager evaluation. > --- > >

[jira] [Commented] (SPARK-42592) Document SS guide doc for supporting multiple stateful operators (especially chained aggregations)

[ https://issues.apache.org/jira/browse/SPARK-42592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713811#comment-17713811 ] Jungtaek Lim commented on SPARK-42592: -- [~XinrongM] Maybe we missed to re-tag fixed versions for

[jira] [Updated] (SPARK-42592) Document SS guide doc for supporting multiple stateful operators (especially chained aggregations)

[ https://issues.apache.org/jira/browse/SPARK-42592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-42592: - Fix Version/s: 3.4.0 (was: 3.4.1) > Document SS guide doc for supporting

[jira] [Created] (SPARK-43178) Migrate UDF errors into error class

Haejoon Lee created SPARK-43178: --- Summary: Migrate UDF errors into error class Key: SPARK-43178 URL: https://issues.apache.org/jira/browse/SPARK-43178 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-43173) `write jdbc` in `ClientE2ETestSuite` will test fail with out `-Phive`

[ https://issues.apache.org/jira/browse/SPARK-43173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43173. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40837

[jira] [Assigned] (SPARK-43173) `write jdbc` in `ClientE2ETestSuite` will test fail with out `-Phive`

[ https://issues.apache.org/jira/browse/SPARK-43173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43173: Assignee: Yang Jie > `write jdbc` in `ClientE2ETestSuite` will test fail with out

[jira] [Resolved] (SPARK-43169) Update mima's previousSparkVersion to 3.4.0

[ https://issues.apache.org/jira/browse/SPARK-43169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43169. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40830

[jira] [Assigned] (SPARK-43169) Update mima's previousSparkVersion to 3.4.0

[ https://issues.apache.org/jira/browse/SPARK-43169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43169: Assignee: Yang Jie > Update mima's previousSparkVersion to 3.4.0 >

[jira] [Commented] (SPARK-42539) User-provided JARs can override Spark's Hive metadata client JARs when using "builtin"

[ https://issues.apache.org/jira/browse/SPARK-42539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713801#comment-17713801 ] Sun Chao commented on SPARK-42539: -- Oops my bad [~xkrogen] - you're right, this is not in Spark 3.5

[jira] [Assigned] (SPARK-42539) User-provided JARs can override Spark's Hive metadata client JARs when using "builtin"

[ https://issues.apache.org/jira/browse/SPARK-42539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sun Chao reassigned SPARK-42539: Assignee: Erik Krogen > User-provided JARs can override Spark's Hive metadata client JARs when

[jira] [Resolved] (SPARK-42539) User-provided JARs can override Spark's Hive metadata client JARs when using "builtin"

[ https://issues.apache.org/jira/browse/SPARK-42539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen resolved SPARK-42539. - Resolution: Fixed [~csun] it looks like this didn't get marked as closed / fix-version updated

[jira] [Updated] (SPARK-42539) User-provided JARs can override Spark's Hive metadata client JARs when using "builtin"

[ https://issues.apache.org/jira/browse/SPARK-42539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-42539: Fix Version/s: 3.5.0 > User-provided JARs can override Spark's Hive metadata client JARs when

[jira] [Commented] (SPARK-43167) Streaming Connect console output format support

[ https://issues.apache.org/jira/browse/SPARK-43167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713787#comment-17713787 ] Wei Liu commented on SPARK-43167: - Should be: ``` Welcome to                     __      / __/__ 

[jira] [Created] (SPARK-43177) Add deprecation warning for input_file_name()

Yaohua Zhao created SPARK-43177: --- Summary: Add deprecation warning for input_file_name() Key: SPARK-43177 URL: https://issues.apache.org/jira/browse/SPARK-43177 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-42452) Remove hadoop-2 profile from Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-42452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved SPARK-42452. -- Fix Version/s: 3.5.0 Assignee: Yang Jie Resolution: Fixed > Remove hadoop-2 profile

[jira] [Commented] (SPARK-30542) Two Spark structured streaming jobs cannot write to same base path

[ https://issues.apache.org/jira/browse/SPARK-30542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713623#comment-17713623 ] Wojciech Indyk commented on SPARK-30542: Will be fixed by this PR:

[jira] [Created] (SPARK-43176) Deduplicate imports in Connect Tests

Ruifeng Zheng created SPARK-43176: - Summary: Deduplicate imports in Connect Tests Key: SPARK-43176 URL: https://issues.apache.org/jira/browse/SPARK-43176 Project: Spark Issue Type: Test

[jira] [Updated] (SPARK-43175) decom.sh can cause an UnsupportedOperationException

[ https://issues.apache.org/jira/browse/SPARK-43175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iain Cardnell updated SPARK-43175: -- Description: decom.sh can cause an UnsupportedOperationException which then causes the

[jira] [Updated] (SPARK-43175) decom.sh can cause an UnsupportedOperationException

[ https://issues.apache.org/jira/browse/SPARK-43175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iain Cardnell updated SPARK-43175: -- Description: decom.sh can cause an UnsupportedOperationException which then causes the

[jira] [Updated] (SPARK-43175) decom.sh can cause an UnsupportedOperationException

[ https://issues.apache.org/jira/browse/SPARK-43175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iain Cardnell updated SPARK-43175: -- Description: decom.sh can cause an UnsupportedOperationException which then causes the

[jira] [Created] (SPARK-43175) decom.sh can cause an UnsupportedOperationException

Iain Cardnell created SPARK-43175: - Summary: decom.sh can cause an UnsupportedOperationException Key: SPARK-43175 URL: https://issues.apache.org/jira/browse/SPARK-43175 Project: Spark Issue

[jira] [Commented] (SPARK-42657) Support to find and transfer client-side REPL classfiles to server as artifacts

[ https://issues.apache.org/jira/browse/SPARK-42657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713583#comment-17713583 ] GridGain Integration commented on SPARK-42657: -- User 'vicennial' has created a pull request

[jira] [Resolved] (SPARK-43172) Expose host and bearer tokens from the spark connect client

[ https://issues.apache.org/jira/browse/SPARK-43172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43172. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40836

[jira] [Assigned] (SPARK-43172) Expose host and bearer tokens from the spark connect client

[ https://issues.apache.org/jira/browse/SPARK-43172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43172: Assignee: Niranjan Jayakar > Expose host and bearer tokens from the spark connect client

[jira] [Commented] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713576#comment-17713576 ] Yuming Wang commented on SPARK-43170: - Why it is {{CachedRDDBuilder}}? {noformat} (2)

[jira] [Commented] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713543#comment-17713543 ] todd commented on SPARK-43170: -- [~yumwang]  no cache > The spark sql like statement is pushed down to

[jira] [Created] (SPARK-43174) Fix SparkSQLCLIDriver completer

Yuming Wang created SPARK-43174: --- Summary: Fix SparkSQLCLIDriver completer Key: SPARK-43174 URL: https://issues.apache.org/jira/browse/SPARK-43174 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-43173) `write jdbc` in `ClientE2ETestSuite` will test fail with out `-Phive`

[ https://issues.apache.org/jira/browse/SPARK-43173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-43173: - Description: both ``` build/mvn clean install -Dtest=none

[jira] [Created] (SPARK-43173) `write jdbc` in `ClientE2ETestSuite` will test fail with out `-Phive`

Yang Jie created SPARK-43173: Summary: `write jdbc` in `ClientE2ETestSuite` will test fail with out `-Phive` Key: SPARK-43173 URL: https://issues.apache.org/jira/browse/SPARK-43173 Project: Spark

[jira] [Created] (SPARK-43172) Expose host and bearer tokens from the spark connect client

Niranjan Jayakar created SPARK-43172: Summary: Expose host and bearer tokens from the spark connect client Key: SPARK-43172 URL: https://issues.apache.org/jira/browse/SPARK-43172 Project: Spark

[jira] [Commented] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713503#comment-17713503 ] Yuming Wang commented on SPARK-43170: - Have you cached dwm_user_app_action_sum_all? > The spark sql

[jira] [Commented] (SPARK-42669) Short circuit local relation rpcs

[ https://issues.apache.org/jira/browse/SPARK-42669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713495#comment-17713495 ] ASF GitHub Bot commented on SPARK-42669: User 'beliefer' has created a pull request for this

[jira] [Commented] (SPARK-43137) Improve ArrayInsert if the position is foldable and equals to zero.

[ https://issues.apache.org/jira/browse/SPARK-43137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713493#comment-17713493 ] ASF GitHub Bot commented on SPARK-43137: User 'beliefer' has created a pull request for this

[jira] [Commented] (SPARK-42585) Streaming createDataFrame implementation

[ https://issues.apache.org/jira/browse/SPARK-42585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713494#comment-17713494 ] ASF GitHub Bot commented on SPARK-42585: User 'MaxGekk' has created a pull request for this

[jira] [Commented] (SPARK-43137) Improve ArrayInsert if the position is foldable and equals to zero.

[ https://issues.apache.org/jira/browse/SPARK-43137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713492#comment-17713492 ] ASF GitHub Bot commented on SPARK-43137: User 'beliefer' has created a pull request for this

[jira] [Commented] (SPARK-42552) Get ParseException when run sql: "SELECT 1 UNION SELECT 1;"

[ https://issues.apache.org/jira/browse/SPARK-42552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713489#comment-17713489 ] Ignite TC Bot commented on SPARK-42552: --- User 'Hisoka-X' has created a pull request for this

[jira] [Resolved] (SPARK-42151) Align UPDATE assignments with table attributes

[ https://issues.apache.org/jira/browse/SPARK-42151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-42151. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40308

[jira] [Assigned] (SPARK-42151) Align UPDATE assignments with table attributes

[ https://issues.apache.org/jira/browse/SPARK-42151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-42151: --- Assignee: Anton Okolnychyi > Align UPDATE assignments with table attributes >

[jira] [Resolved] (SPARK-43153) Skip Spark execution when the dataframe is local.

[ https://issues.apache.org/jira/browse/SPARK-43153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-43153. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40806

[jira] [Assigned] (SPARK-43153) Skip Spark execution when the dataframe is local.

[ https://issues.apache.org/jira/browse/SPARK-43153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-43153: - Assignee: Takuya Ueshin > Skip Spark execution when the dataframe is local. >

[jira] [Commented] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713412#comment-17713412 ] Hyukjin Kwon commented on SPARK-43170: -- There'd be no more releases in Spark 3.2.X unless there's

[jira] [Commented] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713403#comment-17713403 ] todd commented on SPARK-43170: -- Spark3.2.x is currently used in production, and there is no plan to upgrade