[jira] [Resolved] (SPARK-47875) Remove `spark.deploy.recoverySerializer`

2024-04-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47875. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46087 [https

[jira] [Created] (SPARK-47875) Remove `spark.deploy.recoverySerializer`

2024-04-16 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47875: - Summary: Remove `spark.deploy.recoverySerializer` Key: SPARK-47875 URL: https://issues.apache.org/jira/browse/SPARK-47875 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-47871) Oracle: Map TimestampType to TIMESTAMP WITH LOCAL TIME ZONE

2024-04-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47871. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46080 [https

[jira] [Assigned] (SPARK-47871) Oracle: Map TimestampType to TIMESTAMP WITH LOCAL TIME ZONE

2024-04-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47871: - Assignee: Kent Yao > Oracle: Map TimestampType to TIMESTAMP WITH LOCAL TIME Z

[jira] [Updated] (SPARK-47745) Add License to Spark Operator

2024-04-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47745: -- Affects Version/s: kubernetes-operator-0.1.0 (was: 4.0.0) >

[jira] [Updated] (SPARK-47745) Add License to Spark Operator

2024-04-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47745: -- Fix Version/s: kubernetes-operator-0.1.0 (was: 4.0.0) > Add Lice

[jira] [Resolved] (SPARK-47745) Add License to Spark Operator

2024-04-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47745. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 3 [https

[jira] [Assigned] (SPARK-47745) Add License to Spark Operator

2024-04-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47745: - Assignee: Zhou JIANG > Add License to Spark Opera

[jira] [Resolved] (SPARK-47739) Avro does not register custom logical types on spark startup

2024-04-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47739. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45895 [https

[jira] [Resolved] (SPARK-46574) Upgrade maven plugin to latest version

2024-04-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46574. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46043 [https

[jira] [Created] (SPARK-47865) Deflaky PythonForeachWriterSuite

2024-04-15 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47865: - Summary: Deflaky PythonForeachWriterSuite Key: SPARK-47865 URL: https://issues.apache.org/jira/browse/SPARK-47865 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-47866) Deflaky PythonForeachWriterSuite

2024-04-15 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47866: - Summary: Deflaky PythonForeachWriterSuite Key: SPARK-47866 URL: https://issues.apache.org/jira/browse/SPARK-47866 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-43394) Upgrade maven to 3.8.8

2024-04-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-43394: -- Fix Version/s: 3.4.4 > Upgrade maven to 3.

[jira] [Updated] (SPARK-47828) DataFrameWriterV2.overwrite fails with invalid plan

2024-04-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47828: -- Fix Version/s: 3.5.2 > DataFrameWriterV2.overwrite fails with invalid p

[jira] [Assigned] (SPARK-47855) Warn `spark.sql.execution.arrow.pyspark.fallback.enabled` in Connect

2024-04-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47855: - Assignee: Ruifeng Zheng > Warn `spark.sql.execution.arrow.pyspark.fallback.enab

[jira] [Resolved] (SPARK-47855) Warn `spark.sql.execution.arrow.pyspark.fallback.enabled` in Connect

2024-04-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47855. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46056 [https

[jira] [Resolved] (SPARK-47860) Upgrade `kubernetes-client` to 6.12.0

2024-04-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47860. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46066 [https

[jira] [Created] (SPARK-47860) Upgrade `kubernetes-client` to 6.12.0

2024-04-15 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47860: - Summary: Upgrade `kubernetes-client` to 6.12.0 Key: SPARK-47860 URL: https://issues.apache.org/jira/browse/SPARK-47860 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-47856) Oracle: Document Mapping Spark SQL Data Types from Oracle

2024-04-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47856. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46059 [https

[jira] [Assigned] (SPARK-47856) Oracle: Document Mapping Spark SQL Data Types from Oracle

2024-04-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47856: - Assignee: Kent Yao > Oracle: Document Mapping Spark SQL Data Types from Ora

[jira] [Updated] (SPARK-47856) Oracle: Document Mapping Spark SQL Data Types from Oracle and add tests

2024-04-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47856: -- Summary: Oracle: Document Mapping Spark SQL Data Types from Oracle and add tests

Re: [VOTE] Release Spark 3.4.3 (RC2)

2024-04-14 Thread Dongjoon Hyun
I'll start with my +1. - Checked checksum and signature - Checked Scala/Java/R/Python/SQL Document's Spark version - Checked published Maven artifacts - All CIs passed. Thanks, Dongjoon. On 2024/04/15 04:22:26 Dongjoon Hyun wrote: > Please vote on releasing the following candidate as Apa

[VOTE] Release Spark 3.4.3 (RC2)

2024-04-14 Thread Dongjoon Hyun
Please vote on releasing the following candidate as Apache Spark version 3.4.3. The vote is open until April 18th 1AM (PDT) and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.4.3 [ ] -1 Do not release this package because

[jira] [Updated] (SPARK-47694) Make max message size configurable on client side

2024-04-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47694: -- Fix Version/s: (was: 3.4.3) > Make max message size configurable on client s

[jira] [Updated] (SPARK-47694) Make max message size configurable on client side

2024-04-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47694: -- Affects Version/s: 4.0.0 (was: 3.4.1) > Make max message s

[ANNOUNCE] Announcing Apache ORC 1.8.7

2024-04-14 Thread Dongjoon Hyun
Hi All. We are happy to announce the availability of Apache ORC 1.8.7! https://orc.apache.org/news/2024/04/14/ORC-1.8.7/ 1.8.7 is a maintenance release containing important fixes. It's available in Apache Downloads and Maven Central. https://downloads.apache.org/orc/orc-1.8.7/

[ANNOUNCE] Announcing Apache ORC 1.8.7

2024-04-14 Thread Dongjoon Hyun
Hi All. We are happy to announce the availability of Apache ORC 1.8.7! https://orc.apache.org/news/2024/04/14/ORC-1.8.7/ 1.8.7 is a maintenance release containing important fixes. It's available in Apache Downloads and Maven Central. https://downloads.apache.org/orc/orc-1.8.7/

[VOTE][RESULT] Release Apache ORC 1.8.7 (RC0)

2024-04-14 Thread Dongjoon Hyun
The vote passed. Thanks to all who helped with the release! (* = binding) +1: - Dongjoon Hyun * - Gang Wu * - Shaoyun Chen - William Hyun * +0: None -1: None

Re: [VOTE] Release Apache ORC 1.8.7 (RC0)

2024-04-14 Thread Dongjoon Hyun
Thank you all. This vote passed. Dongjoon. On 2024/04/13 21:56:05 Dongjoon Hyun wrote: > Thank you, all. > > The vote will end on April 14th 1AM (PST). > > Dongjoon. > > On 2024/04/11 18:27:18 "William H." wrote: > > +1 > > > > Thank you

[jira] [Resolved] (SPARK-47841) Upgrade `postgresql` to 42.7.3

2024-04-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47841. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46038 [https

[jira] [Assigned] (ORC-1669) [C++] Deprecate HDFS support

2024-04-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1669: -- Assignee: Gang Wu (was: Yuanping Wu) > [C++] Deprecate HDFS supp

[jira] [Assigned] (ORC-1669) [C++] Deprecate HDFS support

2024-04-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1669: -- Assignee: Yuanping Wu > [C++] Deprecate HDFS supp

[jira] [Resolved] (ORC-1669) [C++] Deprecate HDFS support

2024-04-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1669. Fix Version/s: 2.0.1 2.1.0 Resolution: Fixed Issue resolved by pull

[jira] [Updated] (SPARK-47318) AuthEngine key exchange needs additional KDF round

2024-04-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47318: -- Fix Version/s: 3.4.3 > AuthEngine key exchange needs additional KDF ro

[jira] [Updated] (SPARK-47318) AuthEngine key exchange needs additional KDF round

2024-04-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47318: -- Fix Version/s: 3.5.2 > AuthEngine key exchange needs additional KDF ro

[jira] [Commented] (SPARK-44444) Use ANSI mode by default

2024-04-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836869#comment-17836869 ] Dongjoon Hyun commented on SPARK-4: --- Thank you all. The vote is started here. - https

Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

2024-04-13 Thread Dongjoon Hyun
I'll start from my +1. Dongjoon. On 2024/04/13 22:22:05 Dongjoon Hyun wrote: > Please vote on SPARK-4 to use ANSI SQL mode by default. > The technical scope is defined in the following PR which is > one line of code change and one line of migration guide. > > - DISC

[jira] [Updated] (SPARK-44444) Use ANSI mode by default

2024-04-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-4: -- Summary: Use ANSI mode by default (was: Enabled ANSI mode by default) > Use ANSI m

[VOTE] SPARK-44444: Use ANSI SQL mode by default

2024-04-13 Thread Dongjoon Hyun
Please vote on SPARK-4 to use ANSI SQL mode by default. The technical scope is defined in the following PR which is one line of code change and one line of migration guide. - DISCUSSION: https://lists.apache.org/thread/ztlwoz1v1sn81ssks12tb19x37zozxlz - JIRA:

Re: [DISCUSS] SPARK-44444: Use ANSI SQL mode by default

2024-04-13 Thread Dongjoon Hyun
ror Attribution Framework > > <https://issues.apache.org/jira/browse/SPARK-38615> will also be beneficial > > in migrating to ANSI SQL mode. > > > > > > Gengliang > > > > > > On Thu, Apr 11, 2024 at 7:56 PM Dongjoon Hyun > <mailto:dongjoon.h

Re: [VOTE] Release Apache ORC 1.8.7 (RC0)

2024-04-13 Thread Dongjoon Hyun
-test > > > 7/8 Test #7: java-bench-spark-test Passed 215.06 sec > > > Start 8: tool-test > > > 8/8 Test #8: tool-test Passed 17.01 sec > > > > > > 100% tests passed, 0 tests failed out of 8 &

[jira] [Commented] (SPARK-46122) Disable spark.sql.legacy.createHiveTableByDefault by default

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836708#comment-17836708 ] Dongjoon Hyun commented on SPARK-46122: --- Gentle ping, [~yumwang] . You can throw a discussion

Re: [DISCUSS] Spark 4.0.0 release

2024-04-12 Thread Dongjoon Hyun
Thank you for volunteering, Wenchen. Dongjoon. On 2024/04/12 15:11:04 Wenchen Fan wrote: > Hi all, > > It's close to the previously proposed 4.0.0 release date (June 2024), and I > think it's time to prepare for it and discuss the ongoing projects: > >- ANSI by default >- Spark Connect

[jira] [Updated] (SPARK-44518) Completely make hive as a data source

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44518: -- Parent: (was: SPARK-44111) Issue Type: Improvement (was: Sub-task) > Complet

[jira] [Updated] (SPARK-44518) Completely make hive as a data source

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44518: -- Target Version/s: (was: 4.0.0) > Completely make hive as a data sou

[jira] [Updated] (SPARK-44518) Completely make hive as a data source

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44518: -- Fix Version/s: (was: 4.0.0) > Completely make hive as a data sou

[jira] [Commented] (SPARK-44518) Completely make hive as a data source

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836704#comment-17836704 ] Dongjoon Hyun commented on SPARK-44518: --- In addition, I'll convert this to an independent JIRA

[jira] [Commented] (SPARK-44518) Completely make hive as a data source

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836703#comment-17836703 ] Dongjoon Hyun commented on SPARK-44518: --- Hi, [~roryqi], this looks like a question instead of any

[jira] [Updated] (SPARK-44116) Utilize Hadoop vectorized APIs

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44116: -- Parent: (was: SPARK-44111) Issue Type: New Feature (was: Sub-task) > Util

[jira] [Commented] (HADOOP-18296) Memory fragmentation in ChecksumFileSystem Vectored IO implementation.

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836697#comment-17836697 ] Dongjoon Hyun commented on HADOOP-18296: Thank you for sharing the direction. > Mem

[jira] [Assigned] (SPARK-44444) Enabled ANSI mode by default

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-4: - Assignee: Dongjoon Hyun > Enabled ANSI mode by defa

Re: [VOTE] Add new `Versions` in Apache Spark JIRA for Versioning of Spark Operator

2024-04-12 Thread Dongjoon Hyun
+1 Thank you! I hope we can customize `dev/merge_spark_pr.py` script per repository after this PR. Dongjoon. On 2024/04/12 03:28:36 "L. C. Hsieh" wrote: > Hi all, > > Thanks for all discussions in the thread of "Versioning of Spark > Operator":

[jira] [Commented] (SPARK-44444) Enabled ANSI mode by default

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836397#comment-17836397 ] Dongjoon Hyun commented on SPARK-4: --- Hi, All. Here is the discussion thread I sent a few

[DISCUSS] SPARK-44444: Use ANSI SQL mode by default

2024-04-11 Thread Dongjoon Hyun
Hi, All. Thanks to you, we've been achieving many things and have on-going SPIPs. I believe it's time to scope Apache Spark 4.0.0 (SPARK-44111) more narrowly by asking your opinions about Apache Spark's ANSI SQL mode. https://issues.apache.org/jira/browse/SPARK-44111 Prepare Apache Spark

[jira] [Commented] (SPARK-44444) Enabled ANSI mode by default

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836389#comment-17836389 ] Dongjoon Hyun commented on SPARK-4: --- I made a draft PR and will initiate the discussion thread

[jira] [Assigned] (SPARK-47814) Remove the `KinesisTestUtils` from `main` to `test`

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47814: - Assignee: BingKun Pan > Remove the `KinesisTestUtils` from `main` to `t

[jira] [Resolved] (SPARK-47814) Remove the `KinesisTestUtils` from `main` to `test`

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47814. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46000 [https

[jira] [Updated] (SPARK-47777) Add spark connect test for python streaming data source

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-4: -- Component/s: Tests > Add spark connect test for python streaming data sou

[jira] [Assigned] (SPARK-47777) Add spark connect test for python streaming data source

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-4: - Assignee: Chaoqin Li > Add spark connect test for python streaming data sou

[jira] [Resolved] (SPARK-47777) Add spark connect test for python streaming data source

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-4. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45950 [https

[jira] [Resolved] (SPARK-47820) Run `ANSI` SQL CI twice per day

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47820. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46010 [https

[jira] [Updated] (SPARK-47820) Run `ANSI` SQL CI twice per day

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47820: -- Summary: Run `ANSI` SQL CI twice per day (was: Run `ANSI` SQL Daily CI twice) > Run `A

[jira] [Created] (SPARK-47820) Run `ANSI` SQL Daily CI twice

2024-04-11 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47820: - Summary: Run `ANSI` SQL Daily CI twice Key: SPARK-47820 URL: https://issues.apache.org/jira/browse/SPARK-47820 Project: Spark Issue Type: Test

[jira] [Updated] (SPARK-47820) Run `ANSI` SQL Daily CI twice

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47820: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Test) > Run `ANSI` SQL Daily CI tw

[jira] [Updated] (SPARK-47799) Preserve parameter information when using SBT package jar

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47799: -- Affects Version/s: 4.0.0 (was: 3.5.1) > Preserve parame

[jira] [Resolved] (SPARK-47799) Preserve parameter information when using SBT package jar

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47799. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45983 [https

[jira] [Assigned] (SPARK-47799) Preserve parameter information when using SBT package jar

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47799: - Assignee: dzcxzl > Preserve parameter information when using SBT package

[jira] [Updated] (SPARK-47797) Skip deleting pod from k8s if the pod does not exists

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47797: -- Affects Version/s: 4.0.0 (was: 3.5.1) > Skip deleting pod from

[jira] [Resolved] (SPARK-47797) Skip deleting pod from k8s if the pod does not exists

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47797. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45979 [https

[jira] [Assigned] (SPARK-47797) Skip deleting pod from k8s if the pod does not exists

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47797: - Assignee: leesf > Skip deleting pod from k8s if the pod does not exi

[jira] [Assigned] (SPARK-47817) Update pandas to 2.2.2

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47817: - Assignee: Bjørn Jørgensen > Update pandas to 2.

[jira] [Resolved] (SPARK-47817) Update pandas to 2.2.2

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47817. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46009 [https

[jira] [Updated] (SPARK-46812) Make `mapInPandas` / mapInArrow` support ResourceProfile (Stage-Level scheduling)

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46812: -- Affects Version/s: (was: 3.5.2) > Make `mapInPandas` / mapInArrow` supp

[jira] [Commented] (HADOOP-18296) Memory fragmentation in ChecksumFileSystem Vectored IO implementation.

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836017#comment-17836017 ] Dongjoon Hyun commented on HADOOP-18296: According to the description, is this a memory leak

Re: Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine

2024-04-10 Thread Dongjoon Hyun
ten v1.1.1 support Spark3.2 and > 3.3. > > We are planning to support Spark3.4 and 3.5 in Gluten v1.2.0. > > Spark4.0 support for Gluten is depending on the release schedule in > Spark community. > > > > On 2024/04/09 07:14:13 Dongjoon Hyun wrote: > > > Thank you for s

[jira] [Commented] (ORC-642) Update PatchedBase doc with patch ceiling in spec

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835971#comment-17835971 ] Dongjoon Hyun commented on ORC-642: --- I added you to the higher group. Please let me know if you hit

[jira] [Commented] (ORC-642) Update PatchedBase doc with patch ceiling in spec

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835970#comment-17835970 ] Dongjoon Hyun commented on ORC-642: --- I checked the JIRA group . It seems that `Committer` group has

[jira] [Updated] (ORC-642) Update PatchedBase doc with patch ceiling in spec

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated ORC-642: -- Attachment: Screenshot 2024-04-10 at 20.24.43.png > Update PatchedBase doc with patch ceiling in s

[jira] [Commented] (ORC-642) Update PatchedBase doc with patch ceiling in spec

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835969#comment-17835969 ] Dongjoon Hyun commented on ORC-642: --- Oh, got it. > Update PatchedBase doc with patch ceiling in s

Re: [VOTE] Release Apache ORC 1.8.7 (RC0)

2024-04-10 Thread Dongjoon Hyun
1.3.x branch Dongjoon. On 2024/04/10 18:49:11 Dongjoon Hyun wrote: > Please vote on releasing the following candidate as Apache ORC version > 1.8.7. This vote is open until April 14th 1AM (PST) and passes if a > majority +1 PMC votes are cast, with a minimum of 3 +1 votes. > >

[VOTE] Release Apache ORC 1.8.7 (RC0)

2024-04-10 Thread Dongjoon Hyun
Please vote on releasing the following candidate as Apache ORC version 1.8.7. This vote is open until April 14th 1AM (PST) and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache ORC 1.8.7 [ ] -1 Do not release this package because ...

[jira] [Resolved] (ORC-1686) [C++] Avoid using std::filesystem

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1686. Fix Version/s: 2.0.1 2.1.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (ORC-1685) Use `Pattern Matching for instanceof` in RecordReaderImpl

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1685: -- Assignee: dzcxzl > Use `Pattern Matching for instanceof` in RecordReaderI

[jira] [Resolved] (ORC-1685) Use `Pattern Matching for instanceof` in RecordReaderImpl

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1685. Fix Version/s: 2.1.0 Resolution: Fixed Issue resolved by pull request 1884 [https

[jira] (SPARK-44478) Executor decommission causes stage failure

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44478 ] Dongjoon Hyun deleted comment on SPARK-44478: --- was (Author: dongjoon): Ya, it's difficult to reproduce. BTW, could you try 3.4.2, [~juhai], because SPARK-44630 is in Apache Spark 3.4.2

[jira] [Commented] (SPARK-44478) Executor decommission causes stage failure

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835831#comment-17835831 ] Dongjoon Hyun commented on SPARK-44478: --- Ya, it's difficult to reproduce. BTW, could you try 3.4.2

[jira] [Updated] (ORC-642) Update PatchedBase doc with patch ceiling in spec

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated ORC-642: -- Summary: Update PatchedBase doc with patch ceiling in spec (was: ORC specification for RLEv2

[jira] [Updated] (ORC-642) ORC specification for RLEv2 is incomplete

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated ORC-642: -- Component/s: documentation > ORC specification for RLEv2 is incompl

[jira] [Resolved] (ORC-642) ORC specification for RLEv2 is incomplete

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-642. --- Fix Version/s: 2.1.0 Resolution: Fixed > ORC specification for RLEv2 is incompl

Re: ORC 1.8.7 Release

2024-04-10 Thread Dongjoon Hyun
Oh, thank you, William. Let me volunteer and start it now. I'm the release manager of Apache Spark 3.4.3 (for next Monday). I guess I can include Apache ORC 1.8.7 there too. Thanks, Dongjoon. On 2024/04/10 01:48:35 "William H." wrote: > Hey All, > > I hope everyone had a great weekend and

[jira] [Comment Edited] (ORC-642) ORC specification for RLEv2 is incomplete

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835810#comment-17835810 ] Dongjoon Hyun edited comment on ORC-642 at 4/10/24 4:18 PM: I manually added

[jira] [Commented] (ORC-642) ORC specification for RLEv2 is incomplete

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835810#comment-17835810 ] Dongjoon Hyun commented on ORC-642: --- I manually added and assigned him. > ORC specification for RL

[jira] [Commented] (ORC-642) ORC specification for RLEv2 is incomplete

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835808#comment-17835808 ] Dongjoon Hyun commented on ORC-642: --- Did you try to use the official merge script, [~deshanxiao

[jira] [Assigned] (ORC-642) ORC specification for RLEv2 is incomplete

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-642: - Assignee: Jeffrey Vo > ORC specification for RLEv2 is incompl

[jira] [Updated] (SPARK-47790) Upgrade `commons-io` to 2.16.1

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47790: -- Fix Version/s: 3.5.2 > Upgrade `commons-io` to 2.1

[jira] [Resolved] (SPARK-47790) Upgrade `commons-io` to 2.16.1

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47790. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45973 [https

Re: Versioning of Spark Operator

2024-04-10 Thread Dongjoon Hyun
Ya, that would work. Inevitably, I looked at Apache Flink K8s Operator's JIRA and GitHub repo. It looks reasonable to me. Although they share the same JIRA, they choose different patterns per place. 1. In POM file and Maven Artifact, independent version number. 1.8.0 2. Tag is also based on

[jira] [Resolved] (SPARK-47706) Bump json4s 4.0.7

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47706. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45838 [https

[jira] [Updated] (SPARK-45444) Upgrade `commons-io` to 2.14.0

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45444: -- Summary: Upgrade `commons-io` to 2.14.0 (was: Upgrade `commons-io` to 1.24.0) > Upgr

[jira] [Updated] (SPARK-45444) Upgrade `commons-io` to 1.24.0

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45444: -- Parent: SPARK-47046 Issue Type: Sub-task (was: Improvement) > Upgrade `commons

<    1   2   3   4   5   6   7   8   9   10   >