[jira] [Created] (ORC-1691) Bump `spotbugs-maven-plugin` to 4.8.4.0

2024-04-17 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created ORC-1691: -- Summary: Bump `spotbugs-maven-plugin` to 4.8.4.0 Key: ORC-1691 URL: https://issues.apache.org/jira/browse/ORC-1691 Project: ORC Issue Type: Test

[jira] [Resolved] (ORC-1691) Bump `spotbugs-maven-plugin` to 4.8.4.0

2024-04-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1691. Fix Version/s: 2.1.0 Resolution: Fixed Issue resolved by pull request 1892 [https

[jira] [Resolved] (SPARK-47886) Postgres: Add test and doc for Postgres special numeric values

2024-04-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47886. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46102 [https

[jira] [Updated] (SPARK-47884) Switch ANSI SQL CI job to NON-ANSI SQL CI job

2024-04-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47884: -- Summary: Switch ANSI SQL CI job to NON-ANSI SQL CI job (was: Switch ANSI SQL CI to NON-ANSI

[jira] [Assigned] (SPARK-47884) Switch ANSI SQL CI job to NON-ANSI SQL CI job

2024-04-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47884: - Assignee: Dongjoon Hyun > Switch ANSI SQL CI job to NON-ANSI SQL CI

[jira] [Created] (SPARK-47884) Switch ANSI SQL CI to NON-ANSI SQL CI

2024-04-17 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47884: - Summary: Switch ANSI SQL CI to NON-ANSI SQL CI Key: SPARK-47884 URL: https://issues.apache.org/jira/browse/SPARK-47884 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-44444) Use ANSI mode by default

2024-04-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-4. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46013 [https

[jira] [Updated] (SPARK-44444) Use ANSI SQL mode by default

2024-04-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-4: -- Summary: Use ANSI SQL mode by default (was: Use ANSI mode by default) > Use ANSI SQL m

[VOTE][RESULT] SPARK-44444: Use ANSI SQL mode by default

2024-04-17 Thread Dongjoon Hyun
The vote passes with 24 +1s (13 binding +1s). Thanks to all who helped with the vote! (* = binding) +1: - Dongjoon Hyun * - Gengliang Wang * - Chao Sun * - Hyukjin Kwon * - Liang-Chi Hsieh * - Holden Karau * - Huaxin Gao * - Denny Lee - Xiao Li * - Mich Talebzadeh - Christiano Anderson - Yang Jie

Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

2024-04-17 Thread Dongjoon Hyun
t; Thanks Dongjoon to drive this! > >> > >> > >> -Rui > >> > >> On Mon, Apr 15, 2024 at 10:10 AM Xinrong Meng wrote: > >> > >>> +1 > >>> > >>> Thank you @Dongjoon Hyun ! > >>> > >>> On

[jira] [Resolved] (SPARK-47875) Remove `spark.deploy.recoverySerializer`

2024-04-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47875. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46087 [https

[jira] [Created] (SPARK-47875) Remove `spark.deploy.recoverySerializer`

2024-04-16 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47875: - Summary: Remove `spark.deploy.recoverySerializer` Key: SPARK-47875 URL: https://issues.apache.org/jira/browse/SPARK-47875 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-47871) Oracle: Map TimestampType to TIMESTAMP WITH LOCAL TIME ZONE

2024-04-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47871. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46080 [https

[jira] [Assigned] (SPARK-47871) Oracle: Map TimestampType to TIMESTAMP WITH LOCAL TIME ZONE

2024-04-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47871: - Assignee: Kent Yao > Oracle: Map TimestampType to TIMESTAMP WITH LOCAL TIME Z

[jira] [Updated] (SPARK-47745) Add License to Spark Operator

2024-04-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47745: -- Affects Version/s: kubernetes-operator-0.1.0 (was: 4.0.0) >

[jira] [Updated] (SPARK-47745) Add License to Spark Operator

2024-04-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47745: -- Fix Version/s: kubernetes-operator-0.1.0 (was: 4.0.0) > Add Lice

[jira] [Resolved] (SPARK-47745) Add License to Spark Operator

2024-04-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47745. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 3 [https

[jira] [Assigned] (SPARK-47745) Add License to Spark Operator

2024-04-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47745: - Assignee: Zhou JIANG > Add License to Spark Opera

[jira] [Resolved] (SPARK-47739) Avro does not register custom logical types on spark startup

2024-04-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47739. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45895 [https

[jira] [Resolved] (SPARK-46574) Upgrade maven plugin to latest version

2024-04-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46574. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46043 [https

[jira] [Created] (SPARK-47865) Deflaky PythonForeachWriterSuite

2024-04-15 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47865: - Summary: Deflaky PythonForeachWriterSuite Key: SPARK-47865 URL: https://issues.apache.org/jira/browse/SPARK-47865 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-47866) Deflaky PythonForeachWriterSuite

2024-04-15 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47866: - Summary: Deflaky PythonForeachWriterSuite Key: SPARK-47866 URL: https://issues.apache.org/jira/browse/SPARK-47866 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-43394) Upgrade maven to 3.8.8

2024-04-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-43394: -- Fix Version/s: 3.4.4 > Upgrade maven to 3.

[jira] [Updated] (SPARK-47828) DataFrameWriterV2.overwrite fails with invalid plan

2024-04-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47828: -- Fix Version/s: 3.5.2 > DataFrameWriterV2.overwrite fails with invalid p

[jira] [Assigned] (SPARK-47855) Warn `spark.sql.execution.arrow.pyspark.fallback.enabled` in Connect

2024-04-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47855: - Assignee: Ruifeng Zheng > Warn `spark.sql.execution.arrow.pyspark.fallback.enab

[jira] [Resolved] (SPARK-47855) Warn `spark.sql.execution.arrow.pyspark.fallback.enabled` in Connect

2024-04-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47855. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46056 [https

[jira] [Resolved] (SPARK-47860) Upgrade `kubernetes-client` to 6.12.0

2024-04-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47860. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46066 [https

[jira] [Created] (SPARK-47860) Upgrade `kubernetes-client` to 6.12.0

2024-04-15 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47860: - Summary: Upgrade `kubernetes-client` to 6.12.0 Key: SPARK-47860 URL: https://issues.apache.org/jira/browse/SPARK-47860 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-47856) Oracle: Document Mapping Spark SQL Data Types from Oracle

2024-04-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47856. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46059 [https

[jira] [Assigned] (SPARK-47856) Oracle: Document Mapping Spark SQL Data Types from Oracle

2024-04-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47856: - Assignee: Kent Yao > Oracle: Document Mapping Spark SQL Data Types from Ora

[jira] [Updated] (SPARK-47856) Oracle: Document Mapping Spark SQL Data Types from Oracle and add tests

2024-04-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47856: -- Summary: Oracle: Document Mapping Spark SQL Data Types from Oracle and add tests

Re: [VOTE] Release Spark 3.4.3 (RC2)

2024-04-14 Thread Dongjoon Hyun
I'll start with my +1. - Checked checksum and signature - Checked Scala/Java/R/Python/SQL Document's Spark version - Checked published Maven artifacts - All CIs passed. Thanks, Dongjoon. On 2024/04/15 04:22:26 Dongjoon Hyun wrote: > Please vote on releasing the following candidate as Apa

[VOTE] Release Spark 3.4.3 (RC2)

2024-04-14 Thread Dongjoon Hyun
Please vote on releasing the following candidate as Apache Spark version 3.4.3. The vote is open until April 18th 1AM (PDT) and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.4.3 [ ] -1 Do not release this package because

[jira] [Updated] (SPARK-47694) Make max message size configurable on client side

2024-04-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47694: -- Fix Version/s: (was: 3.4.3) > Make max message size configurable on client s

[jira] [Updated] (SPARK-47694) Make max message size configurable on client side

2024-04-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47694: -- Affects Version/s: 4.0.0 (was: 3.4.1) > Make max message s

[ANNOUNCE] Announcing Apache ORC 1.8.7

2024-04-14 Thread Dongjoon Hyun
Hi All. We are happy to announce the availability of Apache ORC 1.8.7! https://orc.apache.org/news/2024/04/14/ORC-1.8.7/ 1.8.7 is a maintenance release containing important fixes. It's available in Apache Downloads and Maven Central. https://downloads.apache.org/orc/orc-1.8.7/

[ANNOUNCE] Announcing Apache ORC 1.8.7

2024-04-14 Thread Dongjoon Hyun
Hi All. We are happy to announce the availability of Apache ORC 1.8.7! https://orc.apache.org/news/2024/04/14/ORC-1.8.7/ 1.8.7 is a maintenance release containing important fixes. It's available in Apache Downloads and Maven Central. https://downloads.apache.org/orc/orc-1.8.7/

[VOTE][RESULT] Release Apache ORC 1.8.7 (RC0)

2024-04-14 Thread Dongjoon Hyun
The vote passed. Thanks to all who helped with the release! (* = binding) +1: - Dongjoon Hyun * - Gang Wu * - Shaoyun Chen - William Hyun * +0: None -1: None

Re: [VOTE] Release Apache ORC 1.8.7 (RC0)

2024-04-14 Thread Dongjoon Hyun
Thank you all. This vote passed. Dongjoon. On 2024/04/13 21:56:05 Dongjoon Hyun wrote: > Thank you, all. > > The vote will end on April 14th 1AM (PST). > > Dongjoon. > > On 2024/04/11 18:27:18 "William H." wrote: > > +1 > > > > Thank you

[jira] [Resolved] (SPARK-47841) Upgrade `postgresql` to 42.7.3

2024-04-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47841. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46038 [https

[jira] [Assigned] (ORC-1669) [C++] Deprecate HDFS support

2024-04-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1669: -- Assignee: Gang Wu (was: Yuanping Wu) > [C++] Deprecate HDFS supp

[jira] [Assigned] (ORC-1669) [C++] Deprecate HDFS support

2024-04-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1669: -- Assignee: Yuanping Wu > [C++] Deprecate HDFS supp

[jira] [Resolved] (ORC-1669) [C++] Deprecate HDFS support

2024-04-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1669. Fix Version/s: 2.0.1 2.1.0 Resolution: Fixed Issue resolved by pull

[jira] [Updated] (SPARK-47318) AuthEngine key exchange needs additional KDF round

2024-04-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47318: -- Fix Version/s: 3.4.3 > AuthEngine key exchange needs additional KDF ro

[jira] [Updated] (SPARK-47318) AuthEngine key exchange needs additional KDF round

2024-04-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47318: -- Fix Version/s: 3.5.2 > AuthEngine key exchange needs additional KDF ro

[jira] [Commented] (SPARK-44444) Use ANSI mode by default

2024-04-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836869#comment-17836869 ] Dongjoon Hyun commented on SPARK-4: --- Thank you all. The vote is started here. - https

Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

2024-04-13 Thread Dongjoon Hyun
I'll start from my +1. Dongjoon. On 2024/04/13 22:22:05 Dongjoon Hyun wrote: > Please vote on SPARK-4 to use ANSI SQL mode by default. > The technical scope is defined in the following PR which is > one line of code change and one line of migration guide. > > - DISC

[jira] [Updated] (SPARK-44444) Use ANSI mode by default

2024-04-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-4: -- Summary: Use ANSI mode by default (was: Enabled ANSI mode by default) > Use ANSI m

[VOTE] SPARK-44444: Use ANSI SQL mode by default

2024-04-13 Thread Dongjoon Hyun
Please vote on SPARK-4 to use ANSI SQL mode by default. The technical scope is defined in the following PR which is one line of code change and one line of migration guide. - DISCUSSION: https://lists.apache.org/thread/ztlwoz1v1sn81ssks12tb19x37zozxlz - JIRA:

Re: [DISCUSS] SPARK-44444: Use ANSI SQL mode by default

2024-04-13 Thread Dongjoon Hyun
ror Attribution Framework > > <https://issues.apache.org/jira/browse/SPARK-38615> will also be beneficial > > in migrating to ANSI SQL mode. > > > > > > Gengliang > > > > > > On Thu, Apr 11, 2024 at 7:56 PM Dongjoon Hyun > <mailto:dongjoon.h

Re: [VOTE] Release Apache ORC 1.8.7 (RC0)

2024-04-13 Thread Dongjoon Hyun
-test > > > 7/8 Test #7: java-bench-spark-test Passed 215.06 sec > > > Start 8: tool-test > > > 8/8 Test #8: tool-test Passed 17.01 sec > > > > > > 100% tests passed, 0 tests failed out of 8 &

[jira] [Commented] (SPARK-46122) Disable spark.sql.legacy.createHiveTableByDefault by default

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836708#comment-17836708 ] Dongjoon Hyun commented on SPARK-46122: --- Gentle ping, [~yumwang] . You can throw a discussion

Re: [DISCUSS] Spark 4.0.0 release

2024-04-12 Thread Dongjoon Hyun
Thank you for volunteering, Wenchen. Dongjoon. On 2024/04/12 15:11:04 Wenchen Fan wrote: > Hi all, > > It's close to the previously proposed 4.0.0 release date (June 2024), and I > think it's time to prepare for it and discuss the ongoing projects: > >- ANSI by default >- Spark Connect

[jira] [Updated] (SPARK-44518) Completely make hive as a data source

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44518: -- Parent: (was: SPARK-44111) Issue Type: Improvement (was: Sub-task) > Complet

[jira] [Updated] (SPARK-44518) Completely make hive as a data source

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44518: -- Target Version/s: (was: 4.0.0) > Completely make hive as a data sou

[jira] [Updated] (SPARK-44518) Completely make hive as a data source

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44518: -- Fix Version/s: (was: 4.0.0) > Completely make hive as a data sou

[jira] [Commented] (SPARK-44518) Completely make hive as a data source

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836704#comment-17836704 ] Dongjoon Hyun commented on SPARK-44518: --- In addition, I'll convert this to an independent JIRA

[jira] [Commented] (SPARK-44518) Completely make hive as a data source

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836703#comment-17836703 ] Dongjoon Hyun commented on SPARK-44518: --- Hi, [~roryqi], this looks like a question instead of any

[jira] [Updated] (SPARK-44116) Utilize Hadoop vectorized APIs

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44116: -- Parent: (was: SPARK-44111) Issue Type: New Feature (was: Sub-task) > Util

[jira] [Commented] (HADOOP-18296) Memory fragmentation in ChecksumFileSystem Vectored IO implementation.

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836697#comment-17836697 ] Dongjoon Hyun commented on HADOOP-18296: Thank you for sharing the direction. > Mem

[jira] [Assigned] (SPARK-44444) Enabled ANSI mode by default

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-4: - Assignee: Dongjoon Hyun > Enabled ANSI mode by defa

Re: [VOTE] Add new `Versions` in Apache Spark JIRA for Versioning of Spark Operator

2024-04-12 Thread Dongjoon Hyun
+1 Thank you! I hope we can customize `dev/merge_spark_pr.py` script per repository after this PR. Dongjoon. On 2024/04/12 03:28:36 "L. C. Hsieh" wrote: > Hi all, > > Thanks for all discussions in the thread of "Versioning of Spark > Operator":

[jira] [Commented] (SPARK-44444) Enabled ANSI mode by default

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836397#comment-17836397 ] Dongjoon Hyun commented on SPARK-4: --- Hi, All. Here is the discussion thread I sent a few

[DISCUSS] SPARK-44444: Use ANSI SQL mode by default

2024-04-11 Thread Dongjoon Hyun
Hi, All. Thanks to you, we've been achieving many things and have on-going SPIPs. I believe it's time to scope Apache Spark 4.0.0 (SPARK-44111) more narrowly by asking your opinions about Apache Spark's ANSI SQL mode. https://issues.apache.org/jira/browse/SPARK-44111 Prepare Apache Spark

[jira] [Commented] (SPARK-44444) Enabled ANSI mode by default

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836389#comment-17836389 ] Dongjoon Hyun commented on SPARK-4: --- I made a draft PR and will initiate the discussion thread

[jira] [Assigned] (SPARK-47814) Remove the `KinesisTestUtils` from `main` to `test`

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47814: - Assignee: BingKun Pan > Remove the `KinesisTestUtils` from `main` to `t

[jira] [Resolved] (SPARK-47814) Remove the `KinesisTestUtils` from `main` to `test`

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47814. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46000 [https

[jira] [Updated] (SPARK-47777) Add spark connect test for python streaming data source

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-4: -- Component/s: Tests > Add spark connect test for python streaming data sou

[jira] [Assigned] (SPARK-47777) Add spark connect test for python streaming data source

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-4: - Assignee: Chaoqin Li > Add spark connect test for python streaming data sou

[jira] [Resolved] (SPARK-47777) Add spark connect test for python streaming data source

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-4. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45950 [https

[jira] [Resolved] (SPARK-47820) Run `ANSI` SQL CI twice per day

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47820. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46010 [https

[jira] [Updated] (SPARK-47820) Run `ANSI` SQL CI twice per day

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47820: -- Summary: Run `ANSI` SQL CI twice per day (was: Run `ANSI` SQL Daily CI twice) > Run `A

[jira] [Created] (SPARK-47820) Run `ANSI` SQL Daily CI twice

2024-04-11 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47820: - Summary: Run `ANSI` SQL Daily CI twice Key: SPARK-47820 URL: https://issues.apache.org/jira/browse/SPARK-47820 Project: Spark Issue Type: Test

[jira] [Updated] (SPARK-47820) Run `ANSI` SQL Daily CI twice

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47820: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Test) > Run `ANSI` SQL Daily CI tw

[jira] [Updated] (SPARK-47799) Preserve parameter information when using SBT package jar

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47799: -- Affects Version/s: 4.0.0 (was: 3.5.1) > Preserve parame

[jira] [Resolved] (SPARK-47799) Preserve parameter information when using SBT package jar

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47799. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45983 [https

[jira] [Assigned] (SPARK-47799) Preserve parameter information when using SBT package jar

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47799: - Assignee: dzcxzl > Preserve parameter information when using SBT package

[jira] [Updated] (SPARK-47797) Skip deleting pod from k8s if the pod does not exists

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47797: -- Affects Version/s: 4.0.0 (was: 3.5.1) > Skip deleting pod from

[jira] [Resolved] (SPARK-47797) Skip deleting pod from k8s if the pod does not exists

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47797. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45979 [https

[jira] [Assigned] (SPARK-47797) Skip deleting pod from k8s if the pod does not exists

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47797: - Assignee: leesf > Skip deleting pod from k8s if the pod does not exi

[jira] [Assigned] (SPARK-47817) Update pandas to 2.2.2

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47817: - Assignee: Bjørn Jørgensen > Update pandas to 2.

[jira] [Resolved] (SPARK-47817) Update pandas to 2.2.2

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47817. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46009 [https

[jira] [Updated] (SPARK-46812) Make `mapInPandas` / mapInArrow` support ResourceProfile (Stage-Level scheduling)

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46812: -- Affects Version/s: (was: 3.5.2) > Make `mapInPandas` / mapInArrow` supp

[jira] [Commented] (HADOOP-18296) Memory fragmentation in ChecksumFileSystem Vectored IO implementation.

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836017#comment-17836017 ] Dongjoon Hyun commented on HADOOP-18296: According to the description, is this a memory leak

Re: Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine

2024-04-10 Thread Dongjoon Hyun
ten v1.1.1 support Spark3.2 and > 3.3. > > We are planning to support Spark3.4 and 3.5 in Gluten v1.2.0. > > Spark4.0 support for Gluten is depending on the release schedule in > Spark community. > > > > On 2024/04/09 07:14:13 Dongjoon Hyun wrote: > > > Thank you for s

[jira] [Commented] (ORC-642) Update PatchedBase doc with patch ceiling in spec

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835971#comment-17835971 ] Dongjoon Hyun commented on ORC-642: --- I added you to the higher group. Please let me know if you hit

[jira] [Commented] (ORC-642) Update PatchedBase doc with patch ceiling in spec

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835970#comment-17835970 ] Dongjoon Hyun commented on ORC-642: --- I checked the JIRA group . It seems that `Committer` group has

[jira] [Updated] (ORC-642) Update PatchedBase doc with patch ceiling in spec

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated ORC-642: -- Attachment: Screenshot 2024-04-10 at 20.24.43.png > Update PatchedBase doc with patch ceiling in s

[jira] [Commented] (ORC-642) Update PatchedBase doc with patch ceiling in spec

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835969#comment-17835969 ] Dongjoon Hyun commented on ORC-642: --- Oh, got it. > Update PatchedBase doc with patch ceiling in s

Re: [VOTE] Release Apache ORC 1.8.7 (RC0)

2024-04-10 Thread Dongjoon Hyun
1.3.x branch Dongjoon. On 2024/04/10 18:49:11 Dongjoon Hyun wrote: > Please vote on releasing the following candidate as Apache ORC version > 1.8.7. This vote is open until April 14th 1AM (PST) and passes if a > majority +1 PMC votes are cast, with a minimum of 3 +1 votes. > >

[VOTE] Release Apache ORC 1.8.7 (RC0)

2024-04-10 Thread Dongjoon Hyun
Please vote on releasing the following candidate as Apache ORC version 1.8.7. This vote is open until April 14th 1AM (PST) and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache ORC 1.8.7 [ ] -1 Do not release this package because ...

[jira] [Resolved] (ORC-1686) [C++] Avoid using std::filesystem

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1686. Fix Version/s: 2.0.1 2.1.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (ORC-1685) Use `Pattern Matching for instanceof` in RecordReaderImpl

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1685: -- Assignee: dzcxzl > Use `Pattern Matching for instanceof` in RecordReaderI

[jira] [Resolved] (ORC-1685) Use `Pattern Matching for instanceof` in RecordReaderImpl

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1685. Fix Version/s: 2.1.0 Resolution: Fixed Issue resolved by pull request 1884 [https

[jira] (SPARK-44478) Executor decommission causes stage failure

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44478 ] Dongjoon Hyun deleted comment on SPARK-44478: --- was (Author: dongjoon): Ya, it's difficult to reproduce. BTW, could you try 3.4.2, [~juhai], because SPARK-44630 is in Apache Spark 3.4.2

[jira] [Commented] (SPARK-44478) Executor decommission causes stage failure

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835831#comment-17835831 ] Dongjoon Hyun commented on SPARK-44478: --- Ya, it's difficult to reproduce. BTW, could you try 3.4.2

[jira] [Updated] (ORC-642) Update PatchedBase doc with patch ceiling in spec

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated ORC-642: -- Summary: Update PatchedBase doc with patch ceiling in spec (was: ORC specification for RLEv2

[jira] [Updated] (ORC-642) ORC specification for RLEv2 is incomplete

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated ORC-642: -- Component/s: documentation > ORC specification for RLEv2 is incompl

[jira] [Resolved] (ORC-642) ORC specification for RLEv2 is incomplete

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-642. --- Fix Version/s: 2.1.0 Resolution: Fixed > ORC specification for RLEv2 is incompl

Re: ORC 1.8.7 Release

2024-04-10 Thread Dongjoon Hyun
Oh, thank you, William. Let me volunteer and start it now. I'm the release manager of Apache Spark 3.4.3 (for next Monday). I guess I can include Apache ORC 1.8.7 there too. Thanks, Dongjoon. On 2024/04/10 01:48:35 "William H." wrote: > Hey All, > > I hope everyone had a great weekend and

<    1   2   3   4   5   6   7   8   9   10   >