[jira] [Commented] (SPARK-44444) Use ANSI mode by default

2024-04-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836869#comment-17836869 ] Dongjoon Hyun commented on SPARK-4: --- Thank you all. The vote is started here. - https

Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

2024-04-13 Thread Dongjoon Hyun
I'll start from my +1. Dongjoon. On 2024/04/13 22:22:05 Dongjoon Hyun wrote: > Please vote on SPARK-4 to use ANSI SQL mode by default. > The technical scope is defined in the following PR which is > one line of code change and one line of migration guide. > > - DISC

[jira] [Updated] (SPARK-44444) Use ANSI mode by default

2024-04-13 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-4: -- Summary: Use ANSI mode by default (was: Enabled ANSI mode by default) > Use ANSI m

[VOTE] SPARK-44444: Use ANSI SQL mode by default

2024-04-13 Thread Dongjoon Hyun
Please vote on SPARK-4 to use ANSI SQL mode by default. The technical scope is defined in the following PR which is one line of code change and one line of migration guide. - DISCUSSION: https://lists.apache.org/thread/ztlwoz1v1sn81ssks12tb19x37zozxlz - JIRA:

Re: [DISCUSS] SPARK-44444: Use ANSI SQL mode by default

2024-04-13 Thread Dongjoon Hyun
ror Attribution Framework > > <https://issues.apache.org/jira/browse/SPARK-38615> will also be beneficial > > in migrating to ANSI SQL mode. > > > > > > Gengliang > > > > > > On Thu, Apr 11, 2024 at 7:56 PM Dongjoon Hyun > <mailto:dongjoon.h

Re: [VOTE] Release Apache ORC 1.8.7 (RC0)

2024-04-13 Thread Dongjoon Hyun
-test > > > 7/8 Test #7: java-bench-spark-test Passed 215.06 sec > > > Start 8: tool-test > > > 8/8 Test #8: tool-test Passed 17.01 sec > > > > > > 100% tests passed, 0 tests failed out of 8 &

[jira] [Commented] (SPARK-46122) Disable spark.sql.legacy.createHiveTableByDefault by default

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836708#comment-17836708 ] Dongjoon Hyun commented on SPARK-46122: --- Gentle ping, [~yumwang] . You can throw a discussion

Re: [DISCUSS] Spark 4.0.0 release

2024-04-12 Thread Dongjoon Hyun
Thank you for volunteering, Wenchen. Dongjoon. On 2024/04/12 15:11:04 Wenchen Fan wrote: > Hi all, > > It's close to the previously proposed 4.0.0 release date (June 2024), and I > think it's time to prepare for it and discuss the ongoing projects: > >- ANSI by default >- Spark Connect

[jira] [Updated] (SPARK-44518) Completely make hive as a data source

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44518: -- Parent: (was: SPARK-44111) Issue Type: Improvement (was: Sub-task) > Complet

[jira] [Updated] (SPARK-44518) Completely make hive as a data source

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44518: -- Target Version/s: (was: 4.0.0) > Completely make hive as a data sou

[jira] [Updated] (SPARK-44518) Completely make hive as a data source

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44518: -- Fix Version/s: (was: 4.0.0) > Completely make hive as a data sou

[jira] [Commented] (SPARK-44518) Completely make hive as a data source

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836704#comment-17836704 ] Dongjoon Hyun commented on SPARK-44518: --- In addition, I'll convert this to an independent JIRA

[jira] [Commented] (SPARK-44518) Completely make hive as a data source

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836703#comment-17836703 ] Dongjoon Hyun commented on SPARK-44518: --- Hi, [~roryqi], this looks like a question instead of any

[jira] [Updated] (SPARK-44116) Utilize Hadoop vectorized APIs

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44116: -- Parent: (was: SPARK-44111) Issue Type: New Feature (was: Sub-task) > Util

[jira] [Commented] (HADOOP-18296) Memory fragmentation in ChecksumFileSystem Vectored IO implementation.

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836697#comment-17836697 ] Dongjoon Hyun commented on HADOOP-18296: Thank you for sharing the direction. > Mem

[jira] [Assigned] (SPARK-44444) Enabled ANSI mode by default

2024-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-4: - Assignee: Dongjoon Hyun > Enabled ANSI mode by defa

Re: [VOTE] Add new `Versions` in Apache Spark JIRA for Versioning of Spark Operator

2024-04-12 Thread Dongjoon Hyun
+1 Thank you! I hope we can customize `dev/merge_spark_pr.py` script per repository after this PR. Dongjoon. On 2024/04/12 03:28:36 "L. C. Hsieh" wrote: > Hi all, > > Thanks for all discussions in the thread of "Versioning of Spark > Operator":

[jira] [Commented] (SPARK-44444) Enabled ANSI mode by default

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836397#comment-17836397 ] Dongjoon Hyun commented on SPARK-4: --- Hi, All. Here is the discussion thread I sent a few

[DISCUSS] SPARK-44444: Use ANSI SQL mode by default

2024-04-11 Thread Dongjoon Hyun
Hi, All. Thanks to you, we've been achieving many things and have on-going SPIPs. I believe it's time to scope Apache Spark 4.0.0 (SPARK-44111) more narrowly by asking your opinions about Apache Spark's ANSI SQL mode. https://issues.apache.org/jira/browse/SPARK-44111 Prepare Apache Spark

[jira] [Commented] (SPARK-44444) Enabled ANSI mode by default

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836389#comment-17836389 ] Dongjoon Hyun commented on SPARK-4: --- I made a draft PR and will initiate the discussion thread

[jira] [Assigned] (SPARK-47814) Remove the `KinesisTestUtils` from `main` to `test`

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47814: - Assignee: BingKun Pan > Remove the `KinesisTestUtils` from `main` to `t

[jira] [Resolved] (SPARK-47814) Remove the `KinesisTestUtils` from `main` to `test`

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47814. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46000 [https

[jira] [Updated] (SPARK-47777) Add spark connect test for python streaming data source

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-4: -- Component/s: Tests > Add spark connect test for python streaming data sou

[jira] [Assigned] (SPARK-47777) Add spark connect test for python streaming data source

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-4: - Assignee: Chaoqin Li > Add spark connect test for python streaming data sou

[jira] [Resolved] (SPARK-47777) Add spark connect test for python streaming data source

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-4. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45950 [https

[jira] [Resolved] (SPARK-47820) Run `ANSI` SQL CI twice per day

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47820. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46010 [https

[jira] [Updated] (SPARK-47820) Run `ANSI` SQL CI twice per day

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47820: -- Summary: Run `ANSI` SQL CI twice per day (was: Run `ANSI` SQL Daily CI twice) > Run `A

[jira] [Created] (SPARK-47820) Run `ANSI` SQL Daily CI twice

2024-04-11 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47820: - Summary: Run `ANSI` SQL Daily CI twice Key: SPARK-47820 URL: https://issues.apache.org/jira/browse/SPARK-47820 Project: Spark Issue Type: Test

[jira] [Updated] (SPARK-47820) Run `ANSI` SQL Daily CI twice

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47820: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Test) > Run `ANSI` SQL Daily CI tw

[jira] [Updated] (SPARK-47799) Preserve parameter information when using SBT package jar

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47799: -- Affects Version/s: 4.0.0 (was: 3.5.1) > Preserve parame

[jira] [Resolved] (SPARK-47799) Preserve parameter information when using SBT package jar

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47799. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45983 [https

[jira] [Assigned] (SPARK-47799) Preserve parameter information when using SBT package jar

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47799: - Assignee: dzcxzl > Preserve parameter information when using SBT package

[jira] [Updated] (SPARK-47797) Skip deleting pod from k8s if the pod does not exists

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47797: -- Affects Version/s: 4.0.0 (was: 3.5.1) > Skip deleting pod from

[jira] [Resolved] (SPARK-47797) Skip deleting pod from k8s if the pod does not exists

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47797. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45979 [https

[jira] [Assigned] (SPARK-47797) Skip deleting pod from k8s if the pod does not exists

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47797: - Assignee: leesf > Skip deleting pod from k8s if the pod does not exi

[jira] [Assigned] (SPARK-47817) Update pandas to 2.2.2

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47817: - Assignee: Bjørn Jørgensen > Update pandas to 2.

[jira] [Resolved] (SPARK-47817) Update pandas to 2.2.2

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47817. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46009 [https

[jira] [Updated] (SPARK-46812) Make `mapInPandas` / mapInArrow` support ResourceProfile (Stage-Level scheduling)

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46812: -- Affects Version/s: (was: 3.5.2) > Make `mapInPandas` / mapInArrow` supp

[jira] [Commented] (HADOOP-18296) Memory fragmentation in ChecksumFileSystem Vectored IO implementation.

2024-04-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836017#comment-17836017 ] Dongjoon Hyun commented on HADOOP-18296: According to the description, is this a memory leak

Re: Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine

2024-04-10 Thread Dongjoon Hyun
ten v1.1.1 support Spark3.2 and > 3.3. > > We are planning to support Spark3.4 and 3.5 in Gluten v1.2.0. > > Spark4.0 support for Gluten is depending on the release schedule in > Spark community. > > > > On 2024/04/09 07:14:13 Dongjoon Hyun wrote: > > > Thank you for s

[jira] [Commented] (ORC-642) Update PatchedBase doc with patch ceiling in spec

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835971#comment-17835971 ] Dongjoon Hyun commented on ORC-642: --- I added you to the higher group. Please let me know if you hit

[jira] [Commented] (ORC-642) Update PatchedBase doc with patch ceiling in spec

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835970#comment-17835970 ] Dongjoon Hyun commented on ORC-642: --- I checked the JIRA group . It seems that `Committer` group has

[jira] [Updated] (ORC-642) Update PatchedBase doc with patch ceiling in spec

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated ORC-642: -- Attachment: Screenshot 2024-04-10 at 20.24.43.png > Update PatchedBase doc with patch ceiling in s

[jira] [Commented] (ORC-642) Update PatchedBase doc with patch ceiling in spec

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835969#comment-17835969 ] Dongjoon Hyun commented on ORC-642: --- Oh, got it. > Update PatchedBase doc with patch ceiling in s

Re: [VOTE] Release Apache ORC 1.8.7 (RC0)

2024-04-10 Thread Dongjoon Hyun
1.3.x branch Dongjoon. On 2024/04/10 18:49:11 Dongjoon Hyun wrote: > Please vote on releasing the following candidate as Apache ORC version > 1.8.7. This vote is open until April 14th 1AM (PST) and passes if a > majority +1 PMC votes are cast, with a minimum of 3 +1 votes. > >

[VOTE] Release Apache ORC 1.8.7 (RC0)

2024-04-10 Thread Dongjoon Hyun
Please vote on releasing the following candidate as Apache ORC version 1.8.7. This vote is open until April 14th 1AM (PST) and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache ORC 1.8.7 [ ] -1 Do not release this package because ...

[jira] [Resolved] (ORC-1686) [C++] Avoid using std::filesystem

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1686. Fix Version/s: 2.0.1 2.1.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (ORC-1685) Use `Pattern Matching for instanceof` in RecordReaderImpl

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1685: -- Assignee: dzcxzl > Use `Pattern Matching for instanceof` in RecordReaderI

[jira] [Resolved] (ORC-1685) Use `Pattern Matching for instanceof` in RecordReaderImpl

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1685. Fix Version/s: 2.1.0 Resolution: Fixed Issue resolved by pull request 1884 [https

[jira] (SPARK-44478) Executor decommission causes stage failure

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44478 ] Dongjoon Hyun deleted comment on SPARK-44478: --- was (Author: dongjoon): Ya, it's difficult to reproduce. BTW, could you try 3.4.2, [~juhai], because SPARK-44630 is in Apache Spark 3.4.2

[jira] [Commented] (SPARK-44478) Executor decommission causes stage failure

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835831#comment-17835831 ] Dongjoon Hyun commented on SPARK-44478: --- Ya, it's difficult to reproduce. BTW, could you try 3.4.2

[jira] [Updated] (ORC-642) Update PatchedBase doc with patch ceiling in spec

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated ORC-642: -- Summary: Update PatchedBase doc with patch ceiling in spec (was: ORC specification for RLEv2

[jira] [Updated] (ORC-642) ORC specification for RLEv2 is incomplete

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated ORC-642: -- Component/s: documentation > ORC specification for RLEv2 is incompl

[jira] [Resolved] (ORC-642) ORC specification for RLEv2 is incomplete

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-642. --- Fix Version/s: 2.1.0 Resolution: Fixed > ORC specification for RLEv2 is incompl

Re: ORC 1.8.7 Release

2024-04-10 Thread Dongjoon Hyun
Oh, thank you, William. Let me volunteer and start it now. I'm the release manager of Apache Spark 3.4.3 (for next Monday). I guess I can include Apache ORC 1.8.7 there too. Thanks, Dongjoon. On 2024/04/10 01:48:35 "William H." wrote: > Hey All, > > I hope everyone had a great weekend and

[jira] [Comment Edited] (ORC-642) ORC specification for RLEv2 is incomplete

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835810#comment-17835810 ] Dongjoon Hyun edited comment on ORC-642 at 4/10/24 4:18 PM: I manually added

[jira] [Commented] (ORC-642) ORC specification for RLEv2 is incomplete

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835810#comment-17835810 ] Dongjoon Hyun commented on ORC-642: --- I manually added and assigned him. > ORC specification for RL

[jira] [Commented] (ORC-642) ORC specification for RLEv2 is incomplete

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835808#comment-17835808 ] Dongjoon Hyun commented on ORC-642: --- Did you try to use the official merge script, [~deshanxiao

[jira] [Assigned] (ORC-642) ORC specification for RLEv2 is incomplete

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-642: - Assignee: Jeffrey Vo > ORC specification for RLEv2 is incompl

[jira] [Updated] (SPARK-47790) Upgrade `commons-io` to 2.16.1

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47790: -- Fix Version/s: 3.5.2 > Upgrade `commons-io` to 2.1

[jira] [Resolved] (SPARK-47790) Upgrade `commons-io` to 2.16.1

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47790. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45973 [https

Re: Versioning of Spark Operator

2024-04-10 Thread Dongjoon Hyun
Ya, that would work. Inevitably, I looked at Apache Flink K8s Operator's JIRA and GitHub repo. It looks reasonable to me. Although they share the same JIRA, they choose different patterns per place. 1. In POM file and Maven Artifact, independent version number. 1.8.0 2. Tag is also based on

[jira] [Resolved] (SPARK-47706) Bump json4s 4.0.7

2024-04-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47706. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45838 [https

[jira] [Updated] (SPARK-45444) Upgrade `commons-io` to 2.14.0

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45444: -- Summary: Upgrade `commons-io` to 2.14.0 (was: Upgrade `commons-io` to 1.24.0) > Upgr

[jira] [Updated] (SPARK-45444) Upgrade `commons-io` to 1.24.0

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45444: -- Parent: SPARK-47046 Issue Type: Sub-task (was: Improvement) > Upgrade `commons

[jira] [Updated] (SPARK-45734) Upgrade commons-io to 2.15.0

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45734: -- Parent: SPARK-47046 Issue Type: Sub-task (was: Improvement) > Upgrade commons

[jira] [Updated] (SPARK-47083) Upgrade `commons-codec` to 1.16.1

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47083: -- Fix Version/s: 3.5.2 > Upgrade `commons-codec` to 1.1

[jira] [Updated] (SPARK-47182) Exclude `commons-(io|lang3)` transitive dependencies from `commons-compress` and `avro-*`

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47182: -- Fix Version/s: 3.5.2 > Exclude `commons-(io|lang3)` transitive dependencies from `comm

[jira] [Resolved] (SPARK-47787) Upgrade `commons-compress` to 1.26.1`

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47787. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45966 [https

Re: Versioning of Spark Operator

2024-04-09 Thread Dongjoon Hyun
ctor Go Client? For example, > Spark Operator 3.5.x supports Spark 3.5 and above. > > Best, > Bo > > > On Tue, Apr 9, 2024 at 10:14 AM Dongjoon Hyun wrote: > > > Ya, that's simple and possible. > > > > However, it may cause many confusions because it implie

[jira] [Updated] (SPARK-47787) Upgrade `commons-compress` to 1.26.1`

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47787: -- Summary: Upgrade `commons-compress` to 1.26.1` (was: Upgrade `commons-compress` to 1.26

[jira] [Created] (SPARK-47787) Upgrade `commons-compress` to 1.26.`

2024-04-09 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47787: - Summary: Upgrade `commons-compress` to 1.26.` Key: SPARK-47787 URL: https://issues.apache.org/jira/browse/SPARK-47787 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (ORC-1681) Remove redundant import statement in tests to fix checkstyle failures

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1681: -- Assignee: Dongjoon Hyun > Remove redundant import statement in tests to fix checkstyle failu

[jira] [Updated] (ORC-1681) Remove redundant import statement in tests to fix checkstyle failures

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated ORC-1681: --- Summary: Remove redundant import statement in tests to fix checkstyle failures (was: Remove redundant

[jira] [Created] (ORC-1681) Remove redundant import statement to fix checkstyle failures

2024-04-09 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created ORC-1681: -- Summary: Remove redundant import statement to fix checkstyle failures Key: ORC-1681 URL: https://issues.apache.org/jira/browse/ORC-1681 Project: ORC Issue Type

[jira] [Resolved] (ORC-1680) Bump `bcpkix-jdk18on` to 1.78

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1680. Fix Version/s: 2.1.0 Resolution: Fixed Issue resolved by pull request 1880 [https

[jira] [Assigned] (ORC-1680) Bump `bcpkix-jdk18on` to 1.78

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1680: -- Assignee: Dongjoon Hyun > Bump `bcpkix-jdk18on` to 1

[jira] [Created] (ORC-1680) Bump `bcpkix-jdk18on` to 1.78

2024-04-09 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created ORC-1680: -- Summary: Bump `bcpkix-jdk18on` to 1.78 Key: ORC-1680 URL: https://issues.apache.org/jira/browse/ORC-1680 Project: ORC Issue Type: Test Components

[jira] [Resolved] (ORC-1679) Bump `zstd-jni` 1.5.6-2

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1679. Fix Version/s: 2.0.1 2.1.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (ORC-1679) Bump `zstd-jni` 1.5.6-2

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1679: -- Assignee: Dongjoon Hyun > Bump `zstd-jni` 1.5.

[jira] [Resolved] (ORC-1678) Bump `checkstyle` to 10.15.0

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1678. Fix Version/s: 2.1.0 Resolution: Fixed Issue resolved by pull request 1872 [https

[jira] [Created] (ORC-1679) Bump `zstd-jni` 1.5.6-2

2024-04-09 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created ORC-1679: -- Summary: Bump `zstd-jni` 1.5.6-2 Key: ORC-1679 URL: https://issues.apache.org/jira/browse/ORC-1679 Project: ORC Issue Type: Bug Components: Java

[jira] [Assigned] (ORC-1678) Bump `checkstyle` to 10.15.0

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1678: -- Assignee: Dongjoon Hyun > Bump `checkstyle` to 10.1

[jira] [Created] (ORC-1678) Bump `checkstyle` to 10.15.0

2024-04-09 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created ORC-1678: -- Summary: Bump `checkstyle` to 10.15.0 Key: ORC-1678 URL: https://issues.apache.org/jira/browse/ORC-1678 Project: ORC Issue Type: Test Components: Java

Re: Versioning of Spark Operator

2024-04-09 Thread Dongjoon Hyun
elopers and > > intuitive for users. > > > > Regards, > > Mridul > > > > > > On Tue, Apr 9, 2024 at 10:09 AM Dongjoon Hyun > <mailto:dongj...@apache.org>> wrote: > >> Hi, Liang-Chi. > >> > >> Thank you for leading Ap

[jira] [Created] (SPARK-47785) Upgrade `bouncycastle` to 1.78

2024-04-09 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47785: - Summary: Upgrade `bouncycastle` to 1.78 Key: SPARK-47785 URL: https://issues.apache.org/jira/browse/SPARK-47785 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-47748) Upgrade `zstd-jni` to 1.5.6-2

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47748: -- Parent: SPARK-47046 Issue Type: Sub-task (was: Improvement) > Upgrade `zstd-

[jira] [Assigned] (SPARK-47782) Remove redundant json4s-jackson definition in sql/api POM

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47782: - Assignee: Cheng Pan (was: Dongjoon Hyun) > Remove redundant json4s-jackson definit

[jira] [Assigned] (SPARK-47782) Remove redundant json4s-jackson definition in sql/api POM

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47782: - Assignee: Dongjoon Hyun > Remove redundant json4s-jackson definition in sql/api

[jira] [Resolved] (SPARK-47782) Remove redundant json4s-jackson definition in sql/api POM

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47782. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45943 [https

[jira] [Updated] (SPARK-47782) Remove redundant json4s-jackson definition in sql/api POM

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47782: -- Parent: SPARK-47046 Issue Type: Sub-task (was: Task) > Remove redundant jso

[jira] [Created] (SPARK-47782) Remove redundant json4s-jackson definition in sql/api POM

2024-04-09 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47782: - Summary: Remove redundant json4s-jackson definition in sql/api POM Key: SPARK-47782 URL: https://issues.apache.org/jira/browse/SPARK-47782 Project: Spark

Re: Versioning of Spark Operator

2024-04-09 Thread Dongjoon Hyun
Hi, Liang-Chi. Thank you for leading Apache Spark K8s operator as a shepherd. I took a look at `Apache Spark Connect Go` repo mentioned in the thread. Sadly, there is no release at all and no activity since last 6 months. It seems to be the first time for Apache Spark community to consider

[jira] [Updated] (SPARK-47774) Remove redundant rules from `MimaExcludes`

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47774: -- Fix Version/s: 3.4.3 > Remove redundant rules from `MimaExclu

Re: SPIP: Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines

2024-04-09 Thread Dongjoon Hyun
Thank you for sharing, Jia. I have the same questions like the previous Weiting's thread. Do you think you can share the future milestone of Apache Gluten? I'm wondering when the first stable release will come and how we can coordinate across the ASF communities. > This project is still under

[jira] [Updated] (SPARK-47774) Remove redundant rules from `MimaExcludes`

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47774: -- Fix Version/s: 3.5.2 > Remove redundant rules from `MimaExclu

Re: Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine

2024-04-09 Thread Dongjoon Hyun
Thank you for sharing, Weiting. Do you think you can share the future milestone of Apache Gluten? I'm wondering when the first stable release will come and how we can coordinate across the ASF communities. > This project is still under active development now, and doesn't have a stable release. >

[jira] [Resolved] (SPARK-47774) Remove redundant rules from `MimaExcludes`

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47774. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45944 [https

[jira] [Updated] (SPARK-47773) Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47773: -- Affects Version/s: 4.0.0 (was: 3.5.1) > Enhancing the Flexibil

[jira] [Commented] (SPARK-47773) Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines

2024-04-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835181#comment-17835181 ] Dongjoon Hyun commented on SPARK-47773: --- Since this is a new feature which cannot affect old

<    1   2   3   4   5   6   7   8   9   10   >