Thanks, Okumin. I'll build the distro and check whether it's being packaged. If it's not, I believe it's permissible — but I'll run it by Legal to be completely sure. Also, from a quick scan of other Apache repositories, it seems some projects are using this as well [1].
That said, I'll explore whether we can switch to GraalVM and eliminate this dependency altogether. Let me dig a bit deeper and circle back with a clearer plan to address this. -Ayush [1] https://github.com/search?q=org%3Aapache+nashorn-core&type=code&p=1 On Sun, 1 Jun 2025 at 16:12, Shohei Okumiya <oku...@apache.org> wrote: > > Hi, > > Thanks for making it. It's a massive step for us! > > I found the compile dependency on "nashorn-core" was newly added, > which is licensed under "gpl-v2-with-the-classpath-exception-license". > I'm not confident that we can distribute the jar file on a release. > > I'm still testing the master branch with JDK 17. So far, so good. > > Best, > Okumin > > > > On Sat, May 31, 2025 at 4:02 AM Simhadri G <simhad...@apache.org> wrote: > > > > Congratulations guys! That's really awesome! > > > > Will test it out and let you know if I run into any bugs! > > > > Thanks everyone ! > > Simhadri G > > > > > > On Sat, May 31, 2025, 12:28 AM Ayush Saxena <ayush...@gmail.com> wrote: > >> > >> Hi Folks, > >> > >> The Hive master branch has now been upgraded to JDK 17. This means > >> that building or running the latest code from the master branch will > >> require JDK 17 going forward. > >> You're encouraged to try it out and share your feedback. If you > >> encounter any issues, please feel free to report them here or create a > >> ticket with the details. > >> > >> A big thank you to all the contributors who helped make this > >> transition possible! > >> > >> Looking ahead, Hive 4.1.0 is expected to be released by the end of > >> June or early July, and will officially support JDK 17. > >> > >> Best regards, > >> Ayush > >> > >> ---------- Forwarded message --------- > >> From: Vlad Rozov <vro...@apache.org> > >> Date: Tue, 6 May 2025 at 20:29 > >> Subject: Re: Move to JDK-11 > >> To: <dev@hive.apache.org> > >> > >> > >> I am looking to upgrade Apache Spark 4.x to use Apache Hive 4.x as > >> both compile and runtime dependencies. Spark 4.x requires JDK 17 and > >> while working on upgrade I have not encountered issues caused by Hive > >> using JDK 8. I will keep an eye and report any, though I don't expect > >> to see a problem till Spark upgrades to JDK 21. > >> > >> Thank you, > >> > >> Vlad > >> > >> On 2025/04/08 14:27:33 Butao Zhang wrote: > >> > The upgrade to JDK 17 is a significant change that introduces > >> > compatibility and adaptation requirements for both Apache Tez and Apache > >> > Hive. Relevant work is currently in progress—thanks to all community > >> > contributors who have already been involved in adaptation and fixes. We > >> > encourage everyone to actively report any issues encountered during > >> > usage or integration, so that we can collaborate on discussions and > >> > resolutions, and accelerate the overall transition to JDK 17. > >> > > >> > Thanks, > >> > Butao Zhang > >> > > >> > On 2025/04/07 11:25:33 Shohei Okumiya wrote: > >> > > Hi Ayush and all people working on the JDK upgrade, > >> > > > >> > > Thanks for sharing the timeline and detailing the current progress. I'm > >> > > more than happy to hear the good news and look forward to seeing Hive > >> > > running on Java 17! > >> > > > >> > > Mid-May is not so far away. I don't have a firm preference about how to > >> > > package 4.1. If anyone is interested, let's discuss it in the thread > >> > > for > >> > > the Hive 4.1.x release. > >> > > > >> > > Finally, let me express my gratitude to everyone working on the > >> > > challenging > >> > > upgrade project. It will definitely be one of the most outstanding > >> > > achievements! > >> > > > >> > > Sincerely, > >> > > Okumin > >> > > > >> > > On Mon, Apr 7, 2025 at 7:03 PM Ayush Saxena <ayush...@gmail.com> wrote: > >> > > > >> > > > Hi Okumin, > >> > > > > >> > > > I know I'm one of the people responsible for misjudging the timeline > >> > > > around the JDK-17 upgrade, which has ultimately blocked the 4.1.0 > >> > > > release. > >> > > > I want to take a moment to share my thoughts and where things > >> > > > currently > >> > > > stand. > >> > > > > >> > > > Q: Can we confirm the importance and ETA of the JDK upgrade and how > >> > > > we > >> > > > should proceed? > >> > > > > >> > > > Ans: I think we all agree that moving to JDK-17 is a major > >> > > > milestone, and > >> > > > we're admittedly late in adopting it. Many projects have already > >> > > > moved or > >> > > > are actively working towards it. Not being on JDK-17 restricts us > >> > > > from > >> > > > adopting modern third-party libraries like Iceberg and from > >> > > > leveraging > >> > > > several improvements that come with it. > >> > > > > >> > > > As for the ETA: I realize I previously gave an inaccurate timeline, > >> > > > and I > >> > > > sincerely regret that. However, I now have a much clearer picture. > >> > > > I'm in > >> > > > active discussions with folks internally who are working on it. I > >> > > > believe > >> > > > we should have the code ready by mid-May, at which point we can start > >> > > > gathering feedback from the community. > >> > > > > >> > > > Q: We've been working on this upgrade for over half a year. When > >> > > > will it > >> > > > merge? PR: https://github.com/apache/hive/pull/5404 > >> > > > In my opinion, #5404 should be divided into multiple pull requests. I > >> > > > really understand we want to test simultaneously rather than make > >> > > > incremental progress. However, the consolidated PR makes it > >> > > > impossible for > >> > > > other developers to help the initiative, reviewers have to check huge > >> > > > changes, and authors must occasionally resolve many conflicts. I > >> > > > don't > >> > > > prefer any HOW as long as the delivery is predictable, meaning it is > >> > > > okay > >> > > > for me if we are confident with the agreed ETA. If it is still > >> > > > uncertain, > >> > > > we may try another project management, which is > >> > > > incremental/predictable and > >> > > > where we can invest other developers' resources in it. > >> > > > > >> > > > Ans: I completely agree with your observations. The original PR > >> > > > started > >> > > > more as an experiment to see what changes were needed. At the time, > >> > > > I don’t > >> > > > think anyone anticipated how large it would become, and > >> > > > unfortunately, it > >> > > > kept growing incrementally over time. > >> > > > That said, I’ve already asked the team to start breaking down the PR > >> > > > into > >> > > > smaller, more manageable parts. Dependency upgrades will be > >> > > > separated out > >> > > > soon, and changes like those related to SpotBugs (which account for > >> > > > a large > >> > > > number of line changes) will also be split. The goal is to make the > >> > > > PR > >> > > > easier to review and more accessible for other contributors to jump > >> > > > in. I > >> > > > understand this is a genuine blocker, and folks are doing their > >> > > > best to > >> > > > address it. > >> > > > > >> > > > Q: [1] Should we ship 4.1 without JDK-17, given it's been a long time > >> > > > since 4.0.1? > >> > > > > >> > > > Ans: Honestly, I’m not in a position to oppose this. The community > >> > > > kindly > >> > > > respected my request last time when I asked for more time for the > >> > > > JDK-17 > >> > > > upgrade, and unfortunately, I couldn’t follow through as expected. > >> > > > So, with > >> > > > sincere apologies for being one of the reasons for the delay, I’m > >> > > > okay with > >> > > > whatever direction the community chooses to take from here. > >> > > > > >> > > > -Ayush > >> > > > > >> > > > > >> > > > On Sun, 6 Apr 2025 at 11:55, Shohei Okumiya <oku...@apache.org> > >> > > > wrote: > >> > > > > >> > > >> Hi, > >> > > >> > >> > > >> I appreciate everyone working on this big and challenging project! > >> > > >> > >> > > >> Can we confirm the importance and ETA of the JDK upgrade and how we > >> > > >> should work on it? > >> > > >> > >> > > >> I recognize it is one of the most essential initiatives blocking the > >> > > >> release of Hive 4.1.0[1] and the upgrade of Iceberg libraries[2]. > >> > > >> Users and > >> > > >> developers would expect it to be shipped soon. > >> > > >> > >> > > >> We have to complete this effort to ship JDK 17. We've been working > >> > > >> on it > >> > > >> for over half a year. Some of us might be interested in when we > >> > > >> will merge > >> > > >> it. > >> > > >> https://github.com/apache/hive/pull/5404 > >> > > >> > >> > > >> In my opinion, #5404 should be divided into multiple pull requests. > >> > > >> I > >> > > >> really understand we want to test simultaneously rather than make > >> > > >> incremental progress. However, the consolidated PR makes it > >> > > >> impossible for > >> > > >> other developers to help the initiative, reviewers have to check > >> > > >> huge > >> > > >> changes, and authors must occasionally resolve many conflicts. I > >> > > >> don't > >> > > >> prefer any HOW as long as the delivery is predictable, meaning it > >> > > >> is okay > >> > > >> for me if we are confident with the agreed ETA. If it is still > >> > > >> uncertain, > >> > > >> we may try another project management, which is > >> > > >> incremental/predictable and > >> > > >> where we can invest other developers' resources in it. > >> > > >> > >> > > >> - [1] Honestly, I feel we should ship 4.1 without JDK 17, as it's > >> > > >> been a > >> > > >> long time since the last release of 4.0.1 > >> > > >> - [2] Iceberg 1.7 or later doesn't support Java 8 > >> > > >> > >> > > >> Best, > >> > > >> Okumin > >> > > >> > >> > > >> On Wed, Dec 11, 2024 at 2:51 AM Ayush Saxena <ayush...@gmail.com> > >> > > >> wrote: > >> > > >> > >> > > >>> Regarding JDK-11, we didn't actively chased that, so its status > >> > > >>> remains > >> > > >>> uncertain. With lower compile-time support like JDK-8, we can > >> > > >>> certainly > >> > > >>> ensure runtime support for both JDK-11 and JDK-17, which I believe > >> > > >>> should > >> > > >>> work for Hive 4.0.1. As far as I know, this approach works > >> > > >>> downstream for > >> > > >>> us atleast. > >> > > >>> > >> > > >>> However, maintaining compile-time compatibility across JDK-17, > >> > > >>> JDK-11, > >> > > >>> and JDK-8 simultaneously may not be feasible. JDK-17 introduces > >> > > >>> significant > >> > > >>> changes, and I don’t believe it’s naive—or even possible—to > >> > > >>> compile Hive > >> > > >>> using both JDK-8 and JDK-17 & expect things to work properly. > >> > > >>> > >> > > >>> The initial plan appeared to be supporting JDK-8 for Hive 4.0.x > >> > > >>> release > >> > > >>> line, with potential tweaks if required to enable runtime support > >> > > >>> for > >> > > >>> JDK-17. Starting with Hive 4.1.x, we could shift to JDK-17 for > >> > > >>> compile-time > >> > > >>> support, and later evaluate the feasibility of supporting JDK-21 > >> > > >>> or newer > >> > > >>> versions. > >> > > >>> > >> > > >>> Once we see a state when nobody want JDK-8 compile time support & > >> > > >>> everyone has moved to JDK-17 or say using 4.1.x release line, We > >> > > >>> can then > >> > > >>> drop having bug fix releases for 4.0.x > >> > > >>> > >> > > >>> -Ayush > >> > > >>> > >> > > >>> On Mon, 9 Dec 2024 at 15:11, Butao Zhang <butaozha...@163.com> > >> > > >>> wrote: > >> > > >>> > >> > > >>>> IMO, I think there two challenges if we want to maintain JDK8 & > >> > > >>>> JDK11 & > >> > > >>>> JDK17 at the same time. > >> > > >>>> > >> > > >>>> - Different jdk syntactic sugar between multiple jdks, and > >> > > >>>> advanced > >> > > >>>> feats in jdk17&jdk21, such as jdk vector api. If we want the > >> > > >>>> multiple jdks, > >> > > >>>> we can not use the advanced feats in high jdk in the future. > >> > > >>>> Such as the > >> > > >>>> discussion in https://issues.apache.org/jira/browse/HIVE-28614 > >> > > >>>> - We need to run the jenkins CI process multiple times to run > >> > > >>>> the > >> > > >>>> multiple jdk env. I'm not sure we have enough jenkins > >> > > >>>> resources to do this. > >> > > >>>> However, i want to say if we can use github actions to run the > >> > > >>>> whole CI? I > >> > > >>>> have seen Apache Spark have done the good job which using the > >> > > >>>> forked > >> > > >>>> repository github actions to run the CI. FRY > >> > > >>>> > >> > > >>>> https://lists.apache.org/thread/dmv1h2snqc7qtdjlzj4k8qxookclp720 > >> > > >>>> > >> > > >>>> > >> > > >>>> Thanks, > >> > > >>>> Butao Zhang > >> > > >>>> ---- Replied Message ---- > >> > > >>>> From Shohei Okumiya<oku...@apache.org> <undefined> > >> > > >>>> Date 12/9/2024 17:06 > >> > > >>>> To <dev@hive.apache.org> <dev@hive.apache.org> > >> > > >>>> Subject Re: Move to JDK-11 > >> > > >>>> Hi, > >> > > >>>> > >> > > >>>> I appreciate everyone's work on the JDK upgrade. It would be > >> > > >>>> ideal if > >> > > >>>> we could migrate JDK like this. That will give us an > >> > > >>>> incremental migration path. > >> > > >>>> > >> > > >>>> 1. (now) Hive 4.0 compiled by JDK 8, and run by JDK 8 > >> > > >>>> 2. Hive 4.x compiled by JDK 8, and run by JDK 17 > >> > > >>>> 3. Hive 4.x compiled by JDK 17, and run by JDK 17 > >> > > >>>> > >> > > >>>> I guess it is technically possible that Hive supports JDK 8 and > >> > > >>>> 17 at > >> > > >>>> runtime. Apache Spark supports JDK 8, 11, and 17. What is the most > >> > > >>>> challenging point? > >> > > >>>> > >> > > >>>> Regards, > >> > > >>>> Okumin > >> > > >>>> > >> > > >>>> On Thu, Aug 29, 2024 at 10:48 PM Butao Zhang <butaozha...@163.com> > >> > > >>>> wrote: > >> > > >>>> > >> > > >>>>> Hi Akshat, > >> > > >>>>> > >> > > >>>>> Thanks for your reply! I agree that supporting both JDK 8 and > >> > > >>>>> JDK 17 > >> > > >>>>> is not easy. I think we can safely to move to JDK 17 if we can > >> > > >>>>> ensure *hive > >> > > >>>>> application* can access to HiveServer2/Metastore without forcing > >> > > >>>>> to > >> > > >>>>> use the JDK 17. > >> > > >>>>> For example, if a hive application is developed by JDK 8, it can > >> > > >>>>> import the *hive-exec.jar & hive-jdbc.jar* which is compiled by > >> > > >>>>> JDK > >> > > >>>>> 17 and can access to hs2/hms successfully. If this case can be > >> > > >>>>> supported, > >> > > >>>>> I think we can move to JDK 17 as soon as possible. > >> > > >>>>> > >> > > >>>>> BTW, If Apache Hive , Apache Tez and Apache Hadoop are all > >> > > >>>>> compiled > >> > > >>>>> with JDK17 or higer version, i think the hive job will run more > >> > > >>>>> faster. > >> > > >>>>> Users will enjoy significant performance improvements due to JDK > >> > > >>>>> version > >> > > >>>>> upgrades. > >> > > >>>>> I'm looking forward to the performance improvement. :) > >> > > >>>>> > >> > > >>>>> Thanks, > >> > > >>>>> Butao Zhang > >> > > >>>>> ---- Replied Message ---- > >> > > >>>>> From Akshat m<akshatats...@gmail.com> <undefined> > >> > > >>>>> Date 8/29/2024 18:44 > >> > > >>>>> To <dev@hive.apache.org> <dev@hive.apache.org> > >> > > >>>>> Subject Re: Move to JDK-11 > >> > > >>>>> Hi Butao, > >> > > >>>>> > >> > > >>>>> It won’t be feasible to support both JDK 8 and JDK 17 in the > >> > > >>>>> same Hive > >> > > >>>>> version due to compatibility issues and behavior changes, as > >> > > >>>>> detailed in > >> > > >>>>> the PR. The internal changes in JDK implementations have made > >> > > >>>>> this > >> > > >>>>> challenging. Additionally, many other libraries maintain > >> > > >>>>> separate versions > >> > > >>>>> for JDK 17+ and JDK 8 support. > >> > > >>>>> > >> > > >>>>> To align with this approach, I suggest we continue with Hive 4 > >> > > >>>>> supporting JDK 8. Once the JDK 17 PR is merged, we could plan a > >> > > >>>>> separate > >> > > >>>>> release that specifically targets JDK 17+ support. This would > >> > > >>>>> ensure that > >> > > >>>>> Hive stays up-to-date while maintaining stability across > >> > > >>>>> different > >> > > >>>>> environments. > >> > > >>>>> > >> > > >>>>> Would love to hear your thoughts or any additional suggestions > >> > > >>>>> the > >> > > >>>>> community might have! > >> > > >>>>> > >> > > >>>>> Regards > >> > > >>>>> > >> > > >>>>> Akshat > >> > > >>>>> > >> > > >>>>> > >> > > >>>>> > >> > > >>>>> On Wed, Aug 28, 2024 at 8:00 AM Butao Zhang <butaozha...@163.com> > >> > > >>>>> wrote: > >> > > >>>>> > >> > > >>>>>> Good job! Thanks everyone who put effort into the JDK upgrading. > >> > > >>>>>> That's a big good news for Apache Hive community! > >> > > >>>>>> Just a question: Given that many people are still using the > >> > > >>>>>> lower > >> > > >>>>>> jdk8 version. Do we consider to keeping compatibility with > >> > > >>>>>> lower jdk8 > >> > > >>>>>> version? > >> > > >>>>>> > >> > > >>>>>> > >> > > >>>>>> Thanks, > >> > > >>>>>> Butao Zhang > >> > > >>>>>> ---- Replied Message ---- > >> > > >>>>>> From Akshat m<akshatats...@gmail.com> <undefined> > >> > > >>>>>> Date 8/27/2024 23:12 > >> > > >>>>>> To dev<dev@hive.apache.org> <dev@hive.apache.org> > >> > > >>>>>> Cc kokila narayanan<kokilanarayana...@gmail.com>, > >> > > >>>>>> <kokilanarayana...@gmail.com>Raghav > >> > > >>>>>> Aggarwal<raghavaggarwal03...@gmail.com>, > >> > > >>>>>> <raghavaggarwal03...@gmail.com>Ayush Saxena<ayush...@gmail.com>, > >> > > >>>>>> <ayush...@gmail.com>tanishq chugh<tanishqchugh4...@gmail.com>, > >> > > >>>>>> <tanishqchugh4...@gmail.com><aturo...@cloudera.com> > >> > > >>>>>> <aturo...@cloudera.com> > >> > > >>>>>> Subject Re: Move to JDK-11 > >> > > >>>>>> > >> > > >>>>>> Hi everyone, > >> > > >>>>>> > >> > > >>>>>> I’m excited to share that after months of dedicated work, we’ve > >> > > >>>>>> finally achieved a passing PR[1] for running *Apache Hive on > >> > > >>>>>> JDK 17*! > >> > > >>>>>> > >> > > >>>>>> This was a challenging task, requiring to address various > >> > > >>>>>> compatibility issues and ensure our comprehensive test suite > >> > > >>>>>> passed without > >> > > >>>>>> a hitch. > >> > > >>>>>> > >> > > >>>>>> Thanks Kokila and Tanishq for joining me in the initiative and > >> > > >>>>>> Raghav > >> > > >>>>>> for contributing fix for the errorProne issue, and thanks to > >> > > >>>>>> Attila, Ayush > >> > > >>>>>> and other Hive PMC members for their guidance in the initiative > >> > > >>>>>> > >> > > >>>>>> It’s been a real team effort, and I’m grateful for the > >> > > >>>>>> collaboration > >> > > >>>>>> and support from everyone involved. > >> > > >>>>>> > >> > > >>>>>> I’d appreciate it if you could take some time to review the PR. > >> > > >>>>>> Community feedback is really important to make this happen. > >> > > >>>>>> > >> > > >>>>>> The PR #5404 addresses the integration of Apache Hive with JDK > >> > > >>>>>> 17, > >> > > >>>>>> introducing a range of changes aimed at ensuring compatibility > >> > > >>>>>> and > >> > > >>>>>> improving the codebase. Here’s a detailed breakdown: > >> > > >>>>>> > >> > > >>>>>> 1. Core Code Adjustments: > >> > > >>>>>> > >> > > >>>>>> - Compatibility Fixes: Various parts of the Hive codebase > >> > > >>>>>> have > >> > > >>>>>> been modified to ensure they work seamlessly with JDK 17. This > >> > > >>>>>> includes > >> > > >>>>>> addressing language or API changes that JDK 17 introduces. > >> > > >>>>>> > >> > > >>>>>> - Reflection and Modules: Adjustments related to how Hive > >> > > >>>>>> interacts with Java's reflection API, which has stricter module > >> > > >>>>>> system > >> > > >>>>>> rules in JDK 17. > >> > > >>>>>> > >> > > >>>>>> 2. Dependency Management: > >> > > >>>>>> > >> > > >>>>>> - Version Upgrades: Several third-party dependencies have > >> > > >>>>>> been > >> > > >>>>>> updated to versions that are compatible with JDK 17. This is > >> > > >>>>>> critical > >> > > >>>>>> because older versions of some libraries may not work properly > >> > > >>>>>> or might use > >> > > >>>>>> deprecated features removed in JDK 17. > >> > > >>>>>> > >> > > >>>>>> - Data nucleus upgrade: To ensure compatibility with Java > >> > > >>>>>> 17, we > >> > > >>>>>> upgraded DataNucleus to version 6.x. This upgrade, however, > >> > > >>>>>> introduced an > >> > > >>>>>> issue with Hive schema creation. The issue stems from a commit > >> > > >>>>>> in > >> > > >>>>>> DataNucleus[2] that alters the handling of primary key column > >> > > >>>>>> names by > >> > > >>>>>> enclosing them in double quotes. This change caused conflicts > >> > > >>>>>> during Hive > >> > > >>>>>> schema generation. As an interim solution, we implemented a > >> > > >>>>>> patched > >> > > >>>>>> DataNucleus-rdbms module to address the issue in Hive. The > >> > > >>>>>> necessary fix is > >> > > >>>>>> now included in the DataNucleus repository[3]. Once DataNucleus > >> > > >>>>>> 6.0.8 is > >> > > >>>>>> officially released, we will upgrade to this version and remove > >> > > >>>>>> the > >> > > >>>>>> temporary patched module. > >> > > >>>>>> > >> > > >>>>>> - Dependency Cleanup: Unused or outdated dependencies were > >> > > >>>>>> removed, > >> > > >>>>>> reducing potential conflicts and simplifying the build process. > >> > > >>>>>> > >> > > >>>>>> 3. Testing Enhancements: > >> > > >>>>>> > >> > > >>>>>> - Test Suite Updates: The PR includes updates to Hive’s test > >> > > >>>>>> suite > >> > > >>>>>> to ensure all tests pass under JDK 17. This likely involved > >> > > >>>>>> modifying tests > >> > > >>>>>> that needed adjustments to align with JDK 17’s behaviour. > >> > > >>>>>> > >> > > >>>>>> - New Tests: New tests may have been added to cover areas > >> > > >>>>>> where JDK > >> > > >>>>>> 17 introduced new features or changes that could impact Hive. > >> > > >>>>>> > >> > > >>>>>> 4. Compatibility Checks: > >> > > >>>>>> > >> > > >>>>>> - JDK Version Verification: A new compatibility check has been > >> > > >>>>>> introduced to verify the JDK version at runtime. This ensures > >> > > >>>>>> that Hive is > >> > > >>>>>> running on an appropriate JDK version, preventing unexpected > >> > > >>>>>> errors or > >> > > >>>>>> behaviour due to version mismatches. > >> > > >>>>>> > >> > > >>>>>> 5. Build Updates: > >> > > >>>>>> > >> > > >>>>>> - Build System Tweaks: The PR likely includes changes to the > >> > > >>>>>> Maven > >> > > >>>>>> build configuration to accommodate JDK 17, ensuring that the > >> > > >>>>>> build process > >> > > >>>>>> is smooth and compatible. > >> > > >>>>>> > >> > > >>>>>> Overall, this PR is a comprehensive effort to bring Apache Hive > >> > > >>>>>> up to > >> > > >>>>>> date with the latest Java platform, ensuring long-term > >> > > >>>>>> maintainability and > >> > > >>>>>> performance improvements. > >> > > >>>>>> > >> > > >>>>>> As the next steps, we’ll be focusing on further sanity checks > >> > > >>>>>> and > >> > > >>>>>> extensive testing to make sure everything functions smoothly. > >> > > >>>>>> Community > >> > > >>>>>> involvement in this process will be crucial, so please do share > >> > > >>>>>> any issues > >> > > >>>>>> or feedback as you test the changes. > >> > > >>>>>> > >> > > >>>>>> I’m looking forward to seeing the benefits this will bring to > >> > > >>>>>> our > >> > > >>>>>> community and users. > >> > > >>>>>> > >> > > >>>>>> > >> > > >>>>>> [1] https://github.com/apache/hive/pull/5404 > >> > > >>>>>> > >> > > >>>>>> [2] > >> > > >>>>>> https://github.com/datanucleus/datanucleus-rdbms/commit/b99df3c803862f9e9feacc7891461bcb21160b38 > >> > > >>>>>> > >> > > >>>>>> [3] > >> > > >>>>>> https://github.com/datanucleus/datanucleus-rdbms/commit/b99df3c803862f9e9feacc7891461bcb21160b38 > >> > > >>>>>> > >> > > >>>>>> > >> > > >>>>>> Best regards, > >> > > >>>>>> > >> > > >>>>>> Akshat Mathur > >> > > >>>>>> > >> > > >>>>>> > >> > > >>>>>> On Tue, 11 Jul 2023 at 2:25 PM, Attila Turoczy < > >> > > >>>>>> aturo...@cloudera.com.invalid> wrote: > >> > > >>>>>> > >> > > >>>>>>> Returning to this topic, I kindly request those who would like > >> > > >>>>>>> to > >> > > >>>>>>> advocate > >> > > >>>>>>> for the continued support of JDK8 to please share their > >> > > >>>>>>> reasoning and > >> > > >>>>>>> insights with us. Your input and perspective are greatly > >> > > >>>>>>> appreciated! > >> > > >>>>>>> Thank you. > >> > > >>>>>>> > >> > > >>>>>>> -Attila > >> > > >>>>>>> > >> > > >>>>>>> On Fri, Jun 2, 2023 at 12:43 PM Attila Turoczy < > >> > > >>>>>>> aturo...@cloudera.com> > >> > > >>>>>>> wrote: > >> > > >>>>>>> > >> > > >>>>>>> > Hi All, > >> > > >>>>>>> > > >> > > >>>>>>> > I know my opinion might not be the most popular, but I > >> > > >>>>>>> > advocate > >> > > >>>>>>> for using* > >> > > >>>>>>> > JDK 17*. Here's why: > >> > > >>>>>>> > > >> > > >>>>>>> > Let's consider a scenario where a customer wants to use the > >> > > >>>>>>> > latest > >> > > >>>>>>> version > >> > > >>>>>>> > of Apache Hive. They would typically install it locally or > >> > > >>>>>>> > on a > >> > > >>>>>>> small > >> > > >>>>>>> > cluster. In 2023, is it realistic to assume that this > >> > > >>>>>>> > customer > >> > > >>>>>>> won't be > >> > > >>>>>>> > able to install JDK 17 on their cluster? Even in large > >> > > >>>>>>> enterprises, it > >> > > >>>>>>> > should be feasible to install an LTS JDK, especially > >> > > >>>>>>> > considering > >> > > >>>>>>> the > >> > > >>>>>>> > widespread adoption of cloud computing. Sungwoo Park's > >> > > >>>>>>> measurements also > >> > > >>>>>>> > support this recommendation to go with JDK 17. It > >> > > >>>>>>> > outperforms JDK > >> > > >>>>>>> 11 by 8% > >> > > >>>>>>> > in terms of runtime speed, and JDK 11 itself is 10+% faster > >> > > >>>>>>> > than > >> > > >>>>>>> JDK 8. > >> > > >>>>>>> > This is a significant value proposition. Who would be the > >> > > >>>>>>> > customer > >> > > >>>>>>> that > >> > > >>>>>>> > says, "I don't want faster query execution! I'd rather use > >> > > >>>>>>> > JDK 8 > >> > > >>>>>>> and pay > >> > > >>>>>>> > more for cloud or data center resources instead of using JDK > >> > > >>>>>>> > 17!" > >> > > >>>>>>> It > >> > > >>>>>>> > doesn't make sense to me. > >> > > >>>>>>> > > >> > > >>>>>>> > The tech industry has been evolving at an incredible pace, > >> > > >>>>>>> > with > >> > > >>>>>>> > improvements in serialization, IPC mechanisms, and > >> > > >>>>>>> > parallelized > >> > > >>>>>>> frameworks > >> > > >>>>>>> > since the release of JDK 8 ten years ago. We should leverage > >> > > >>>>>>> > these > >> > > >>>>>>> > advancements! Couple years ago, we invested a lot to improve > >> > > >>>>>>> > 1-2% > >> > > >>>>>>> of the > >> > > >>>>>>> > execution. We prayed for 3 gods, sacrificed 2 ships and > >> > > >>>>>>> > traveled > >> > > >>>>>>> around the > >> > > >>>>>>> > world to make it happen. :-) Now, the JDK itself provides a > >> > > >>>>>>> substantial > >> > > >>>>>>> > amount of improvement. So, why would we resist progress just > >> > > >>>>>>> because there > >> > > >>>>>>> > are a few lazy or conservative admins who don't want to > >> > > >>>>>>> > spend two > >> > > >>>>>>> minutes > >> > > >>>>>>> > installing a JDK? > >> > > >>>>>>> > > >> > > >>>>>>> > A platform needs to be modern and incorporate the latest > >> > > >>>>>>> technologies to > >> > > >>>>>>> > attract developers and users. I understand that some may > >> > > >>>>>>> > prefer to > >> > > >>>>>>> stay > >> > > >>>>>>> > with JDK 8 as it seems like the safest position, but I > >> > > >>>>>>> > believe in > >> > > >>>>>>> taking > >> > > >>>>>>> > bold bets to achieve big wins. Even if we decide to stick > >> > > >>>>>>> > with JDK > >> > > >>>>>>> 8, I > >> > > >>>>>>> > would still be happy since we are moving forward and not > >> > > >>>>>>> > dwelling > >> > > >>>>>>> on a JDK > >> > > >>>>>>> > that is a decade old. Personally, I think focusing on one > >> > > >>>>>>> > thing > >> > > >>>>>>> that brings > >> > > >>>>>>> > more value to us and our users is the idealistic path > >> > > >>>>>>> > forward. > >> > > >>>>>>> > > >> > > >>>>>>> > -Attila > >> > > >>>>>>> > > >> > > >>>>>>> > > >> > > >>>>>>> > > >> > > >>>>>>> > > >> > > >>>>>>> > > >> > > >>>>>>> > > >> > > >>>>>>> > > >> > > >>>>>>> > > >> > > >>>>>>> > > >> > > >>>>>>> > On Thu, Jun 1, 2023 at 11:23 AM Stamatis Zampetakis < > >> > > >>>>>>> zabe...@gmail.com> > >> > > >>>>>>> > wrote: > >> > > >>>>>>> > > >> > > >>>>>>> >> Hey everyone, > >> > > >>>>>>> >> > >> > > >>>>>>> >> If we claim that Hive supports a certain JDK then we should > >> > > >>>>>>> compile and > >> > > >>>>>>> >> run > >> > > >>>>>>> >> tests with it. > >> > > >>>>>>> >> > >> > > >>>>>>> >> The more JDKs we can support the better for everyone but > >> > > >>>>>>> >> this > >> > > >>>>>>> comes at a > >> > > >>>>>>> >> cost (resources mostly). We should have a precommit run for > >> > > >>>>>>> >> every > >> > > >>>>>>> >> supported > >> > > >>>>>>> >> JDK (frequency to be determined once per day/week) that > >> > > >>>>>>> >> compiles > >> > > >>>>>>> and run > >> > > >>>>>>> >> all tests. > >> > > >>>>>>> >> > >> > > >>>>>>> >> From my perspective, I would be pretty happy if we could > >> > > >>>>>>> >> cover > >> > > >>>>>>> the two > >> > > >>>>>>> >> edge > >> > > >>>>>>> >> LTS releases at every point in time. > >> > > >>>>>>> >> > >> > > >>>>>>> >> Then we have to decide also which JDK shall we use for the > >> > > >>>>>>> >> pull > >> > > >>>>>>> requests > >> > > >>>>>>> >> and local dev environment. I think it makes sense to use the > >> > > >>>>>>> latest. > >> > > >>>>>>> >> People > >> > > >>>>>>> >> like working on modern stuff and also it makes sense that > >> > > >>>>>>> >> newer > >> > > >>>>>>> releases > >> > > >>>>>>> >> will also use newer versions. It would be pretty awkward if > >> > > >>>>>>> someone wants > >> > > >>>>>>> >> to use the latest Hive version and it turns out that it can > >> > > >>>>>>> >> only > >> > > >>>>>>> run on > >> > > >>>>>>> >> JDK8. > >> > > >>>>>>> >> > >> > > >>>>>>> >> Best, > >> > > >>>>>>> >> Stamatis > >> > > >>>>>>> >> > >> > > >>>>>>> >> On Thu, Jun 1, 2023, 3:42 AM Sungwoo Park > >> > > >>>>>>> >> <glap...@gmail.com> > >> > > >>>>>>> wrote: > >> > > >>>>>>> >> > >> > > >>>>>>> >> > Hi, everyone. > >> > > >>>>>>> >> > > >> > > >>>>>>> >> > I have not tested the master branch with Java 11/17 yet, > >> > > >>>>>>> >> > but I > >> > > >>>>>>> would > >> > > >>>>>>> >> like > >> > > >>>>>>> >> > to share my experience with testing a fork of branch-3.1 > >> > > >>>>>>> >> > with > >> > > >>>>>>> Java 11/17 > >> > > >>>>>>> >> > (as part of developing Hive-MR3), in case that it can be > >> > > >>>>>>> >> > useful > >> > > >>>>>>> for the > >> > > >>>>>>> >> > discussion. I merged the patches listed in [1] HIVE-22415 > >> > > >>>>>>> >> > and > >> > > >>>>>>> updated > >> > > >>>>>>> >> the > >> > > >>>>>>> >> > Maven configuration for Java 11. > >> > > >>>>>>> >> > > >> > > >>>>>>> >> > 1. Building Hive was fine and I was able to run it with > >> > > >>>>>>> >> > Java 11 > >> > > >>>>>>> as well > >> > > >>>>>>> >> as > >> > > >>>>>>> >> > Java 17. So, it seems that the work reported in [1] is > >> > > >>>>>>> >> > indeed > >> > > >>>>>>> complete > >> > > >>>>>>> >> for > >> > > >>>>>>> >> > upgrading to Java 11 (and Java 17) and getting Hive to > >> > > >>>>>>> >> > work. > >> > > >>>>>>> >> > > >> > > >>>>>>> >> > 2. However, there was a problem with running tests, so > >> > > >>>>>>> >> > this can > >> > > >>>>>>> be > >> > > >>>>>>> >> > additional work for upgrading to Java 11. > >> > > >>>>>>> >> > > >> > > >>>>>>> >> > 3. For performance, Java 17 gives about 8 percent of > >> > > >>>>>>> >> > (free) > >> > > >>>>>>> performance > >> > > >>>>>>> >> > improvement. When tested with 10TB TPC-DS, Java 8 takes > >> > > >>>>>>> >> > 8074 > >> > > >>>>>>> seconds, > >> > > >>>>>>> >> > whereas Java 17 takes 7415 seconds. Considering the > >> > > >>>>>>> >> > maturity of > >> > > >>>>>>> Hive, I > >> > > >>>>>>> >> > think this is not a small improvement because almost every > >> > > >>>>>>> query gets > >> > > >>>>>>> >> some > >> > > >>>>>>> >> > speedup. > >> > > >>>>>>> >> > > >> > > >>>>>>> >> > Thanks, > >> > > >>>>>>> >> > > >> > > >>>>>>> >> > --- Sungwoo > >> > > >>>>>>> >> > > >> > > >>>>>>> >> > [1] https://issues.apache.org/jira/browse/HIVE-22415 > >> > > >>>>>>> >> > > >> > > >>>>>>> >> > > >> > > >>>>>>> >> > On Thu, Jun 1, 2023 at 3:53 AM Sai Hemanth Gantasala > >> > > >>>>>>> >> > <saihema...@cloudera.com.invalid> wrote: > >> > > >>>>>>> >> > > >> > > >>>>>>> >> > > Hi All, > >> > > >>>>>>> >> > > > >> > > >>>>>>> >> > > I would strongly advocate keeping support for JDK8. > >> > > >>>>>>> >> > > Between JDK11 and JDK17, Depending on the amount of > >> > > >>>>>>> >> > > effort on > >> > > >>>>>>> the > >> > > >>>>>>> >> upgrade > >> > > >>>>>>> >> > > I'm inclined towards JDK17 (JDK21 LTS will be released > >> > > >>>>>>> >> > > in Sep > >> > > >>>>>>> 2023). > >> > > >>>>>>> >> > > > >> > > >>>>>>> >> > > Thanks, > >> > > >>>>>>> >> > > Sai. > >> > > >>>>>>> >> > > > >> > > >>>>>>> >> > > On Wed, May 31, 2023 at 5:39 AM László Bodor < > >> > > >>>>>>> >> bodorlaszlo0...@gmail.com> > >> > > >>>>>>> >> > > wrote: > >> > > >>>>>>> >> > > > >> > > >>>>>>> >> > > > *Hi!* > >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > *Should we support both JDK-11 & JDK-8?* > >> > > >>>>>>> >> > > > IMO absolutely yes, let's not break up with JDK-8: > >> > > >>>>>>> according to its > >> > > >>>>>>> >> > > > lifecycle, it's going to stay with us for a long time. > >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > I believe > >> > > >>>>>>> >> > > > a) we should be able to compile on JDK8, JDK11, and > >> > > >>>>>>> >> > > > JDK17 > >> > > >>>>>>> (github > >> > > >>>>>>> >> > actions > >> > > >>>>>>> >> > > > can cover this conveniently in precommit time, like > >> > > >>>>>>> >> > > > tez > >> > > >>>>>>> >> > > > < > >> > > >>>>>>> >> > >> > > >>>>>>> https://github.com/apache/tez/blob/master/.github/workflows/build.yml > >> > > >>>>>>> >> > >) > >> > > >>>>>>> >> > > > b) the release artifacts should be compatible with > >> > > >>>>>>> >> > > > JDK8 as > >> > > >>>>>>> long as > >> > > >>>>>>> >> it > >> > > >>>>>>> >> > is > >> > > >>>>>>> >> > > > with us. > >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > Regards, > >> > > >>>>>>> >> > > > Laszlo Bodor > >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > Butao Zhang <butaozha...@163.com> ezt írta (időpont: > >> > > >>>>>>> >> > > > 2023. > >> > > >>>>>>> máj. > >> > > >>>>>>> >> 31., > >> > > >>>>>>> >> > > Sze, > >> > > >>>>>>> >> > > > 14:33): > >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > > Thanks Ayush for driving this! Good to know that > >> > > >>>>>>> >> > > > > Hive is > >> > > >>>>>>> getting > >> > > >>>>>>> >> > ready > >> > > >>>>>>> >> > > > for > >> > > >>>>>>> >> > > > > newer JDK. > >> > > >>>>>>> >> > > > > From my opinon, if we have more community energy to > >> > > >>>>>>> >> > > > > put > >> > > >>>>>>> into it, > >> > > >>>>>>> >> we > >> > > >>>>>>> >> > can > >> > > >>>>>>> >> > > > > support both JDK-11 and JDK-17 like Spark[1]. If we > >> > > >>>>>>> >> > > > > have > >> > > >>>>>>> to make > >> > > >>>>>>> >> a > >> > > >>>>>>> >> > > > choice > >> > > >>>>>>> >> > > > > between a JDK-11 and JDK-17, i would like to choose > >> > > >>>>>>> >> > > > > the > >> > > >>>>>>> relatively > >> > > >>>>>>> >> > new > >> > > >>>>>>> >> > > > > version JDK-17, meanwhile, we should maintain > >> > > >>>>>>> compatibility with > >> > > >>>>>>> >> > jdk8, > >> > > >>>>>>> >> > > as > >> > > >>>>>>> >> > > > > JDK-8 is still widely used in most big data > >> > > >>>>>>> >> > > > > platforms. > >> > > >>>>>>> >> > > > > > >> > > >>>>>>> >> > > > > > >> > > >>>>>>> >> > > > > Thanks, > >> > > >>>>>>> >> > > > > Butao Zhang > >> > > >>>>>>> >> > > > > > >> > > >>>>>>> >> > > > > > >> > > >>>>>>> >> > > > > [1]https://issues.apache.org/jira/browse/SPARK-33772 > >> > > >>>>>>> >> > > > > ---- Replied Message ---- > >> > > >>>>>>> >> > > > > | From | Ayush Saxena<ayush...@gmail.com> | > >> > > >>>>>>> >> > > > > | Date | 5/31/2023 18:39 | > >> > > >>>>>>> >> > > > > | To | dev<dev@hive.apache.org> | > >> > > >>>>>>> >> > > > > | Subject | Move to JDK-11 | > >> > > >>>>>>> >> > > > > Hi Everyone, > >> > > >>>>>>> >> > > > > Want to pull in the attention of folks towards > >> > > >>>>>>> >> > > > > moving to > >> > > >>>>>>> JDK-11 > >> > > >>>>>>> >> > compile > >> > > >>>>>>> >> > > > > time support in Hive. There was a ticket in the > >> > > >>>>>>> >> > > > > past [1] > >> > > >>>>>>> which > >> > > >>>>>>> >> talks > >> > > >>>>>>> >> > > > about > >> > > >>>>>>> >> > > > > it and If I could decode it right, it was blocked > >> > > >>>>>>> >> > > > > because > >> > > >>>>>>> the > >> > > >>>>>>> >> Hadoop > >> > > >>>>>>> >> > > > > version used by Hive didn't had JDK-11 runtime > >> > > >>>>>>> >> > > > > support, > >> > > >>>>>>> But with > >> > > >>>>>>> >> [2] > >> > > >>>>>>> >> > in > >> > > >>>>>>> >> > > > we > >> > > >>>>>>> >> > > > > have upgraded the Hadoop version, so that problem is > >> > > >>>>>>> sorted out. I > >> > > >>>>>>> >> > > > couldn't > >> > > >>>>>>> >> > > > > even see any unresolved tickets in the blocked state > >> > > >>>>>>> either. > >> > > >>>>>>> >> > > > > > >> > > >>>>>>> >> > > > > I quickly tried* a mvn clean install -DskipTests > >> > > >>>>>>> -Piceberg > >> > > >>>>>>> >> -Pitests > >> > > >>>>>>> >> > > > > -Dmaven.javadoc.skip=true > >> > > >>>>>>> >> > > > > > >> > > >>>>>>> >> > > > > And no surprises it failed with some weird > >> > > >>>>>>> >> > > > > exceptions > >> > > >>>>>>> towards the > >> > > >>>>>>> >> > end. > >> > > >>>>>>> >> > > > But > >> > > >>>>>>> >> > > > > I think that should be solvable. > >> > > >>>>>>> >> > > > > > >> > > >>>>>>> >> > > > > So, Questions? > >> > > >>>>>>> >> > > > > > >> > > >>>>>>> >> > > > > - What do folks think about this? Should we put in > >> > > >>>>>>> >> > > > > some > >> > > >>>>>>> effort > >> > > >>>>>>> >> > towards > >> > > >>>>>>> >> > > > > JDK-11 > >> > > >>>>>>> >> > > > > - Should we support both JDK-11 & JDK-8? > >> > > >>>>>>> >> > > > > - Ditch JDK-11 and directly shoot for JDK-17? > >> > > >>>>>>> >> > > > > > >> > > >>>>>>> >> > > > > Let me know your thoughts, In case anyone has some > >> > > >>>>>>> experience in > >> > > >>>>>>> >> this > >> > > >>>>>>> >> > > > area > >> > > >>>>>>> >> > > > > and have tried something in the context, feel free > >> > > >>>>>>> >> > > > > to > >> > > >>>>>>> share or > >> > > >>>>>>> >> may be > >> > > >>>>>>> >> > > if > >> > > >>>>>>> >> > > > > someone has any potential action plan or so > >> > > >>>>>>> >> > > > > > >> > > >>>>>>> >> > > > > -Ayush > >> > > >>>>>>> >> > > > > > >> > > >>>>>>> >> > > > > [1] https://issues.apache.org/jira/browse/HIVE-22415 > >> > > >>>>>>> >> > > > > [2] https://issues.apache.org/jira/browse/HIVE-24484 > >> > > >>>>>>> >> > > > > > >> > > >>>>>>> >> > > > > * changed the maven.compiler.source & > >> > > >>>>>>> maven.compiler.target to 11 > >> > > >>>>>>> >> > > > > > >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > >> > > >>>>>>> >> > > >> > > >>>>>>> >> > >> > > >>>>>>> > > >> > > >>>>>>> > >> > > >>>>>> > >> > > > >> >
