At least we should upgrade hadoop to the latest version https://hadoop.apache.org/release/2.10.2.html
Are there some spesial reasons why we have a hadoop version that is 7 years old? ons. 14. sep. 2022, 20:25 skrev Dongjoon Hyun <dongjoon.h...@gmail.com>: > Ya, +1 for Sean's comment. > > In addition, all Apache Spark's Maven artifacts are depending on Hadoop > 3.3.x already. > > > https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/3.3.0 > > https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.13/3.3.0 > > Apache Spark has been moving away from Hadoop 2 due to many many reasons. > > Dongjoon. > > > On Wed, Sep 14, 2022 at 10:54 AM Sean Owen <sro...@gmail.com> wrote: > >> Yeah we're not going to make convenience binaries for all possible >> combinations. It's a pretty good assumption that anyone moving to later >> Scala versions is also off old Hadoop versions. >> You can of course build the combo you like. >> >> On Wed, Sep 14, 2022 at 11:26 AM Denis Bolshakov < >> bolshakov.de...@gmail.com> wrote: >> >>> Unfortunately it's for hadoop 3 only. >>> >>> ср, 14 сент. 2022 г., 19:04 Dongjoon Hyun <dongjoon.h...@gmail.com>: >>> >>>> Hi, Denis. >>>> >>>> Apache Spark community already provides both Scala 2.12 and 2.13 >>>> pre-built distributions. >>>> Please check the distribution site and Apache Spark download page. >>>> >>>> https://dlcdn.apache.org/spark/spark-3.3.0/ >>>> >>>> spark-3.3.0-bin-hadoop3-scala2.13.tgz >>>> spark-3.3.0-bin-hadoop3.tgz >>>> >>>> [image: Screenshot 2022-09-14 at 9.03.27 AM.png] >>>> >>>> Dongjoon. >>>> >>>> On Wed, Sep 14, 2022 at 12:31 AM Denis Bolshakov < >>>> bolshakov.de...@gmail.com> wrote: >>>> >>>>> Hello, >>>>> >>>>> It would be great if it's possible to provide a spark distro for both >>>>> scala 2.12 and scala 2.13. >>>>> >>>>> It will encourage spark users to switch to scala 2.13. >>>>> >>>>> I know that spark jar artifacts available for both scala versions, but >>>>> it does not make sense to migrate to scala 2.13 while there is no spark >>>>> distro for this version. >>>>> >>>>> Kind regards, >>>>> Denis >>>>> >>>>> On Tue, 13 Sept 2022 at 17:38, Yuming Wang <wgy...@gmail.com> wrote: >>>>> >>>>>> Thank you all. >>>>>> >>>>>> I will be preparing 3.3.1 RC1 soon. >>>>>> >>>>>> On Tue, Sep 13, 2022 at 12:09 PM John Zhuge <jzh...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> +1 >>>>>>> >>>>>>> On Mon, Sep 12, 2022 at 9:08 PM Yang,Jie(INF) <yangji...@baidu.com> >>>>>>> wrote: >>>>>>> >>>>>>>> +1 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks Yuming ~ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *发件人**: *Hyukjin Kwon <gurwls...@gmail.com> >>>>>>>> *日期**: *2022年9月13日 星期二 08:19 >>>>>>>> *收件人**: *Gengliang Wang <ltn...@gmail.com> >>>>>>>> *抄送**: *"L. C. Hsieh" <vii...@gmail.com>, Dongjoon Hyun < >>>>>>>> dongjoon.h...@gmail.com>, Yuming Wang <wgy...@gmail.com>, dev < >>>>>>>> dev@spark.apache.org> >>>>>>>> *主题**: *Re: Time for Spark 3.3.1 release? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> +1 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, 13 Sept 2022 at 06:45, Gengliang Wang <ltn...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>> +1. >>>>>>>> >>>>>>>> Thank you, Yuming! >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Sep 12, 2022 at 12:10 PM L. C. Hsieh <vii...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>> +1 >>>>>>>> >>>>>>>> Thanks Yuming! >>>>>>>> >>>>>>>> On Mon, Sep 12, 2022 at 11:50 AM Dongjoon Hyun < >>>>>>>> dongjoon.h...@gmail.com> wrote: >>>>>>>> > >>>>>>>> > +1 >>>>>>>> > >>>>>>>> > Thanks, >>>>>>>> > Dongjoon. >>>>>>>> > >>>>>>>> > On Mon, Sep 12, 2022 at 6:38 AM Yuming Wang <wgy...@gmail.com> >>>>>>>> wrote: >>>>>>>> >> >>>>>>>> >> Hi, All. >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> Since Apache Spark 3.3.0 tag creation (Jun 10), new 138 patches >>>>>>>> including 7 correctness patches arrived at branch-3.3. >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> Shall we make a new release, Apache Spark 3.3.1, as the second >>>>>>>> release at branch-3.3? I'd like to volunteer as the release manager for >>>>>>>> Apache Spark 3.3.1. >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> All changes: >>>>>>>> >> >>>>>>>> >> https://github.com/apache/spark/compare/v3.3.0...branch-3.3 >>>>>>>> <https://mailshield.baidu.com/check?q=WzRnV6InLAPdBDRyJZecGtPwF02t%2bnFNwOI8oAyGcb60kX%2bRCS6N3SUnFjTdf47bb94KmZHTTKE%2bBHbIT27Rog%3d%3d> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> Correctness issues: >>>>>>>> >> >>>>>>>> >> SPARK-40149: Propagate metadata columns through Project >>>>>>>> >> >>>>>>>> >> SPARK-40002: Don't push down limit through window using ntile >>>>>>>> >> >>>>>>>> >> SPARK-39976: ArrayIntersect should handle null in left >>>>>>>> expression correctly >>>>>>>> >> >>>>>>>> >> SPARK-39833: Disable Parquet column index in DSv1 to fix a >>>>>>>> correctness issue in the case of overlapping partition and data columns >>>>>>>> >> >>>>>>>> >> SPARK-39061: Set nullable correctly for Inline output attributes >>>>>>>> >> >>>>>>>> >> SPARK-39887: RemoveRedundantAliases should keep aliases that >>>>>>>> make the output of projection nodes unique >>>>>>>> >> >>>>>>>> >> SPARK-38614: Don't push down limit through window that's using >>>>>>>> percent_rank >>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>>> >>>>>>>> -- >>>>>>> John Zhuge >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> //with Best Regards >>>>> --Denis Bolshakov >>>>> e-mail: bolshakov.de...@gmail.com >>>>> >>>>
