I understand that, and we do so, but news about official images were breaking me ))))
Ok, I will follow you on those activities. Thanks for the quick response. On Tue, 22 Feb 2022 at 22:03, Holden Karau <hol...@pigscanfly.ca> wrote: > So your more than welcome to still build your own Spark docker containers > with the docker image tool, these are provided to make it easier for folks > without specific needs. In the future well hopefully have published Spark > containers tagged for different JDKs but that work has not yet been done. > > On Tue, Feb 22, 2022 at 10:51 AM Denis Bolshakov < > bolshakov.de...@gmail.com> wrote: > >> Hello Holden, >> >> Could you please provide more details and plan for docker images support? >> >> So far I see that there are only two tags, I get from them spark version, >> but there is no information about java, hadoop, scala versions. >> >> Also there is no description on docker hub, probably it would be nice to >> put a link to Docker files in github repository. >> >> What directories are expected to be mounted and ports forwarded? How can >> I mount the krb5.conf file and directory where my kerberos ticket is >> located? >> >> I've pulled the docker image with tag spark 3.2.1 and I see that there is >> java 11 and hadoop 3.3, but our environment requires us to have other >> versions. >> >> On Tue, 22 Feb 2022 at 16:29, Mich Talebzadeh <mich.talebza...@gmail.com> >> wrote: >> >>> Well that is just a recommendation. >>> >>> The onus is on me the user to download and go through dev and test >>> running suite of batch jobs to ensure that all work ok, especially on the >>> edge, sign the release off and roll it in out into production. It won’t be >>> prudent otherwise. >>> >>> HHH >>> >>> On Tue, 22 Feb 2022 at 12:12, Bjørn Jørgensen <bjornjorgen...@gmail.com> >>> wrote: >>> >>>> "Spark 3.1.3 is a maintenance release containing stability fixes. This >>>> release is based on the branch-3.1 maintenance branch of Spark. We strongly >>>> recommend all 3.1.3 users to upgrade to this stable release." >>>> https://spark.apache.org/releases/spark-release-3-1-3.html >>>> >>>> Do we have another 3.13 or do we strongly recommend all 3.1.2 users to >>>> upgrade to this stable release ? >>>> >>>> tir. 22. feb. 2022 kl. 09:50 skrev angers zhu <angers....@gmail.com>: >>>> >>>>> Hi, seems >>>>> >>>>> - [SPARK-35391] <https://issues.apache.org/jira/browse/SPARK-36339>: >>>>> Memory leak in ExecutorAllocationListener breaks dynamic allocation >>>>> under >>>>> high load >>>>> >>>>> Links to wrong jira ticket? >>>>> >>>>> Mich Talebzadeh <mich.talebza...@gmail.com> 于2022年2月22日周二 15:49写道： >>>>> >>>>>> Well, that is pretty easy to do. >>>>>> >>>>>> However, a quick fix for now could be to retag the image created. It >>>>>> is a small volume which can be done manually for now. For example, I just >>>>>> downloaded v3.1.3 >>>>>> >>>>>> >>>>>> docker image ls >>>>>> >>>>>> REPOSITORY TAG >>>>>> IMAGE ID CREATED SIZE >>>>>> >>>>>> apache/spark v3.1.3 >>>>>> 31ed15daa2bf 12 hours ago 531MB >>>>>> >>>>>> Retag it with >>>>>> >>>>>> >>>>>> docker tag 31ed15daa2bf >>>>>> apache/spark/tags/spark-3.1.3-scala_2.12-8-jre-slim-buster >>>>>> >>>>>> docker image ls >>>>>> >>>>>> REPOSITORY TAG >>>>>> IMAGE ID CREATED >>>>>> SIZE >>>>>> >>>>>> apache/spark/tags/spark-3.1.3-scala_2.12-8-jre-slim-buster latest >>>>>> 31ed15daa2bf 12 hours >>>>>> ago >>>>>> 531MB >>>>>> >>>>>> Then push it with (example) >>>>>> >>>>>> docker push apache/spark/tags/spark-3.1.3-scala_2.12-8-jre-slim-buster >>>>>> >>>>>> >>>>>> HTH >>>>>> >>>>>> >>>>>> view my Linkedin profile >>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>> >>>>>> >>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>> >>>>>> >>>>>> >>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility >>>>>> for any loss, damage or destruction of data or any other property which >>>>>> may >>>>>> arise from relying on this email's technical content is explicitly >>>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>>> arising from such loss, damage or destruction. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mon, 21 Feb 2022 at 23:51, Holden Karau <hol...@pigscanfly.ca> >>>>>> wrote: >>>>>> >>>>>>> Yeah I think we should still adopt that naming convention, however >>>>>>> no one has taken the time submit write a script to do it yet so until we >>>>>>> get that script merged I think we'll just have one build. I can try and >>>>>>> do >>>>>>> that for the next release but it would be a great 2nd issue for someone >>>>>>> getting more familiar with the release tooling. >>>>>>> >>>>>>> On Mon, Feb 21, 2022 at 2:18 PM Mich Talebzadeh < >>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>> >>>>>>>> Ok thanks for the correction. >>>>>>>> >>>>>>>> The docker pull line shows as follows: >>>>>>>> >>>>>>>> docker pull apache/spark:v3.2.1 >>>>>>>> >>>>>>>> >>>>>>>> So this only tells me the version of Spark 3.2.1 >>>>>>>> >>>>>>>> >>>>>>>> I thought we discussed deciding on the docker naming conventions in >>>>>>>> detail, and broadly agreed on what needs to be in the naming >>>>>>>> convention. >>>>>>>> For example, in this thread: >>>>>>>> >>>>>>>> >>>>>>>> Time to start publishing Spark Docker Images? - >>>>>>>> mich.talebza...@gmail.com - Gmail (google.com) >>>>>>>> <https://mail.google.com/mail/u/0/?hl=en-GB#search/publishing/FMfcgzGkZQSzbXWQDWfddGDNRDQfPCpg> >>>>>>>> dated >>>>>>>> 22nd July 2021 >>>>>>>> >>>>>>>> >>>>>>>> Referring to that, I think the broad agreement was that the docker >>>>>>>> image name should be of the form: >>>>>>>> >>>>>>>> >>>>>>>> The name of the file provides: >>>>>>>> >>>>>>>> - Built for spark or spark-py (PySpark) spark-r >>>>>>>> - Spark version: 3.1.1, 3.1.2, 3.2.1 etc. >>>>>>>> - Scala version; 2.1.2 >>>>>>>> - The OS version based on JAVA: 8-jre-slim-buster, >>>>>>>> 11-jre-slim-buster meaning JAVA 8 and JAVA 11 respectively >>>>>>>> >>>>>>>> I believe it is a good thing and we ought to adopt that convention. >>>>>>>> For example: >>>>>>>> >>>>>>>> >>>>>>>> spark-py-3.2.1-scala_2.12-11-jre-slim-buster >>>>>>>> >>>>>>>> >>>>>>>> HTH >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> view my Linkedin profile >>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>> >>>>>>>> >>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility >>>>>>>> for any loss, damage or destruction of data or any other property >>>>>>>> which may >>>>>>>> arise from relying on this email's technical content is explicitly >>>>>>>> disclaimed. The author will in no case be liable for any monetary >>>>>>>> damages >>>>>>>> arising from such loss, damage or destruction. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, 21 Feb 2022 at 21:58, Holden Karau <hol...@pigscanfly.ca> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> My bad, the correct link is: >>>>>>>>> >>>>>>>>> https://hub.docker.com/r/apache/spark/tags >>>>>>>>> >>>>>>>>> On Mon, Feb 21, 2022 at 1:17 PM Mich Talebzadeh < >>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> well that docker link is not found! may be permission issue >>>>>>>>>> >>>>>>>>>> [image: image.png] >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> view my Linkedin profile >>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Disclaimer:* Use it at your own risk. Any and all >>>>>>>>>> responsibility for any loss, damage or destruction of data or any >>>>>>>>>> other >>>>>>>>>> property which may arise from relying on this email's technical >>>>>>>>>> content is >>>>>>>>>> explicitly disclaimed. The author will in no case be liable for any >>>>>>>>>> monetary damages arising from such loss, damage or destruction. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, 21 Feb 2022 at 21:09, Holden Karau <hol...@pigscanfly.ca> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> We are happy to announce the availability of Spark 3.1.3! >>>>>>>>>>> >>>>>>>>>>> Spark 3.1.3 is a maintenance release containing stability fixes. >>>>>>>>>>> This >>>>>>>>>>> release is based on the branch-3.1 maintenance branch of Spark. >>>>>>>>>>> We strongly >>>>>>>>>>> recommend all 3.1 users to upgrade to this stable release. >>>>>>>>>>> >>>>>>>>>>> To download Spark 3.1.3, head over to the download page: >>>>>>>>>>> https://spark.apache.org/downloads.html >>>>>>>>>>> >>>>>>>>>>> To view the release notes: >>>>>>>>>>> https://spark.apache.org/releases/spark-release-3-1-3.html >>>>>>>>>>> >>>>>>>>>>> We would like to acknowledge all community members for >>>>>>>>>>> contributing to this >>>>>>>>>>> release. This release would not have been possible without you. >>>>>>>>>>> >>>>>>>>>>> *New Dockerhub magic in this release:* >>>>>>>>>>> >>>>>>>>>>> We've also started publishing docker containers to the Apache >>>>>>>>>>> Dockerhub, >>>>>>>>>>> these contain non-ASF artifacts that are subject to different >>>>>>>>>>> license terms than the >>>>>>>>>>> Spark release. The docker containers are built for Linux x86 and >>>>>>>>>>> ARM64 since that's >>>>>>>>>>> what I have access to (thanks to NV for the ARM64 machines). >>>>>>>>>>> >>>>>>>>>>> You can get them from https://hub.docker.com/apache/spark (and >>>>>>>>>>> spark-r and spark-py) :) >>>>>>>>>>> (And version 3.2.1 is also now published on Dockerhub). >>>>>>>>>>> >>>>>>>>>>> Holden >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>> >>>>>> >>>> >>>> -- >>>> Bjørn Jørgensen >>>> Vestre Aspehaug 4 >>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4?entry=gmail&source=g>, >>>> 6010 Ålesund >>>> Norge >>>> >>>> +47 480 94 297 >>>> >>> -- >>> >>> >>> >>> view my Linkedin profile >>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>> >>> >>> https://en.everybodywiki.com/Mich_Talebzadeh >>> >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >> >> >> -- >> //with Best Regards >> --Denis Bolshakov >> e-mail: bolshakov.de...@gmail.com >> > -- > Twitter: https://twitter.com/holdenkarau > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > -- //with Best Regards --Denis Bolshakov e-mail: bolshakov.de...@gmail.com
