Hi Mark, I created a spark3.4.1 docker file. Details from spark-py-3.4.1-scala_2.12-11-jre-slim-buster <https://hub.docker.com/repository/docker/michtalebzadeh/spark_dockerfiles/tags?page=1&ordering=last_updated>
Pull instructions are given docker pull michtalebzadeh/spark_dockerfiles:spark-py-3.4.1-scala_2.12-11-jre-slim-buster It is 3.4.1 spark-py with no extra python packages you can tag it as you wish login to it as below docker run -it michtalebzadeh/spark_dockerfiles:spark-py-3.4.1-scala_2.12-11-jre-slim-buster bash 185@b031a15c6730:/opt/spark/work-dir$ pip list Package Version ------------- ------- asn1crypto 0.24.0 cryptography 2.6.1 entrypoints 0.3 keyring 17.1.1 keyrings.alt 3.1.1 pip 23.2.1 pycrypto 2.6.1 PyGObject 3.30.4 pyxdg 0.25 SecretStorage 2.3.1 setuptools 68.0.0 six 1.12.0 wheel 0.32.3 $SPARK_HOME/bin/spark-submit --version Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 3.4.1 /_/ Using Scala version 2.12.17, OpenJDK 64-Bit Server VM, 11.0.11 Branch HEAD Compiled by user centos on 2023-06-19T23:01:01Z Revision 6b1ff22dde1ead51cbf370be6e48a802daae58b6 Url https://github.com/apache/spark Built on java 11 185@b031a15c6730:/opt/spark/work-dir$ java --version openjdk 11.0.11 2021-04-20 OpenJDK Runtime Environment 18.9 (build 11.0.11+9) OpenJDK 64-Bit Server VM 18.9 (build 11.0.11+9, mixed mode, sharing) HTH Mich Talebzadeh, Solutions Architect/Engineering Lead London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Wed, 9 Aug 2023 at 17:41, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Hi Mark, > > you can build it yourself, no big deal :) > > REPOSITORY TAG > IMAGE ID CREATED > SIZE > sparkpy/spark-py > 3.4.1-scala_2.12-11-jre-slim-buster-Dockerfile a876102b2206 1 > second ago 1.09GB > sparkpy/spark > 3.4.1-scala_2.12-11-jre-slim-buster-Dockerfile 6f74f7475e01 3 > minutes ago 695MB > > Based on > > ARG java_image_tag=11-jre-slim ## java 11 > FROM openjdk:${java_image_tag} > > BASE_OS="buster" > SPARK_VERSION="3.4.1" > SCALA_VERSION="scala_2.12" > DOCKERFILE="Dockerfile" > DOCKERIMAGETAG="11-jre-slim" > > You need to modify the file > > $SPARK_HOME/kubernetes/dockerfiles/spark/Dockerfile > > and replace > > #ARG java_image_tag=17-jre > #FROM eclipse-temurin:${java_image_tag} > > With > > ARG java_image_tag=11-jre-slim > FROM openjdk:${java_image_tag} > > Which is Java 11 > > HTH > > > Mich Talebzadeh, > Solutions Architect/Engineering Lead > London > United Kingdom > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Wed, 9 Aug 2023 at 16:43, Mark Elliot <m...@theoremlp.com.invalid> > wrote: > >> Hello, >> >> I noticed that the apache/spark-py image for Spark's 3.4.1 release is not >> available (apache/spark@3.4.1 is available). Would it be possible to get >> the 3.4.1 release build for the apache/spark-py image published? >> >> Thanks, >> >> Mark >> >> ------------------------------ >> >> This communication, together with any attachments, is intended only for >> the addressee(s) and may contain confidential, privileged or proprietary >> information of Theorem Partners LLC ("Theorem"). By accepting this >> communication you agree to keep confidential all information contained in >> this communication, as well as any information derived by you from the >> confidential information contained in this communication. Theorem does not >> waive any confidentiality by misdelivery. >> >> If you receive this communication in error, any use, dissemination, >> printing or copying of all or any part of it is strictly prohibited; please >> destroy all electronic and paper copies and notify the sender immediately. >> Nothing in this email is intended to constitute (1) investment, legal or >> tax advice, (2) any recommendation to purchase or sell any security, (3) >> any advertisement or offer of advisory services or (4) any offer to sell or >> solicitation of an offer to buy any securities or other financial >> instrument in any jurisdiction. >> >> Theorem, including its agents or affiliates, reserves the right to >> intercept, archive, monitor and review all communications to and from its >> network, including this email and any email response to it. >> >> Theorem makes no representation as to the accuracy or completeness of the >> information in this communication and does not accept liability for any >> errors or omissions in this communication, including any liability >> resulting from its transmission by email, and undertakes no obligation to >> update any information in this email or its attachments. >> >