[ https://issues.apache.org/jira/browse/SPARK-35954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-35954: ------------------------------------ Assignee: (was: Apache Spark) > [Deploy] Upgrade Apache Curator Dependency to 4.2.0 > --------------------------------------------------- > > Key: SPARK-35954 > URL: https://issues.apache.org/jira/browse/SPARK-35954 > Project: Spark > Issue Type: Dependency upgrade > Components: Deploy > Affects Versions: 3.1.2 > Environment: * OS: Linux > * JAVA: 1.8.0_292 > * {color:#FF0000}*hadoop-3.3.1*{color} > > Reporter: Nikita ROUSSEAU > Priority: Major > > +Abstract :+ as a Spark Cluster Administrator, I want to connect spark > masters deployed in HA/mode to Zookeeper over SSL/TLS, so that my network > traffic is ciphered between y components. > ([https://spark.apache.org/docs/latest/spark-standalone.html#standby-masters-with-zookeeper] > ) > > With the release of Hadoop 3.3.1, ZKFC libraries and their dependencies were > updated : it is now possible to connect ZKFC to ZooKeeper over TLS. > > +Note:+ TLS is possible with Zookeeper Server Version >= 3.5.6 > ([https://docs.confluent.io/platform/current/installation/versions-interoperability.html#zk] > ) > > Spark 3.2.0 aims to support Hadoop 3.3.1 ; this Hadoop release bundles the > following shared libraries : > * curator-client-4.2.0.jar > * curator-framework-4.2.0.jar > * curator-recipes-4.2.0.jar > * zookeeper-3.5.6.jar > * zookeeper-jute-3.5.6.jar > > Currently, Spark dependency is set to 2.13.0 for the Currator framework > ([https://github.com/apache/spark/blob/master/pom.xml#L127).|https://github.com/apache/spark/blob/master/pom.xml#L127)] > > It would be great to update "curator-*" dependencies to 4.2.0 in order to be > compatible with shared jars of the hadoop stack. > Moreover, it will allow administrators to connect Spark Masters to ZooKeeper > over TLS. > > Some patches will be required, such as : > * > [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkCuratorUtil.scala#L51] > > I will try to prepare a MR for this. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org