[SPARK-46662][K8S][BUILD] Upgrade kubernetes-client to 6.10.0 <https://github.com/apache/spark/pull/44672> a new version of kubernets-client with okio version 1.17.6 is now merged to master and will be in the spark 4.0 version.
tir. 14. nov. 2023 kl. 15:21 skrev Bjørn Jørgensen <bjornjorgen...@gmail.com >: > FYI > I have opened Update okio to version 1.17.6 > <https://github.com/fabric8io/kubernetes-client/pull/5587> for this now. > > tor. 31. aug. 2023 kl. 21:18 skrev Sean Owen <sro...@gmail.com>: > >> It's a dependency of some other HTTP library. Use mvn dependency:tree to >> see where it comes from. It may be more straightforward to upgrade the >> library that brings it in, assuming a later version brings in a later okio. >> You can also manage up the version directly with a new entry in >> <dependencyManagement> >> >> However, does this affect Spark? all else equal it doesn't hurt to >> upgrade, but wondering if there is even a theory that it needs to be >> updated. >> >> >> On Thu, Aug 31, 2023 at 7:42 AM Agrawal, Sanket < >> sankeagra...@deloitte.com> wrote: >> >>> I don’t see an entry in pom.xml while building spark. I think it is >>> being downloaded as part of some other dependency. >>> >>> >>> >>> *From:* Sean Owen <sro...@gmail.com> >>> *Sent:* Thursday, August 31, 2023 5:10 PM >>> *To:* Agrawal, Sanket <sankeagra...@deloitte.com> >>> *Cc:* user@spark.apache.org >>> *Subject:* [EXT] Re: Okio Vulnerability in Spark 3.4.1 >>> >>> >>> >>> Does the vulnerability affect Spark? >>> >>> In any event, have you tried updating Okio in the Spark build? I don't >>> believe you could just replace the JAR, as other libraries probably rely on >>> it and compiled against the current version. >>> >>> >>> >>> On Thu, Aug 31, 2023 at 6:02 AM Agrawal, Sanket < >>> sankeagra...@deloitte.com.invalid> wrote: >>> >>> Hi All, >>> >>> >>> >>> Amazon inspector has detected a vulnerability in okio-1.15.0.jar JAR in >>> Spark 3.4.1. It suggests to upgrade the jar version to 3.4.0. But when we >>> try this version of jar then the spark application is failing with below >>> error: >>> >>> >>> >>> py4j.protocol.Py4JJavaError: An error occurred while calling >>> None.org.apache.spark.api.java.JavaSparkContext. >>> >>> : java.lang.NoClassDefFoundError: okio/BufferedSource >>> >>> at okhttp3.internal.Util.<clinit>(Util.java:62) >>> >>> at okhttp3.OkHttpClient.<clinit>(OkHttpClient.java:127) >>> >>> at okhttp3.OkHttpClient$Builder.<init>(OkHttpClient.java:475) >>> >>> at >>> io.fabric8.kubernetes.client.okhttp.OkHttpClientFactory.newOkHttpClientBuilder(OkHttpClientFactory.java:41) >>> >>> at >>> io.fabric8.kubernetes.client.okhttp.OkHttpClientFactory.newBuilder(OkHttpClientFactory.java:56) >>> >>> at >>> io.fabric8.kubernetes.client.okhttp.OkHttpClientFactory.newBuilder(OkHttpClientFactory.java:68) >>> >>> at >>> io.fabric8.kubernetes.client.okhttp.OkHttpClientFactory.newBuilder(OkHttpClientFactory.java:30) >>> >>> at >>> io.fabric8.kubernetes.client.KubernetesClientBuilder.getHttpClient(KubernetesClientBuilder.java:88) >>> >>> at >>> io.fabric8.kubernetes.client.KubernetesClientBuilder.build(KubernetesClientBuilder.java:78) >>> >>> at >>> org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:120) >>> >>> at >>> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:111) >>> >>> at >>> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:3037) >>> >>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:568) >>> >>> at >>> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58) >>> >>> at >>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>> Method) >>> >>> at >>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown >>> Source) >>> >>> at >>> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown >>> Source) >>> >>> at java.base/java.lang.reflect.Constructor.newInstance(Unknown >>> Source) >>> >>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247) >>> >>> at >>> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374) >>> >>> at py4j.Gateway.invoke(Gateway.java:238) >>> >>> at >>> py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80) >>> >>> at >>> py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69) >>> >>> at >>> py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) >>> >>> at >>> py4j.ClientServerConnection.run(ClientServerConnection.java:106) >>> >>> at java.base/java.lang.Thread.run(Unknown Source) >>> >>> Caused by: java.lang.ClassNotFoundException: okio.BufferedSource >>> >>> at >>> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source) >>> >>> at >>> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown >>> Source) >>> >>> at java.base/java.lang.ClassLoader.loadClass(Unknown Source) >>> >>> ... 26 more >>> >>> >>> >>> Replaced the existing jar with the JAR file at >>> https://repo1.maven.org/maven2/com/squareup/okio/okio/3.4.0/okio-3.4.0.jar >>> <https://secure-web.cisco.com/1bTvNPAJgVtYdy2nfHp1eUSEqLfelqshEI8TO89yzE25dM5y8HHDCwYxrzTLlmcAFi6uIbQLO2OiJht-xgXmI3lFdV8YpP0j3re47gncrBpwO9m6xYQeLhqXUAnUVP2MoxHbdHlZcdSwDqWkjbOKudm7Go1ICzxhw_VBXuK9n8XF3y7__B86mqWNsroDGD3hbH_tTQTHpXK-4tJCeIZTKmwItL1A3zlRL8lBHG_zgTDSiX9W7ufy8rHP2JZEp_FaftGMsnPA56IGHQVQAmOIobPSQDi4MfsiyUj0HsHPH3fZaz8_8TnPu178yfi8pCurkmr7b0X0NmFTdeAuFHKhdoOYooWDPsuBIYxknd3p1wLXrQezp26QrkjEiUMjNH9S18HPLH2BfN627X6zqQD7sVUUo1hzMRvnllVZVQWPL6H7lisyk-7w2pTAX6bm9wZuWTN9U4hZzjoc1-s1YumCiexaMOfiqEbTKppNDB8jOXBPIS9HDdEVDUl8OAIKz-T480x_NePZwHGT4hHtSwUaHCw/https%3A%2F%2Frepo1.maven.org%2Fmaven2%2Fcom%2Fsquareup%2Fokio%2Fokio%2F3.4.0%2Fokio-3.4.0.jar> >>> >>> >>> >>> >>> >>> PFB, the vulnerability details: >>> >>> Link: https://nvd.nist.gov/vuln/detail/CVE-2023-3635 >>> <https://secure-web.cisco.com/1KDv1iIbxjIsZCdyvwVzp9hDXe9ClcztVaj_gKzaoEQJ0Qb1BrTG7ivs0bsKiKVJvN8BJ0KvCwQKgWJGRfrWZYTkrgVMl1RfmnIn2fTYgyXd5ATU-4FBIQstOXRlc1dQnRNW9jr8OZCqV_xqbzAuLEP--uh0URczU8BYxyefL4Ly6ntQ2Y0BtKEOq3LZflTianf1d3UH30m_mmQmt3pE_3S7qFc9R9I3NqWJmkxuYVC1gVhnWBpbelMz5P7Q8D4GXo_L7tgj_nPwQyAcwqLjaIUVf-SYPU8T-WsaxeDkW6gp5oNKuYFqDzxXghsRJxzOj7i5noa1bj3-uSj0f0tT8xZ3L42uUTNgHczw65Kt1WnUK2-A_yhTmEhg07yFdwKQha6bQyn2KoicHjcdlQzAWsRmbBgzVjhDKMGdPn9Mrm5V7lw1QgeoFmddSJsreHy6TcNY2dXtqEzhw-OX2ibRtOrCX4M_n1ONE73yhGXAhqarKsd1tl5IgDfp_MlsFe9bkMa9G2AK5pcO0GeI8r7yDXA/https%3A%2F%2Fnvd.nist.gov%2Fvuln%2Fdetail%2FCVE-2023-3635> >>> >>> >>> >>> Any guidance here would be of great help. >>> >>> >>> >>> Thanks, >>> >>> Sanket A. >>> >>> This message (including any attachments) contains confidential >>> information intended for a specific individual and purpose, and is >>> protected by law. If you are not the intended recipient, you should delete >>> this message and any disclosure, copying, or distribution of this message, >>> or the taking of any action based on it, by you is strictly prohibited. >>> >>> Deloitte refers to a Deloitte member firm, one of its related entities, >>> or Deloitte Touche Tohmatsu Limited ("DTTL"). Each Deloitte member firm is >>> a separate legal entity and a member of DTTL. DTTL does not provide >>> services to clients. Please see www.deloitte.com/about to learn more. >>> >>> v.E.1 >>> >>> > > -- > Bjørn Jørgensen > Vestre Aspehaug 4, 6010 Ålesund > Norge > > +47 480 94 297 > -- Bjørn Jørgensen Vestre Aspehaug 4, 6010 Ålesund Norge +47 480 94 297