Hi, Why SPARK 1.5? Let me guess, are you using JAVA as well?
The world has moved on mate. Regards, Gourav On Fri, Sep 8, 2017 at 8:36 AM, PICARD Damien <damien.picard-...@socgen.com> wrote: > Hi ! > > > > I’m facing a Classloader problem using Spark 1.5.1 > > > > I use javax.validation and hibernate validation annotations on some of my > beans : > > > > @NotBlank > > @Valid > > *private* String attribute1 ; > > > > @Valid > > *private* String attribute2 ; > > > > When Spark tries to unmarshall these beans (after a remote RDD), I get the > ClassNotFoundException > : > > 17/09/07 09:19:25 INFO storage.BlockManager: Found block rdd_8_1 remotely > > 17/09/07 09:19:25 ERROR executor.Executor: Exception in task 3.0 in stage > 2.0 (TID 6) > > java.lang.ClassNotFoundException: org.hibernate.validator. > constraints.NotBlank > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > at java.lang.Class.forName0(Native Method) > > at java.lang.Class.forName(Class.java:348) > > at java.io.ObjectInputStream.resolveProxyClass( > ObjectInputStream.java:700) > > at java.io.ObjectInputStream.readProxyDesc( > ObjectInputStream.java:1566) > > at java.io.ObjectInputStream.readClassDesc( > ObjectInputStream.java:1518) > > at java.io.ObjectInputStream.readOrdinaryObject( > ObjectInputStream.java:1781) > > … > > > > Indeed, it means that the annotation class is not found, because it is not > in the classpath. Why ? I don’t know, because I make a uber JAR that > contains this class. I suppose that at the time the job tries to unmarshall > the RDD, the uber jar is not loaded. > > > > So, I try to add the hibernate JAR to the class loader manually, using > this spark-submit command : > > > > spark-submit --queue default \ > > --class com.my.Launcher \ > > --deploy-mode cluster \ > > --master yarn-cluster \ > > --driver-java-options "-Dfile.encoding=UTF-8" \ > > --jars /home/user/hibernate-validator-5.2.2.Final.jar \ > > --driver-class-path /home/user/hibernate-validator-5.2.2.Final.jar \ > > --conf "spark.executor.extraClassPath=/home/user/ > hibernate-validator-5.2.2.Final.jar" \ > > /home/user/uberjar-job.jar > > > > Without effects. So, is there a way to add this class to the classloader ? > > > > Thank you in advance. > > > > Damien > > ========================================================= > > Ce message et toutes les pieces jointes (ci-apres le "message") > sont confidentiels et susceptibles de contenir des informations > couvertes par le secret professionnel. Ce message est etabli > a l'intention exclusive de ses destinataires. Toute utilisation > ou diffusion non autorisee interdite. > Tout message electronique est susceptible d'alteration. La SOCIETE GENERALE > et ses filiales declinent toute responsabilite au titre de ce message > s'il a ete altere, deforme falsifie. > > ========================================================= > > This message and any attachments (the "message") are confidential, > intended solely for the addresses, and may contain legally privileged > information. Any unauthorized use or dissemination is prohibited. > E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any > of its subsidiaries or affiliates shall be liable for the message > if altered, changed or falsified. > > ========================================================= >
