Sean, Thanks. That worked.
Kevin On Mon, Sep 15, 2014 at 3:37 PM, Sean Owen <so...@cloudera.com> wrote: > This is more of a Java / Maven issue than Spark per se. I would use > the shade plugin to remove signature files in your final META-INF/ > dir. As Spark does, in its <configuration>: > > <filters> > <filter> > <artifact>*:*</artifact> > <excludes> > <exclude>org/datanucleus/**</exclude> > <exclude>META-INF/*.SF</exclude> > <exclude>META-INF/*.DSA</exclude> > <exclude>META-INF/*.RSA</exclude> > </excludes> > </filter> > </filters> > > On Mon, Sep 15, 2014 at 11:33 PM, kpeng1 <kpe...@gmail.com> wrote: > > Hi All, > > > > I am trying to submit a spark job that I have built in maven using the > > following command: > > /usr/bin/spark-submit --deploy-mode client --class com.spark.TheMain > > --master local[1] /home/cloudera/myjar.jar 100 > > > > But I seem to be getting the following error: > > Exception in thread "main" java.lang.SecurityException: Invalid signature > > file digest for Manifest main attributes > > at > > > sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:286) > > at > > > sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:239) > > at java.util.jar.JarVerifier.processEntry(JarVerifier.java:307) > > at java.util.jar.JarVerifier.update(JarVerifier.java:218) > > at java.util.jar.JarFile.initializeVerifier(JarFile.java:345) > > at java.util.jar.JarFile.getInputStream(JarFile.java:412) > > at > sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:775) > > at sun.misc.Resource.cachedInputStream(Resource.java:77) > > at sun.misc.Resource.getByteBuffer(Resource.java:160) > > at java.net.URLClassLoader.defineClass(URLClassLoader.java:436) > > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > > at java.lang.Class.forName0(Native Method) > > at java.lang.Class.forName(Class.java:270) > > at > org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:289) > > at > org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55) > > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > > > > > Here is the pom file I am using to build the jar: > > <project xmlns="http://maven.apache.org/POM/4.0.0"; > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > > http://maven.apache.org/maven-v4_0_0.xsd";> > > <modelVersion>4.0.0</modelVersion> > > <groupId>com.spark</groupId> > > <artifactId>myjar</artifactId> > > <version>0.0.1-SNAPSHOT</version> > > <name>${project.artifactId}</name> > > <description>My wonderfull scala app</description> > > <inceptionYear>2010</inceptionYear> > > <licenses> > > <license> > > <name>My License</name> > > <url>http://....</url> > > <distribution>repo</distribution> > > </license> > > </licenses> > > > > <properties> > > <cdh.version>cdh5.1.0</cdh.version> > > <maven.compiler.source>1.6</maven.compiler.source> > > <maven.compiler.target>1.6</maven.compiler.target> > > <encoding>UTF-8</encoding> > > <scala.tools.version>2.10</scala.tools.version> > > <scala.version>2.10.4</scala.version> > > </properties> > > > > <repositories> > > <repository> > > <id>scala-tools.org</id> > > <name>Scala-tools Maven2 Repository</name> > > <url>https://oss.sonatype.org/content/repositories/snapshots/ > </url> > > </repository> > > <repository> > > <id>maven-hadoop</id> > > <name>Hadoop Releases</name> > > > > <url>https://repository.cloudera.com/content/repositories/releases/ > </url> > > </repository> > > <repository> > > <id>cloudera-repos</id> > > <name>Cloudera Repos</name> > > <url>https://repository.cloudera.com/artifactory/cloudera-repos/ > </url> > > </repository> > > </repositories> > > <pluginRepositories> > > <pluginRepository> > > <id>scala-tools.org</id> > > <name>Scala-tools Maven2 Repository</name> > > <url>https://oss.sonatype.org/content/repositories/snapshots/ > </url> > > </pluginRepository> > > </pluginRepositories> > > > > <dependencies> > > <dependency> > > <groupId>org.scala-lang</groupId> > > <artifactId>scala-library</artifactId> > > <version>${scala.version}</version> > > </dependency> > > <dependency> > > <groupId>org.apache.spark</groupId> > > <artifactId>spark-core_2.10</artifactId> > > <version>1.0.0-${cdh.version}</version> > > </dependency> > > <dependency> > > <groupId>org.apache.spark</groupId> > > <artifactId>spark-tools_2.10</artifactId> > > <version>1.0.0-${cdh.version}</version> > > </dependency> > > <dependency> > > <groupId>org.apache.spark</groupId> > > <artifactId>spark-streaming-flume_2.10</artifactId> > > <version>1.0.0-${cdh.version}</version> > > </dependency> > > <dependency> > > <groupId>org.apache.spark</groupId> > > <artifactId>spark-streaming_2.10</artifactId> > > <version>1.0.0-${cdh.version}</version> > > </dependency> > > <dependency> > > <groupId>org.apache.flume</groupId> > > <artifactId>flume-ng-sdk</artifactId> > > <version>1.5.0-${cdh.version}</version> > > > > <exclusions> > > <exclusion> > > <groupId>io.netty</groupId> > > <artifactId>netty</artifactId> > > </exclusion> > > </exclusions> > > </dependency> > > <dependency> > > <groupId>org.apache.flume</groupId> > > <artifactId>flume-ng-core</artifactId> > > <version>1.5.0-${cdh.version}</version> > > > > <exclusions> > > <exclusion> > > <groupId>io.netty</groupId> > > <artifactId>netty</artifactId> > > </exclusion> > > </exclusions> > > </dependency> > > <dependency> > > <groupId>org.apache.hbase</groupId> > > <artifactId>hbase-client</artifactId> > > <version>0.98.1-${cdh.version}</version> > > > > <exclusions> > > <exclusion> > > <groupId>io.netty</groupId> > > <artifactId>netty</artifactId> > > </exclusion> > > </exclusions> > > </dependency> > > <dependency> > > <groupId>org.apache.hadoop</groupId> > > <artifactId>hadoop-client</artifactId> > > <version>2.3.0-${cdh.version}</version> > > > > </dependency> > > > > > > <dependency> > > <groupId>junit</groupId> > > <artifactId>junit</artifactId> > > <version>4.11</version> > > <scope>test</scope> > > </dependency> > > <dependency> > > <groupId>org.specs2</groupId> > > <artifactId>specs2_${scala.tools.version}</artifactId> > > <version>1.13</version> > > <scope>test</scope> > > </dependency> > > <dependency> > > <groupId>org.scalatest</groupId> > > <artifactId>scalatest_${scala.tools.version}</artifactId> > > <version>2.0.M6-SNAP8</version> > > <scope>test</scope> > > </dependency> > > </dependencies> > > > > <build> > > <sourceDirectory>src/main/scala</sourceDirectory> > > <testSourceDirectory>src/test/scala</testSourceDirectory> > > <plugins> > > <plugin> > > > > <groupId>net.alchim31.maven</groupId> > > <artifactId>scala-maven-plugin</artifactId> > > <version>3.1.3</version> > > <executions> > > <execution> > > <goals> > > <goal>compile</goal> > > <goal>testCompile</goal> > > </goals> > > <configuration> > > <args> > > <arg>-make:transitive</arg> > > <arg>-dependencyfile</arg> > > <arg>${project.build.directory}/.scala_dependencies</arg> > > </args> > > </configuration> > > </execution> > > </executions> > > </plugin> > > <plugin> > > <groupId>org.apache.maven.plugins</groupId> > > <artifactId>maven-surefire-plugin</artifactId> > > <version>2.13</version> > > <configuration> > > <useFile>false</useFile> > > <disableXmlReport>true</disableXmlReport> > > > > > > <includes> > > <include>**/*Test.*</include> > > <include>**/*Suite.*</include> > > </includes> > > </configuration> > > </plugin> > > <plugin> > > <groupId>org.apache.maven.plugins</groupId> > > <artifactId>maven-shade-plugin</artifactId> > > <version>2.2</version> > > <configuration> > > > > <transformers> > > <transformer > > > implementation="org.apache..plugins.shade.resource.ApacheLicenseResourceTransformer"> > > </transformer> > > <transformer > > > implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"> > > <resource>reference.conf</resource> > > </transformer> > > </transformers> > > </configuration> > > <executions> > > <execution> > > <phase>package</phase> > > <goals> > > <goal>shade</goal> > > </goals> > > </execution> > > </executions> > > </plugin> > > </plugins> > > </build> > > </project> > > > > Was wondering what I am missing, since the jar built and a MANIFEST.MF > file > > is created, not sure why it is looking for a signature file digest, am I > > missing some exclusion clause maybe? > > > > I saw a thread based on SBT and tried using the --deploy-mode client > option; > > even though, my understanding is by default it would use that mode > anyway. > > > > > > > > > > -- > > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Invalid-signature-file-digest-for-Manifest-main-attributes-with-spark-job-built-using-maven-tp14299.html > > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > > For additional commands, e-mail: user-h...@spark.apache.org > > >
