Re: Running Scala Word Count Using Maven

[fightf...@163.com]

Hi,
If you use maven, what is the actual compiling errors?



fightf...@163.com
 
From: Su She
Date: 2015-03-16 13:20
To: user@spark.apache.org
Subject: Running Scala Word Count Using Maven
Hello Everyone,  

I am trying to run the Word Count from here: 
https://github.com/holdenk/learning-spark-examples/blob/master/mini-complete-example/src/main/scala/com/oreilly/learningsparkexamples/mini/scala/WordCount.scala

I was able to successfully run the app using SBT, but not Maven. I don't see a 
difference between my pom file and the one used online. 

From what I understand from "Learning Spark", it should be possible to package 
scala code using Maven?

My scala code is HelloWorld.scala under /src/main/scala/ and is exactly the 
same as the example online. 

Thank you for the help!

My pom file:

<project xmlns = "http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://w3.org/2001/XMLSchema-instance"; xsi:schemaLocation 
="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd";>
        <groupId> edu.berkely</groupId>
        <artifactId> simple-project </artifactId>
        <modelVersion> 4.0.0</modelVersion>
        <name> Simple Project </name>
        <packaging> jar </packaging>
        <version> 1.0 </version>
<repositories>
        <repository>
        <id>cloudera</id>
        <url> http://repository.cloudera.com/artifactory/cloudera-repos/</url>
        </repository>
</repositories>
<dependencies>
        <dependency> <!--Spark dependency -->
        <groupId> org.apache.spark</groupId>
        <artifactId>spark-core_2.10</artifactId>
        <version>1.2.0-cdh5.3.0</version>
        </dependency>

        <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>2.5.0-mr1-cdh5.3.0</version>
        </dependency>

        <dependency>
        <groupId> org.apache.spark </groupId>
        <artifactId> spark-streaming-kafka_2.10</artifactId>
        <version>1.2.0</version>
        </dependency>

        <dependency>
        <groupId> org.apache.spark </groupId>
        <artifactId> spark-streaming_2.10 </artifactId>
        <version>1.2.0 </version>
        </dependency>

        <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>2.10.4</version>
        </dependency>

        <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-compiler</artifactId>
        <version>2.10.4</version>
        </dependency>

        <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.10</artifactId>
        <version>0.8.2-beta</version>
        </dependency>

        <dependency>
        <groupId>com.101tec</groupId>
        <artifactId>zkclient</artifactId>
        <version>0.3</version>
        </dependency>

         <dependency>
         <groupId>com.yammer.metrics</groupId>
         <artifactId>metrics-core</artifactId>
         <version>2.2.0</version>
         </dependency>

         <dependency>
         <groupId>org.apache.kafka</groupId>
         <artifactId>kafka-clients</artifactId>
         <version>0.8.2-beta</version>
         </dependency>

        <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-yarn-server-web-proxy</artifactId>
        <version>2.5.0</version>
        </dependency>

        <dependency>
        <groupId>com.datastax.spark</groupId>
        <artifactId>spark-cassandra-connector_2.10</artifactId>
        <version>1.2.0-alpha1</version>
        </dependency>

        <dependency>
        <groupId>com.datastax.spark</groupId>
        <artifactId>spark-cassandra-connector-java_2.10</artifactId>
        <version>1.2.0-alpha1</version>
        </dependency>

        <dependency>
        <groupId>com.datastax.cassandra</groupId>
        <artifactId>cassandra-driver-core</artifactId>
        <version>2.1.3</version>
        </dependency>

        <dependency>
        <groupId>org.apache.cassandra</groupId>
        <artifactId>cassandra-thrift</artifactId>
        <version>2.1.3</version>
        </dependency>

        <dependency>
        <groupId>org.apache.thrift</groupId>
        <artifactId>libthrift</artifactId>
        <version>0.9.2</version>
        </dependency>

        <dependency>
        <groupId>org.apache.cassandra</groupId>
        <artifactId>cassandra-clientutil</artifactId>
        <version>2.1.3</version>
        </dependency>

        <dependency>
        <groupId>com.google.guava</groupId>
        <artifactId>guava</artifactId>
        <version>18.0</version>
        </dependency>

</dependencies>

<build>
        <pluginManagement>
         <plugins>
          <plugin>
           <groupId>org.apache.maven.plugins</groupId>
           <artifactId>maven-compiler-plugin</artifactId>
           <version>3.1</version>
           <configuration>
            <source>1.7</source>
            <target>1.7</target>
           </configuration>
          </plugin>
         </plugins>
        </pluginManagement>
</build>
</project>

Thank you for the help!

-Su